VPS crashes every few day, really can't pin down the reason.

[Greg]

Attached are monthly graphs and htop stats

apache server status doesn't hint on clear winner, there are processes from all sites.

It's 50-60 WP sites with low traffic on the 2cpu, 2gb ram, 6gb swap SSD VPS

wp-login.php anx xmlrpc.php are blocked server wide

I'm not very experienced but at this point I don't know what else I should check to find the culprit.  Or at least narrow it down.

Every several days CPU and RAM usage goes through the roof and i have to restart mysql or the whole server to get back to normal for another few days.

 

[zzrok]

You can't fit 400 Apache HTTPD connections in 2GB of memory, assuming Apache is using the prefork MPM.  This is at least part of your problem.  I would start by turning that down to 100, but you may have to turn that down further.  Apache should be configured such that the maximum number of processes it will spawn will not cause the server to use swap.  You can't fix any other problems until the swapping issue is fixed because swapping often obfuscates other problems.

 

[fixidixi]

..or dont use apache at all..

 

[Premiumn]

..or dont use apache at all..

I prefer apache to nginx, because its quicker and easier to install - and it just works properly.

Being a noob, i always get issues with nginx, and i found that its not worth it.

 

[raindog308]

What exactly does "crash" mean in this context?  

Is there any thing in /var/log files?

If it's not OvZ, does dmesg tell you anything?

Or does it just become nonresponsive because CPU usage is so high?  A top from when that is happening would be more informative than a retrospective chart.

 

[MannDude]

I prefer apache to nginx, because its quicker and easier to install - and it just works properly.

Being a noob, i always get issues with nginx, and i found that its not worth it.

I used to feel the same way. You probably get php and 503 errors with Nginx? I used to get those often and get frustrated, however now I find nginx much easier to work with and would be lost working with apache. If you got a spare few bucks it'd be worth setting up a cheap dev server somewhere to play with nginx.

 

[drmike]

Appears from the monitoring / reporting graphic that number of MySQL queries on those spikes goes nuts.

You are running 0-37 queries per second, but during those blips (assuming they correspond to problem times)  over 7 million queries.

Even if take the graph spikes and apply them to being whole 24 hour period:

7,000,000+ / 24 = 81+ queries per second second sustained each and every second of the 24 hour period.  Looking at doubling of queries and sustained and realistically far more since the graphs pop like that means less on time monitored.

You need to track down what is going on in MySQL, perhaps log slow queries to file.   Such load has to be queuing up and wait timing MySQL so getting the problem queries should help isolate things.

My guess is something runs a cron job, but even that is suspect.   What is this software you are running that is utilizing MySQL?

 

[Dillybob]

I used to feel the same way. You probably get php and 503 errors with Nginx? I used to get those often and get frustrated, however now I find nginx much easier to work with and would be lost working with apache. If you got a spare few bucks it'd be worth setting up a cheap dev server somewhere to play with nginx.

Don't need even need to buy a dev server to play with nginx anymore. Take a look at this puppy!

https://wtriple.com/wtnmp/

Been using this for years for my game server in nodejs. Thing is amazing, makes testing scripts in nginx/ratelimitting so much easier.  

I basically use windows for development, then switch to a linux debian box for lunch for production. I got my 75 cent's worth of iptable testing with CVPS though, DOS'd the shit out of that thing, LOL.

 

[TheLinuxBug]

From experience I can guess of at least two things that are happening here:


1. As others have said, too many connections are allowed for your apache instance. Depending on your php settings you are likely using 50-200mb per instance, especially if this is WordPress. Reducing number of simultaneous connection apache can take will reduce the memory footprint.


2. You need to tune MySql. Most specifically you should research using /dev/shm as you MySql tmpdir setting. Most likely you are seeing all your apache sessions wait on MySql; joins, selects and large queries can take a lot of time to complete, even on ssd. Using memory for this will result in faster queries and less waiting by apache on MySql, less needed apache servers, less memory, hopefully no OOM.


P.S. some WordPress plugins are awful under pressure and have very poorly optimized mysql queries, this could be adding to your isuues especially if you site is hit by a bunch of bots and spiders.


Hope this helps.


Cheers!