Is that a normal iowait level for spinning disks?

[Amitz]

Dear all,

I thought I try my luck here too:

I have moved a busy website from a SSD VPS (Xen) to a dedicated server with two 7.2k spinning disks in Hardware RAID-1.

My munin graphs of the CPU load on the new server shows quite an amount of iowait:



I ran into other limits with the VPS, but I never had so much iowait there. As said, the VPS had SSDs in RAID-10.

Shall I consider this as normal for spinning disks in RAID-1? The site feels snappy, so no problem, but I would be interested anyway...

Thanks a lot for your input in advance!

Kind regards

-Amitz

 

[Amfy]

We need more information to give an opinion about how normal this iowait rate is. It really depends on what you do with the server. Try iotop, top, etc.

If you are writing a lot, it can be normal.

Also, spinning disks have a limit of often less than 100 iops while SSDs are fine with 50k. So if you used couple of hundred iops on your VPS it felt quite normal and your provider didn't say anything since there was enough room for your VPS neighbors. If you start using the same amount of iops on that dedicated servers you run into issues like this.

 

[Amitz]

Hi Amfy,


thanks a lot - I found the reason in the meantime and you are right about constant writing: It was nginx writing its access.log. As soon as I turned off logging, iowait disappeared. As said, the site is busy with many requests. I do not need access logs, so I simply keep it turned off.

 

[Amitz]

double post

 

[Enterprisevpssolutions]

You have to remember that you lose the trim function when using a hardware raid card as they are not supporting that functions yet.

 

[Magiobiwan]

You have to remember that you lose the trim function when using a hardware raid card as they are not supporting that functions yet.

Some do. Don't have the list in front of me but I have looked into it. 

 

[Amfy]

Hi Amfy,


thanks a lot - I found the reason in the meantime and you are right about constant writing: It was nginx writing its access.log. As soon as I turned off logging, iowait disappeared. As said, the site is busy with many requests. I do not need access logs, so I simply keep it turned off.

I probably remember which site you are talking about However, glad to hear it is solved now.

By the way, nginx access logs are kinda async, means they will never impact your visitors browsing the website. The logs are being written in the background. But you are right, turning them off is even better.

Also think about noatime and nodiratime. Every visitor will load a couple of pictures, every time this happens the server has to write when the pictures were accessed the last time. So I'm sure this can also help. Additionally I made the experience that ext3 performs best for such scenarios (compared to ext4).

Feel free to message me if you need any help. 

 

[Amfy]

Just taken a shower and wondering... that page was almost statically? You could throw some Varnish(+Cloudflare) in? Varnish works completely in-memory. No more issues regarding disk i/o.

Varnish perfectly works up to a couple of Gbit/s, then only Nginx can help (8-9/Gbit/s). I had a quite busy project where I had to try out these limits

 

[Amitz]

Dear Amfy,

thank you again!

Yes, it is probably the site that you remember... ;-)

Most of the content is dynamically generated by PHP. I am already using an OPCache for this and it does wonders to the performance. The static content is, as said, served by nginx. I have tried varnish once for that project, but it caused more hassle than benefit. However, in other scenarios, varnish can be just fantastic!

Very good to know that you made your fair share of experience with busy sites - I will gladly ask for your kind advice when a problem arises. Thank you very much for your offer!

Kind regards & Liebe Gruesse

- Amitz

Also think about noatime and nodiratime

Oh yes! I will take a look directly!