# Higher %wa value on OpenVZ server



## ICPH (May 20, 2014)

Hello, there is higher %wa value on top command both on VPS and on host server (OpenVZ virtualization). Both RHEL CentOS 5.x

VPS:
 



> [[email protected]* hqs]# w
> 18:13:05 up 17 days, 12:34,  1 user,  load average: 1.08, 1.40, 1.33
> USER     TTY      FROM              [email protected]   IDLE   JCPU   PCPU WHAT
> root     pts/0    :1.0             03May14 17days  0.01s  0.01s -bash
> ...


Host server:
 



> -bash-3.2# w;iostat;vmstat
> 12:15:17 up 21 days,  7:05,  1 user,  load average: 26.44, 28.58, 20.12
> USER     TTY      FROM              [email protected]   IDLE   JCPU   PCPU WHAT
> root     pts/0    chomsky.torserve 12:00    0.00s  0.03s  0.02s w
> ...





> io, packets per second on host server for all OpenVZ VMs:


host server load is usually around 9, server having 16 cores.
 

Please anyone can advice some commands or ideas? Thank you


----------



## mtwiscool (May 20, 2014)

The wait is caused when the host node has run out of disk io.

Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.

I hope you liked my reply.

Matthew Morgan


----------



## TruvisT (May 20, 2014)

Are you running RAID 10? What is the hard drive health? RAID Card Controller?


----------



## SkylarM (May 20, 2014)

mtwiscool said:


> The wait is caused when the host node has run out of disk io.
> 
> Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.
> 
> ...


 Have you ever been so far even as decided to use go want to look more like?


----------



## mtwiscool (May 20, 2014)

SkylarM said:


> Have you ever been so far even as decided to use go want to look more like?


Because i seen this issue before usely a reboot will fix it


----------



## SkylarM (May 20, 2014)

mtwiscool said:


> Because i seen this issue before usely a reboot will fix it


Can you reboot your brain please?


----------



## mtwiscool (May 20, 2014)

SkylarM said:


> Can you reboot your brain please?


you can first.


----------



## DomainBop (May 20, 2014)

> Please anyone can advice



Don't oversell OpenVZ nodes to the point of overloading is the easiest way to avoid 13.08% iowait



> Are you running RAID 10?


device:
sda
sda1
sda2
heh..what is RAID?



> Can you reboot your brain please?


http://abcnews.go.com/Health/Wellness/10-tricks-reboot-brain/story?id=18252612


----------



## Deleted (May 20, 2014)

Sounds like a process is writing lots of data to disk. SATA disks do not support disconnected writes, which can create high I/O wait times. 

Probably a misconfigured MySQL with stupid innodb buffer settings.


----------



## Damian (May 20, 2014)

Is there a hardware RAID controller in place? Even basic non-caching controllers will markedly improve disk i/o.



DomainBop said:


> device:
> sda
> sda1
> sda2
> heh..what is RAID?


Drives on LSI controllers still appear as sdX


----------



## ICPH (May 21, 2014)

mtwiscool said:


> The wait is caused when the host node has run out of disk io.
> 
> Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.
> 
> ...





Monkburger said:


> Sounds like a process is writing lots of data to disk. SATA disks do not support disconnected writes, which can create high I/O wait times.
> 
> Probably a misconfigured MySQL with stupid innodb buffer settings.


I think You are very near of what is happening on my server..

I have one big mysql on vm 190, but mysql tunners reports optimal settings.

Here is the iotop you adviced to run:


----------



## Thelen (May 21, 2014)

13% isn't that bad. It doesn't mean the disks are at capacity necessarily, just means the io flushes are having to wait a few seeks (50-100ms).


----------



## Xenfinity (May 22, 2014)

CPU I/O wait isn't nearly as important as CPU idle.  Any process blocked by waiting on the disk will be counted in %iowait.

I found some articles (albeit old and about AIX; still relevant to Linux) that explain why I/O wait could be misleading about server performance.

You say that your load average is consistently around 9.00 of 16 cores, which means that your host node can comfortably handle the virtual machines that you have.

Another thing to check is the %idle.  This value being low is more likely indicative of a server performance problem than a higher %iowait.

You could check out your most I/O abusive VPS containers to see what they are doing and throttle them if they do end up causing an I/O bottleneck.

Nick


----------



## ICPH (May 23, 2014)

thx,

%iddle appears to be average 50-60% when doing "sar -u"

i also created this topic because any actions via terminal (SSH) like logging writing/doing command started to be signifficantly slow. Example before password login prompt appear, like sometimes 5 seconds..


----------



## Xenfinity (May 23, 2014)

ICPH said:


> %iddle appears to be average 50-60% when doing "sar -u"


That seems quite healthy to me.



ICPH said:


> i also created this topic because any actions via terminal (SSH) like logging writing/doing command started to be signifficantly slow. Example before password login prompt appear, like sometimes 5 seconds..


Does this happen to all VPS containers and/or the host node?

If writing commands itself is slow (like a significant lag between keystrokes), there could be packet loss or high ping.  Unless you're talking about tab completion for a directory listing, which would request data from the disk.

The password login prompt can be that slow, and it seems to be worse with packet loss or high ping, probably due to reverse DNS lookups and negotiating authentication back and forth.  According to this Super User question, you can switch "GSSAPIAuthentication" and "UseDNS" to "no" in /etc/ssh/sshd_config.  Then, after restarting `sshd`, your login prompt should be faster.

What kinds of commands are slow, aside from the password login prompt?

Nick


----------



## ICPH (May 24, 2014)

Xenfinity said:


> - Does this happen to all VPS containers and/or the host node?
> 
> - What kinds of commands are slow, aside from the password login prompt?


- It happens on host node and also on containers.

- Almost all commands, except for example command "date", free -m, df -h are slow. THe loggin in is like: wait 5 seconds, write username, enter, wait 5 seconds, and evena fter confirming password, last login 5 seconds.. like this, and commands like df -h and free .m around 3 seconds. Then after a while it start going normal - instant. Its not big issue


----------



## Xenfinity (May 24, 2014)

Hmm, I'd consider that _almost_ concerning.  I would suggest not adding any more VPS containers to that host node or I/O would become uncomfortably slow.

What's the disk hardware like?  Model?  Size?  RAID?  Perhaps the community here can help you identify a new hardware rearrangement to help with I/O performance... though it would mean taking the host node down for a hardware upgrade.  You could ask your clients on the server if they'd like to have their hardware upgraded.

You could approach from the other end to find what processes are causing this mediocre I/O performance.  `*iotop*` can help you identify actual I/O usage and `*ps aux | grep ' D'*` would identify processes that are blocked waiting on the disk.  If you want to examine a process in detail, `*strace -p PID*` (where _PID _is the process ID) would show you what the process is doing.

I actually used those three commands earlier today when I tried doing ten full cPanel account restores at the exact same time on a moderately busy web server.  I learned that


reading through backups takes up most of the I/O, since cPanel has to extract and uncompress these big tarballs,
customers' PHP processes wait on MySQL to read from the disk a lot, and
I should do restores in series instead of in parallel...
Nick


----------

