amuck-landowner

Higher %wa value on OpenVZ server

ICPH

ICPH

Member
Hello, there is higher %wa value on top command both on VPS and on host server (OpenVZ virtualization). Both RHEL CentOS 5.x

VPS:
 

[root@* hqs]# w
 18:13:05 up 17 days, 12:34,  1 user,  load average: 1.08, 1.40, 1.33
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    :1.0             03May14 17days  0.01s  0.01s -bash
[root@VPS hqs]# iostat
Linux 2.6.18-371.3.1.el5.028stab110.1 (*)     05/20/14

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.68    0.16    7.90   12.80    0.00   70.46

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

[root@VPS hqs]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 9047468      0      0    0    0    13    24    0    7  9  8 70 13  0
Click to expand...
Host server:
 

-bash-3.2# w;iostat;vmstat
 12:15:17 up 21 days,  7:05,  1 user,  load average: 26.44, 28.58, 20.12
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    chomsky.torserve 12:00    0.00s  0.03s  0.02s w
Linux 2.6.18-371.3.1.el5.028stab110.1 (*)     05/20/14

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.87    0.16    3.60   13.08    0.00   72.29

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             263.58       521.79      9977.21  960046912 18357245870
sda1              0.00         0.00         0.00       2302         38
sda2            263.58       521.79      9977.21  960044330 18357245832
dm-0           1265.03       521.79      9977.21  960041306 18357244688
dm-1              0.00         0.00         0.00       2536       2136

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 8  7    288 561216 999776 15036684    0    0    16   312    1    1 11  4 72 13  0
Click to expand...
io, packets per second on host server for all OpenVZ VMs:
nNtgf.jpg
Click to expand...
host server load is usually around 9, server having 16 cores.
 

Please anyone can advice some commands or ideas? Thank you
 
M

mtwiscool

New Member
The wait is caused when the host node has run out of disk io.

Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.

I hope you liked my reply.

Matthew Morgan
 
TruvisT

TruvisT

Server Management Specialist
Verified Provider
Are you running RAID 10? What is the hard drive health? RAID Card Controller?
 
S

SkylarM

Well-Known Member
Verified Provider
mtwiscool said:
The wait is caused when the host node has run out of disk io.

Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.

I hope you liked my reply.

Matthew Morgan
Click to expand...
Have you ever been so far even as decided to use go want to look more like?
 
D

Deleted

Jail
Sounds like a process is writing lots of data to disk. SATA disks do not support disconnected writes, which can create high I/O wait times. 

Probably a misconfigured MySQL with stupid innodb buffer settings.
 
ICPH

ICPH

Member
mtwiscool said:
The wait is caused when the host node has run out of disk io.

Aka if you are on a vps ask your host to kick of any abusers and if its your own server use the tool iotop to tell you whats using the disk io but it can be also cause by cpu overload but this is less lickly.

I hope you liked my reply.

Matthew Morgan
Click to expand...

Monkburger said:
Sounds like a process is writing lots of data to disk. SATA disks do not support disconnected writes, which can create high I/O wait times. 

Probably a misconfigured MySQL with stupid innodb buffer settings.
Click to expand...
I think You are very near of what is happening on my server..

I have one big mysql on vm 190, but mysql tunners reports optimal settings.

Here is the iotop you adviced to run:

3a3K1.jpg
 
T

Thelen

New Member
Verified Provider
13% isn't that bad. It doesn't mean the disks are at capacity necessarily, just means the io flushes are having to wait a few seeks (50-100ms).
 
Xenfinity

Xenfinity

New Member
Verified Provider
CPU I/O wait isn't nearly as important as CPU idle.  Any process blocked by waiting on the disk will be counted in %iowait.

I found some articles (albeit old and about AIX; still relevant to Linux) that explain why I/O wait could be misleading about server performance.

You say that your load average is consistently around 9.00 of 16 cores, which means that your host node can comfortably handle the virtual machines that you have.

Another thing to check is the %idle.  This value being low is more likely indicative of a server performance problem than a higher %iowait.

You could check out your most I/O abusive VPS containers to see what they are doing and throttle them if they do end up causing an I/O bottleneck.

Nick
 
ICPH

ICPH

Member
thx,

%iddle appears to be average 50-60% when doing "sar -u"

i also created this topic because any actions via terminal (SSH) like logging writing/doing command started to be signifficantly slow. Example before password login prompt appear, like sometimes 5 seconds..
 
Xenfinity

Xenfinity

New Member
Verified Provider
ICPH said:
%iddle appears to be average 50-60% when doing "sar -u"
Click to expand...
That seems quite healthy to me.

ICPH said:
i also created this topic because any actions via terminal (SSH) like logging writing/doing command started to be signifficantly slow. Example before password login prompt appear, like sometimes 5 seconds..
Click to expand...
Does this happen to all VPS containers and/or the host node?

If writing commands itself is slow (like a significant lag between keystrokes), there could be packet loss or high ping.  Unless you're talking about tab completion for a directory listing, which would request data from the disk.

The password login prompt can be that slow, and it seems to be worse with packet loss or high ping, probably due to reverse DNS lookups and negotiating authentication back and forth.  According to this Super User question, you can switch "GSSAPIAuthentication" and "UseDNS" to "no" in /etc/ssh/sshd_config.  Then, after restarting `sshd`, your login prompt should be faster.

What kinds of commands are slow, aside from the password login prompt?

Nick
 
ICPH

ICPH

Member
Xenfinity said:
- Does this happen to all VPS containers and/or the host node?

- What kinds of commands are slow, aside from the password login prompt?
Click to expand...
- It happens on host node and also on containers.

- Almost all commands, except for example command "date", free -m, df -h are slow. THe loggin in is like: wait 5 seconds, write username, enter, wait 5 seconds, and evena fter confirming password, last login 5 seconds.. like this, and commands like df -h and free .m around 3 seconds. Then after a while it start going normal - instant. Its not big issue
 
Last edited by a moderator:
Xenfinity

Xenfinity

New Member
Verified Provider
Hmm, I'd consider that almost concerning.  I would suggest not adding any more VPS containers to that host node or I/O would become uncomfortably slow.

What's the disk hardware like?  Model?  Size?  RAID?  Perhaps the community here can help you identify a new hardware rearrangement to help with I/O performance... though it would mean taking the host node down for a hardware upgrade.  You could ask your clients on the server if they'd like to have their hardware upgraded.

You could approach from the other end to find what processes are causing this mediocre I/O performance.  `iotop` can help you identify actual I/O usage and `ps aux | grep ' D'` would identify processes that are blocked waiting on the disk.  If you want to examine a process in detail, `strace -p PID` (where PID is the process ID) would show you what the process is doing.

I actually used those three commands earlier today when I tried doing ten full cPanel account restores at the exact same time on a moderately busy web server.  I learned that

  • reading through backups takes up most of the I/O, since cPanel has to extract and uncompress these big tarballs,
  • customers' PHP processes wait on MySQL to read from the disk a lot, and
  • I should do restores in series instead of in parallel...
Nick
 
You must log in or register to reply here.
Top
amuck-landowner