Using the terminal to get system statistics

If a service is running not as snappy as it should I use the sysstat tools to check the current load of my vps.
They collect usage statistics for CPU, memory, network, block devices, etc.

Installation is easy:


apt-get install sysstat

The package itself does contain three main components:
- sar to display current information
- sa1 to collect and store statistics
- sa2 to create a daily report in /var/log/sa

Configuration is done through:


nano /etc/sysstat/sysstat

Content:


# sysstat configuration file. See sysstat(5) manual page.

# How long to keep log files (in days).
# Used by sa2(8) script
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=7

# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=10

# Parameters for the system activity data collector (see sadc(8) manual page)
# which are used for the generation of log files.
# By default contains the `-S DISK' option responsible for generating disk
# statisitcs. Use `-S XALL' to collect all available statistics.
SADC_OPTIONS="-S DISK"

The cronjob configuration can be found here:


nano /etc/cron.d/sysstat

Content:


# The first element of the path is a directory where the debian-sa1
# script is located
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

Usage of the command line tool sar:

• Report CPU utilization

sar -u 3 10

Output:


02:06:26 PM CPU %user %nice %system %iowait %steal %idle
02:06:29 PM all 24.83 0.00 0.42 0.00 0.42 74.33
02:06:32 PM all 24.79 0.00 0.50 0.00 0.33 74.37
02:06:35 PM all 24.73 0.00 0.25 0.00 0.33 74.69
02:06:38 PM all 24.64 0.00 0.50 0.00 0.25 74.60
02:06:41 PM all 24.62 0.00 0.67 0.00 0.25 74.46

• Report  per-processor  statistics

• Report queue length and load averages.

Output:


02:07:11 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
02:07:14 PM 1 808 0.64 0.26 0.12 0
02:07:17 PM 1 808 0.67 0.27 0.12 0
02:07:20 PM 1 808 0.67 0.27 0.12 0
02:07:23 PM 1 808 0.69 0.28 0.13 0

• Report I/O and transfer rate statistics

Output:


02:08:34 PM tps rtps wtps bread/s bwrtn/s
02:08:37 PM 0.67 0.00 0.67 0.00 24.00
02:08:40 PM 1.00 0.00 1.00 0.00 26.67
02:08:43 PM 0.33 0.00 0.33 0.00 10.67
02:08:46 PM 0.67 0.00 0.67 0.00 24.00

• Report memory utilization statistics

Output:


02:09:00 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact
02:09:03 PM 240924 1799452 88.19 262428 994380 2235528 54.39 909024 683068
02:09:06 PM 241048 1799328 88.19 262428 994384 2235528 54.39 909024 683068
02:09:09 PM 241048 1799328 88.19 262428 994392 2235528 54.39 909032 683072
02:09:12 PM 240800 1799576 88.20 262428 994396 2235528 54.39 909080 683072

• Report  activity  for  each  block device

Output:


02:10:01 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
02:10:04 PM dev250-0 39.00 0.00 6560.00 168.21 1.85 47.38 3.73 14.53
02:10:04 PM scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:10:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
02:10:07 PM dev250-0 1.33 0.00 90.67 68.00 0.01 5.00 3.00 0.40
02:10:07 PM scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00


Type "man sar" for additional information.

Great stuff. Also remember for network stats:

sar -N DEV

And to reference a file from a previous day, they typically rotate with days of the month and you can reference a date by it's filename, example, March 17th would be:

sar -f /var/log/sa/sa17

So for network stats for March 17, you would use:

sar -f /var/log/sa/sa17 -n DEV

good one.. thanks for sharing

sar is very useful. I picked up a tip from WHT forums for filtering sar historical stats to find specific cpu load averages for troubleshooting. Then just made a one liner with prompt for entering a cpu load average value.

You can find what time of day(s) of had a cpu load average above a certain value. i.e. cpu load above 0.10

one liner looks for 5min interval value


read -p "Enter cpu load threshold >= " loadavg; for log in `/bin/ls -1 /var/log/sa/sa[0-9]*`; do echo $log; sar -q -f $log | egrep -v "Average|ldavg" | awk -v lvg=$loadavg '{if ($5>=lvg) print $1,$2,$4,$5,$6,$7}'; echo ""; done 

output



Enter cpu load threshold >= 0.10
 
/var/log/sa/sa10
Linux 2.6.32-358.6.2.el6.x86_64 03/10/14 _x86_64_ (1 CPU)
16:00:01 3 0.22 0.14 0.04 
 
/var/log/sa/sa11
Linux 2.6.32-358.6.2.el6.x86_64 03/11/14 _x86_64_ (1 CPU)
 
/var/log/sa/sa12
Linux 2.6.32-358.6.2.el6.x86_64 03/12/14 _x86_64_ (1 CPU)
17:10:01 2 0.30 0.12 0.04 
 
/var/log/sa/sa13
Linux 2.6.32-358.6.2.el6.x86_64 03/13/14 _x86_64_ (1 CPU)
13:10:01 1 0.21 0.11 0.04 
 
/var/log/sa/sa14
Linux 2.6.32-358.6.2.el6.x86_64 03/14/14 _x86_64_ (1 CPU)
 
/var/log/sa/sa15
Linux 2.6.32-358.6.2.el6.x86_64 03/15/14 _x86_64_ (1 CPU)
 
/var/log/sa/sa16
Linux 2.6.32-358.6.2.el6.x86_64 03/16/14 _x86_64_ (1 CPU)
05:10:01 1 0.18 0.14 0.05 
 
/var/log/sa/sa17
Linux 2.6.32-358.6.2.el6.x86_64 03/17/14 _x86_64_ (1 CPU)
05:10:01 2 0.26 0.10 0.03 
09:10:01 3 0.08 0.10 0.04 
 
/var/log/sa/sa18
Linux 2.6.32-358.6.2.el6.x86_64 03/18/14 _x86_64_ (1 CPU)
05:10:01 2 0.29 0.11 0.03 
17:10:01 2 0.18 0.12 0.04 


this is on CentOS so one liner might need tweaking on other distros

Great finding. sar can work with keywords: "sar -n KEYWORD" from manpage:


DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6

I like the network history sar offers.

peterw said:

Great finding. sar can work with keywords: "sar -n KEYWORD" from manpage:


DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6

I like the network history sar offers.

Click to expand...

Thank you for the additional parameters - missed them but someone might like to watch all the TCP/UDP stuff.

Good all in one tool, vs iftop etc.