≡ Menu

buffers

Q. How can I find out Linux Resource utilization using vmstat command? How do I get information about high disk I/O and memory usage?

A. vmstat command reports information about processes, memory, paging, block IO, traps, and cpu activity. However, a real advantage of vmstat command output - is to the point and (concise) easy to read/understand. The output of vmstat command use to help identify system bottlenecks. Please note that Linux vmstat does not count itself as a running process.

Here is an output of vmstat command from my enterprise grade system:
$ vmstat -S M
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1963    607   2359    0    0     0     0    0     1 32  0 68  0

Where,

  • The fist line is nothing but six different categories. The second line gives more information about each category. This second line gives all data you need.
  • -S M: vmstat lets you choose units (k, K, m, M) default is K (1024 bytes) in the default mode. I am using M since this system has over 4 GB memory. Without -M option it will use K as unit

$ vmstat
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0 2485120 621952 2415368  0    0     0     0    0     1 32  0 68  0

Field Description For Vm Mode

(a) procs is the process-related fields are:

  • r: The number of processes waiting for run time.
  • b: The number of processes in uninterruptible sleep.

(b) memory is the memory-related fields are:

  • swpd: the amount of virtual memory used.
  • free: the amount of idle memory.
  • buff: the amount of memory used as buffers.
  • cache: the amount of memory used as cache.

(c) swap is swap-related fields are:

  • si: Amount of memory swapped in from disk (/s).
  • so: Amount of memory swapped to disk (/s).

(d) io is the I/O-related fields are:

  • bi: Blocks received from a block device (blocks/s).
  • bo: Blocks sent to a block device (blocks/s).

(e) system is the system-related fields are:

  • in: The number of interrupts per second, including the clock.
  • cs: The number of context switches per second.

(f) cpu is the CPU-related fields are:

These are percentages of total CPU time.

  • us: Time spent running non-kernel code. (user time, including nice time)
  • sy: Time spent running kernel code. (system time)
  • id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
  • wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

As you see the first output produced gives averages data since the last reboot. Additional reports give information on a sampling period of length delay. You need to sample data using delays i.e. collect data by setting intervals. For example collect data every 2 seconds (or collect data every 2 second 5 times only):
$ vmstat -S M 2
OR
$ vmstat -S M 2 5
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1756    607   2359    0    0     0     0    0     1 32  0 68  0
3  0      0   1756    607   2359    0    0     0     0 1018    65 38  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    64 37  0 63  0
3  0      0   1756    607   2359    0    0     0    20 1018    72 37  0 63  0
3  0      0   1756    607   2359    0    0     0     0 1012    64 37  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    65 38  0 63  0
3  0      0   1995    607   2359    0    0     0     0 1012    62 35  2 63  0
3  0      0   1731    607   2359    0    0     0     0 1012    64 34  3 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    72 38  0 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    63 37  0 63  0

This is what most system administrators do to identify system bottlenecks. I hope all of you find vmstat data is concise and easy to read.

See also:

Howto Reboot or halt Linux system in emergency

Linux kernel includes magic system request keys. It was originally developed for kernel hackers. However, you can use this hack to reboot, shutdown or halt computer safely (remember safe reboot/shutdown == flush filesystem buffers and unmount file system and then reboot so that data loss can be avoided).

This is quite useful when Linux based system is not available after boot or after a X server crashed ( svgalib program crashes) or no display on screen. Sysrq key combo forces the kernel to respond it regardless of whatever else it is doing, unless it is completely locked up (dead).

Using further extension to iptables called ipt_sysrq (new iptables target), which allows you to do the same as the magic sysrq key on a keyboard does, but over the network. So if your network server is not responding you can still reboot it. Please note that Magic SysRq support need to be compiled in your kernel. You need to say "yes" to 'Magic SysRq key (CONFIG_MAGIC_SYSRQ)' when configuring the kernel. I'm assuming that you have Magic SysRq key' support is compiled in your kernel.

Enable sysrq keys

By default it is not enabled on many Linux distributions. Add or modify following line (as soon as new Linux system installed) /etc/sysctl.conf:
# vi /etc/sysctl.conf
Append following config directive:
kernel.sysrq=1
Save and close the file. Reload settings:
# sysctl -p

Save and close the file and reboot system to take effect

How do I use the magic SysRq keys in emergency?

You need to use following key combination in order to reboot/halt/sync file system etc:
ALT+SysRq+COMMAND-KEY

The 'SysRq' key is also known as the 'Print Screen' key. COMMAND-KEY can be any one of the following (all keys need to hit simultaneously) :

  • 'b' : Will immediately reboot the system without syncing or unmounting your disks.
  • 'o' : Will shutdown your system off (if configured and supported).
  • 's': Will attempt to sync all mounted filesystems.
  • 'u' : Will attempt to remount all mounted filesystems read-only.
  • 'e' : Send a SIGTERM to all processes, except for init.
  • 'h': Show help, indeed this the one you need to remember.

So whey you need to tell your Linux computer to reboot or when your X server is crashed or you don't see anything going across the screen then just press:

ALT+SysRQ+s : (Press and hold down ALT, then SysRQ (Print Screen) key and press 's') -Will try to syn all mounted system

ALT+SysRQ+r : (Press and hold down ALT, then SysRQ (Print Screen) key and press 'r') -Will reboot the system.

If you wish to shutdown the system instead of reboot then press following key combination:
ALT+SysRQ+o

ipt_sysrq is a new iptables target that allows you to do the same as the magic sysrq key on a keyboard does, but over the network. Sometimes a remote server hangs and only responds to icmp echo request (ping). Every administrator of such machine is very unhappy because (s)he must go there and press the reset button. It takes a long time and it's inconvenient. So use the Network Magic SysRq and you will be able to do more than just pressing a reset button. You can remotely sync disks, remount them read-only, then do a reboot. And everything comfortably and only in a few seconds. Please see Marek Zelem page to enableIP Tables network magic SysRq function.

For more information read official Documentation for sysrq.c version 1.15 stored in /usr/src/linux/Documentation/sysrq.txt and read man page of sysctl, sysctl.conf.