How do I find out Linux Resource utilization to detect system bottlenecks?

by on May 12, 2006 · 9 comments· LAST UPDATED November 23, 2007

in

Q. How can I find out Linux Resource utilization using vmstat command? How do I get information about high disk I/O and memory usage?

A. vmstat command reports information about processes, memory, paging, block IO, traps, and cpu activity. However, a real advantage of vmstat command output - is to the point and (concise) easy to read/understand. The output of vmstat command use to help identify system bottlenecks. Please note that Linux vmstat does not count itself as a running process.

Here is an output of vmstat command from my enterprise grade system:
$ vmstat -S M
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1963    607   2359    0    0     0     0    0     1 32  0 68  0

Where,

  • The fist line is nothing but six different categories. The second line gives more information about each category. This second line gives all data you need.
  • -S M: vmstat lets you choose units (k, K, m, M) default is K (1024 bytes) in the default mode. I am using M since this system has over 4 GB memory. Without -M option it will use K as unit

$ vmstat
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0 2485120 621952 2415368  0    0     0     0    0     1 32  0 68  0

Field Description For Vm Mode

(a) procs is the process-related fields are:

  • r: The number of processes waiting for run time.
  • b: The number of processes in uninterruptible sleep.

(b) memory is the memory-related fields are:

  • swpd: the amount of virtual memory used.
  • free: the amount of idle memory.
  • buff: the amount of memory used as buffers.
  • cache: the amount of memory used as cache.

(c) swap is swap-related fields are:

  • si: Amount of memory swapped in from disk (/s).
  • so: Amount of memory swapped to disk (/s).

(d) io is the I/O-related fields are:

  • bi: Blocks received from a block device (blocks/s).
  • bo: Blocks sent to a block device (blocks/s).

(e) system is the system-related fields are:

  • in: The number of interrupts per second, including the clock.
  • cs: The number of context switches per second.

(f) cpu is the CPU-related fields are:

These are percentages of total CPU time.

  • us: Time spent running non-kernel code. (user time, including nice time)
  • sy: Time spent running kernel code. (system time)
  • id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
  • wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

As you see the first output produced gives averages data since the last reboot. Additional reports give information on a sampling period of length delay. You need to sample data using delays i.e. collect data by setting intervals. For example collect data every 2 seconds (or collect data every 2 second 5 times only):
$ vmstat -S M 2
OR
$ vmstat -S M 2 5
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1756    607   2359    0    0     0     0    0     1 32  0 68  0
3  0      0   1756    607   2359    0    0     0     0 1018    65 38  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    64 37  0 63  0
3  0      0   1756    607   2359    0    0     0    20 1018    72 37  0 63  0
3  0      0   1756    607   2359    0    0     0     0 1012    64 37  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    65 38  0 63  0
3  0      0   1995    607   2359    0    0     0     0 1012    62 35  2 63  0
3  0      0   1731    607   2359    0    0     0     0 1012    64 34  3 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    72 38  0 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    63 37  0 63  0

This is what most system administrators do to identify system bottlenecks. I hope all of you find vmstat data is concise and easy to read.

See also:

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!
1 maya August 10, 2006 at 12:29 pm

about kernel programs with coding

2 Martin December 4, 2007 at 4:35 pm

I’m working with a server, wich has a high iowait, have run the vmstat command, and found out there are allways 3-4 processes in procs > b column, which I read is “uninterruptible sleep” status.

My question is, which are the possible reasons why processes may be in that state? and there on, how can I optimize the server/code/db to lower those processes and allow more concurrency?

Thanks!
Martiniano

Download shareware VoIP Recorder

3 Terrell Prude' Jr. February 14, 2008 at 10:23 pm

Thanks for this info. Good article, and handy. Simple and to the point. Just helped me with a flow-capture I/O bottleneck issue (Cisco Flow-Tools) that I was experiencing. Turns out it just takes a few minutes for flow-capture to “catch up” somehow, then CPU “wait” usage goes back down.

4 thebigfatgeek August 31, 2008 at 8:02 am

I have 8 machines running, and want to be able to check the status of all of the machines on the main local machine. How can that be accomplished? I would not mind being able to graph the results?

Thanks for any help

5 sachin April 16, 2013 at 10:00 am

you can go for Nagios monitoring system

http://www.nagios.org/

6 Amy November 10, 2009 at 2:20 pm

Thank you so much! For the descriptions! That was exactly what I needed for a homework and I couldn’t find it anywhere else! :D

7 Nagaraju January 20, 2011 at 5:47 pm

Hi,

Thanks for the updated, its very valuble information.

Regards,
Nagaraju

8 Karthik February 11, 2011 at 9:41 am

Hi.how to find the processes writing on a file??

9 Mariano Alda April 9, 2011 at 9:41 am

login as root and execute:

lsof | grep “/path/to/file/nameoffile”

Comments on this FAQ are closed. If you'd like to continue the discussion on this topic, you can do so at our forum.

Tagged as: , , , , , , , , , , ,

Previous post:

Next post: