How do I find out Linux Disk utilization?

Posted on in Categories CentOS, Debian Linux, File system, Hardware, Howto, Linux, Monitoring, RedHat/Fedora Linux, Storage, Tips, Troubleshooting, Tuning last updated December 17, 2007

I’ve already written about finding out Linux / UNIX cpu utilization using various tools. You can use same iostat command to find out disk utilization and for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.

iostat syntax for disk utilization report

iostat -d -x interval count

  • -d : Display the device utilization report (d == disk)
  • -x : Display extended statistics including disk utilization
  • interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
  • count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times

Display 3 reports of extended statistics at 5 second intervals for disk

Type the following command:
$ iostat -d -x 5 3
Output:

Linux 2.6.18-53.1.4.el5 (moon.nixcraft.in)   12/17/2007

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               1.10    39.82  3.41 13.59   309.50   427.48    43.36     0.17   10.03   1.03   1.75
sdb               0.20    18.32  1.15  6.08   117.36   195.25    43.22     0.51   71.14   1.26   0.91

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   108.40  1.40 64.40    49.60  1382.40    21.76     0.04    0.67   0.44   2.92
sdb               0.00    37.80  0.00 245.20     0.00  2254.40     9.19    28.91  108.49   1.08  26.36

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    97.01  1.00 57.29    39.92  1234.33    21.86     0.03    0.58   0.50   2.89
sdb               0.00    38.32  0.00 288.42     0.00  2623.55     9.10    32.97  122.30   1.15  33.27

Where,

  • rrqm/s : The number of read requests merged per second that were queued to the hard disk
  • wrqm/s : The number of write requests merged per second that were queued to the hard disk
  • r/s : The number of read requests per second
  • w/s : The number of write requests per second
  • rsec/s : The number of sectors read from the hard disk per second
  • wsec/s : The number of sectors written to the hard disk per second
  • avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
  • avgqu-sz : The average queue length of the requests that were issued to the device
  • await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
  • svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
  • %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

How do I interpret the output result for optimization?

First you need to note down following values from the iostat output:

  1. The average service time (svctm)
  2. Percentage of CPU time during which I/O requests were issued (%util)
  3. See if a hard disk reports consistently high reads/writes (r/s and w/s)

If any one of these are high, you need to take one of the following action:

  • Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
  • Tune software or application or kernel or file system for better disk utilization
  • Use RAID array to spread the file system

For example, from about iostat report it appears that /dev/sdb under load. Hope this information will help you diagnose and optimize disk related issues.

Related: How to find out Linux CPU utilization using vmstat, iostat, mpstat and sar commands.

Please note that command and information discussed here almost applies to any other UNIX like variant.

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on (or read 24 comments/add one below):

24 comment

  1. Please suggest me – system load average is shooting.

    %iowait is high and found with iostat output one disk mentioned below may be the cause.

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
    cciss/c0d0 2269.60 132.60 296.20 77.60 20521.60 1681.60 10260.80 840.80 59.40 4.15 11.18 2.65 99.20

    please suggest

  2. When I enter the command “iostat -d -x 5 3” I do not get any statistics. Here is what I get:

    # iostat -d -x 5 3
    Linux 2.4.33.3 (FILES) 03/01/2008

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

    Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

    Any ideas on what the problem could be? I’m running Slackware 11.0 with the stock kernel.

    Thanks in advance.

  4. Mike – you need to not specify “-d”. Only use -d if you’re going to specify a disk like sda or sdb. Just use the -x interval.

    ie, iostat -x 10

    means it will immediately give one output that represents from boot to now, and then every 10 seconds it will output a line that represents the last 10 seconds, and it will repeat forever.

    You can also take this into excel and graph it, paying attention to the 4-5 columns the author mentioned.

  5. good artice!!!!
    one question, Please tell what is the use of the below specified files.
    /sys/block/sda/queue/iosched/queued
    /sys/block/sda/queue/nr_requests
    /sys/block/sda/device/queue_depth

  6. Hi!

    Thanks for the information, very useful.

    I have one question – what constitutes “high” for r/s & w/s? Is over 10 high? Over 100, 1000?

    I suspect that a hard disk is slowing down on me – whenever there is an I/O intensive operation (swapping, download, starting Firefox or OOo, things like that), everything slows to a crawl, including process switching.

    Is there a way to find out from iostat what bandwidth the disk is giving me? What is the real latency of a disk?

    Thanks!
    Dave.

  7. Great article. Thanks!

    I too would like to know what is “high”

    Also, how do you follow on from this. I have had massive IO wait issues at times and tracking down the process causing the issue is very hard. Any tips in this direction?

    Thanks

  9. Great article. Just a quick note. On one of our older Debian servers we had to install sysstats package to use iostat.

  10. Hi,

    My company uses a VPS hosted in softlayer. Anytime Irun iostat, I hadly get any of these data. Could it be a problem with the VPS?
    The sample of the command output for iostat -d -x 5 3 is as below:

    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

    Please I will like to know if there is a way around this. Thanks

  12. iostat man page :
    The average service time (svctm field) value is meaningless, as I/O statistics are calculated at block level, and we don’t know when the disk driver starts to process a request. For this reason, this field will be removed in a future sysstat version.

  14. Hey What are the statistics for high – Just information. Need to be more specific.
    If you can not evaluate the information and reach to the conclusion then its of no use.

    Thanks

  15. Here’s how you get the information filtered

    iostat -x 5 3 | sed "s: +: :g" | cut -d ' ' -f 1,4,5,13,14  | sed "s: :t:g"

    you would have to wait the full 15 seconds before you see any data.
    So this is an even nicer way.

    for n in {1..3}; do iostat -x 1 1 | sed "s: +: :g" | cut -d ' ' -f 1,4,5,13,14  | sed "s: :t:g" | tail -n +2 && sleep 5; done

    🙂

  16. Hi,

    when i try to capture the disk i/o using iostat in linux server i was getting the below error.
    iostat
    If ‘iostat’ is not a typo you can run the following command to lookup the package that contains the binary:
    command-not-found iostat
    bash: iostat: command not found

    please tel me the alternative command for iostat in Linux.
    i have tried iotop,it was not working.

    Brs,
    Ravi

  17. [email protected] ~ $ iotop -d -x 5 10
    Usage: /usr/lib/python-exec/python3.3/iotop [OPTIONS]

    DISK READ and DISK WRITE are the block I/O bandwidth used during the sampling
    period. SWAPIN and IO are the percentages of time the thread spent respectively
    while swapping in and waiting on I/O more generally. PRIO is the I/O priority at
    which the thread is running (set using the ionice command).

    Controls: left and right arrows to change the sorting column, r to invert the
    sorting order, o to toggle the –only option, p to toggle the –processes
    option, a to toggle the –accumulated option, i to change I/O priority, q to
    quit, any other key to force a refresh.

    iotop: error: option -d: invalid floating-point value: ‘-x’

    Take your bad advice and fuck off my internet.

  18. I use the -d -p options. Check out the man page for iostat as there are several more important options available. I use iostat to keep track of disk writes on my Raspberry Pi to make sure I am not over using my SD card. I have an external USB HDD and have /var and most other folders that get a lot of access there. I run iostat as a cron job every AM and append to a file iostat.txt. This lets me look at the number of writes from day to day and the overall number of writes from boot time. Very helpful for tuning the system.

    Thanks

Comments are closed.

Tagged as: Tags , , , , , , , , , , , , , , ,