I’ve already written about finding out Linux / UNIX cpu utilization using various tools. You can use same iostat command to find out disk utilization and for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.
iostat syntax for disk utilization report
iostat -d -x interval count
- -d : Display the device utilization report (d == disk)
- -x : Display extended statistics including disk utilization
- interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
- count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times
Display 3 reports of extended statistics at 5 second intervals for disk
Type the following command:
$ iostat -d -x 5 3
Output:
Linux 2.6.18-53.1.4.el5 (moon.nixcraft.in) 12/17/2007 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 1.10 39.82 3.41 13.59 309.50 427.48 43.36 0.17 10.03 1.03 1.75 sdb 0.20 18.32 1.15 6.08 117.36 195.25 43.22 0.51 71.14 1.26 0.91 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 108.40 1.40 64.40 49.60 1382.40 21.76 0.04 0.67 0.44 2.92 sdb 0.00 37.80 0.00 245.20 0.00 2254.40 9.19 28.91 108.49 1.08 26.36 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 97.01 1.00 57.29 39.92 1234.33 21.86 0.03 0.58 0.50 2.89 sdb 0.00 38.32 0.00 288.42 0.00 2623.55 9.10 32.97 122.30 1.15 33.27
Where,
- rrqm/s : The number of read requests merged per second that were queued to the hard disk
- wrqm/s : The number of write requests merged per second that were queued to the hard disk
- r/s : The number of read requests per second
- w/s : The number of write requests per second
- rsec/s : The number of sectors read from the hard disk per second
- wsec/s : The number of sectors written to the hard disk per second
- avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
- avgqu-sz : The average queue length of the requests that were issued to the device
- await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
- svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
- %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.
How do I interpret the output result for optimization?
First you need to note down following values from the iostat output:
- The average service time (svctm)
- Percentage of CPU time during which I/O requests were issued (%util)
- See if a hard disk reports consistently high reads/writes (r/s and w/s)
If any one of these are high, you need to take one of the following action:
- Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
- Tune software or application or kernel or file system for better disk utilization
- Use RAID array to spread the file system
For example, from about iostat report it appears that /dev/sdb under load. Hope this information will help you diagnose and optimize disk related issues.
Related: How to find out Linux CPU utilization using vmstat, iostat, mpstat and sar commands.
Please note that command and information discussed here almost applies to any other UNIX like variant.
24 comment
Please suggest me – system load average is shooting.
%iowait is high and found with iostat output one disk mentioned below may be the cause.
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 2269.60 132.60 296.20 77.60 20521.60 1681.60 10260.80 840.80 59.40 4.15 11.18 2.65 99.20
please suggest
When I enter the command “iostat -d -x 5 3” I do not get any statistics. Here is what I get:
# iostat -d -x 5 3
Linux 2.4.33.3 (FILES) 03/01/2008
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
Any ideas on what the problem could be? I’m running Slackware 11.0 with the stock kernel.
Thanks in advance.
where x = device
example : iostat -x -d sda 5
Another good article… thanks for sharing,
Mike – you need to not specify “-d”. Only use -d if you’re going to specify a disk like sda or sdb. Just use the -x interval.
ie, iostat -x 10
means it will immediately give one output that represents from boot to now, and then every 10 seconds it will output a line that represents the last 10 seconds, and it will repeat forever.
You can also take this into excel and graph it, paying attention to the 4-5 columns the author mentioned.
good artice!!!!
one question, Please tell what is the use of the below specified files.
/sys/block/sda/queue/iosched/queued
/sys/block/sda/queue/nr_requests
/sys/block/sda/device/queue_depth
Hi!
Thanks for the information, very useful.
I have one question – what constitutes “high” for r/s & w/s? Is over 10 high? Over 100, 1000?
I suspect that a hard disk is slowing down on me – whenever there is an I/O intensive operation (swapping, download, starting Firefox or OOo, things like that), everything slows to a crawl, including process switching.
Is there a way to find out from iostat what bandwidth the disk is giving me? What is the real latency of a disk?
Thanks!
Dave.
Great article. Thanks!
I too would like to know what is “high”
Also, how do you follow on from this. I have had massive IO wait issues at times and tracking down the process causing the issue is very hard. Any tips in this direction?
Thanks
I should use my gravatar email address 🙂
Great article. Just a quick note. On one of our older Debian servers we had to install sysstats package to use iostat.
Hi,
My company uses a VPS hosted in softlayer. Anytime Irun iostat, I hadly get any of these data. Could it be a problem with the VPS?
The sample of the command output for iostat -d -x 5 3 is as below:
Please I will like to know if there is a way around this. Thanks
Very useful… thanks!
iostat man page :
The average service time (svctm field) value is meaningless, as I/O statistics are calculated at block level, and we don’t know when the disk driver starts to process a request. For this reason, this field will be removed in a future sysstat version.
Please let me know ..how to identify what is causing high disk utilization ??
Hey What are the statistics for high – Just information. Need to be more specific.
If you can not evaluate the information and reach to the conclusion then its of no use.
Thanks
Here’s how you get the information filtered
you would have to wait the full 15 seconds before you see any data.
So this is an even nicer way.
for n in {1..3}; do iostat -x 1 1 | sed "s: +: :g" | cut -d ' ' -f 1,4,5,13,14 | sed "s: :t:g" | tail -n +2 && sleep 5; done🙂
Hi,
when i try to capture the disk i/o using iostat in linux server i was getting the below error.
iostat
If ‘iostat’ is not a typo you can run the following command to lookup the package that contains the binary:
command-not-found iostat
bash: iostat: command not found
please tel me the alternative command for iostat in Linux.
i have tried iotop,it was not working.
Brs,
Ravi
Debian/Ubuntu: apt-get install sysstat
CentOS: yum install sysstat
yum install iostat
[email protected] ~ $ iotop -d -x 5 10
Usage: /usr/lib/python-exec/python3.3/iotop [OPTIONS]
DISK READ and DISK WRITE are the block I/O bandwidth used during the sampling
period. SWAPIN and IO are the percentages of time the thread spent respectively
while swapping in and waiting on I/O more generally. PRIO is the I/O priority at
which the thread is running (set using the ionice command).
Controls: left and right arrows to change the sorting column, r to invert the
sorting order, o to toggle the –only option, p to toggle the –processes
option, a to toggle the –accumulated option, i to change I/O priority, q to
quit, any other key to force a refresh.
iotop: error: option -d: invalid floating-point value: ‘-x’
Take your bad advice and fuck off my internet.
Just use “iotop” , defaults are fine… or read the manual: “man iotop”
I use the -d -p options. Check out the man page for iostat as there are several more important options available. I use iostat to keep track of disk writes on my Raspberry Pi to make sure I am not over using my SD card. I have an external USB HDD and have /var and most other folders that get a lot of access there. I run iostat as a cron job every AM and append to a file iostat.txt. This lets me look at the number of writes from day to day and the overall number of writes from boot time. Very helpful for tuning the system.
Thanks
how can I IOSTAT for particular mount like /dev/XXX/YY
Could you explain me how to check and optimize Disk latency/transfer times