How to find disk I/O latency with ioping monitoring tool on Linux

Posted on in Categories , , , , last updated September 21, 2017

How do I monitor disk I/O latency in real time on Linux, FreeBSD, OpenBSD, and MacOS operating system?

You can use ioping tool to monitor I/O latency in real time. It shows disk latency in the same way as ping command shows network latency on Linux or Unix-like system.

Installation

Install as per your operating system.

Install ioping on a Debian/Ubuntu/Mint Linux

Type the following apt-get command/apt command:
$ sudo apt-get install ioping

Install ioping on an Arch Linux

Type the following pacman command:
$ sudo pacman -S ioping

Install ioping on a Fedora Linux

Type the following dnf command:
$ sudo dnf install ioping

Install ioping on a macOS

Type the following brew command:
$ brew install ioping
Sample outputs:

==> Downloading https://homebrew.bintray.com/bottles/ioping-1.0.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring ioping-1.0.sierra.bottle.tar.gz
*  /usr/local/Cellar/ioping/1.0: 7 files, 79.4KB

Install ioping on a FreeBSD

Type the following pkg command:
# pkg install ioping
Sample outputs:

Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB   6.0MB/s    00:01    
Processing entries: 100%
FreeBSD repository update completed. 26484 packages processed.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):
 
New packages to be INSTALLED:
	ioping: 1.0
 
Number of packages to be installed: 1
 
23 KiB to be downloaded.
 
Proceed with this action? [y/N]: y
[1/1] Fetching ioping-1.0.txz: 100%   23 KiB  23.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing ioping-1.0...
Extracting ioping-1.0: 100%

Install ioping on a OpenBSD

Type the following pkg_add command:
# pkg_add ioping
Sample outputs:

quirks-2.304 signed on 2017-04-02T15:01:33Z
ioping-1.0: ok

How do I use ioping?

Warning: Use the ioping command with care. Wrong options can destroy your data. Read man page for more info.

The syntax is:
ioping /tmp/
ioping /dev/sda1
ioping [options] device/dir/file

See disk I/O latency using the default values and the current directory

$ ioping .
To stop after 12 count, pass the -c option:
$ ioping -c 12 .
Sample outputs from OpenBSD running on Linode server:

ioping: non-cached I/O not supported by this platform
ioping: you can use write I/O to get reliable results
4 KiB <<< . (ffs /dev/sd0k): request=1 time=15.9 us (warmup)
4 KiB <<< . (ffs /dev/sd0k): request=2 time=22.3 us
4 KiB <<< . (ffs /dev/sd0k): request=3 time=20 us
4 KiB <<< . (ffs /dev/sd0k): request=4 time=19.9 us
4 KiB <<< . (ffs /dev/sd0k): request=5 time=21.9 us
4 KiB <<< . (ffs /dev/sd0k): request=6 time=25.7 us
4 KiB <<< . (ffs /dev/sd0k): request=7 time=24.8 us
4 KiB <<< . (ffs /dev/sd0k): request=8 time=25.3 us
4 KiB <<< . (ffs /dev/sd0k): request=9 time=85.5 us (slow)
4 KiB <<< . (ffs /dev/sd0k): request=10 time=23.9 us
4 KiB <<< . (ffs /dev/sd0k): request=11 time=38.2 us
4 KiB <<< . (ffs /dev/sd0k): request=12 time=30.8 us
 
--- . (ffs /dev/sd0k) ioping statistics ---
11 requests completed in 338.3 us, 44 KiB read, 32.5 k iops, 127.0 MiB/s
generated 12 requests in 11.0 s, 48 KiB, 1 iops, 4.36 KiB/s
min/avg/max/mdev = 19.9 us / 30.8 us / 85.5 us / 18.0 u

To see disk seek rate

$ sudo ioping -R /dev/sda

To see disk sequential speed (MiB/s)

$ sudo ioping -RL /dev/rdisk1
Sample outputs:

Fig.01: Finding disk seek rate and disk sequential speed
Fig.01: Finding disk seek rate and disk sequential speed

How to set interval between requests

The default is 1 second. To change pass the -i {interval} option. Set 3 seconds:
$ ioping -i 3 .
$ ioping -i 3 /dev/sdb

How do I use asynchronous I/O?

Pass the -A option:
# ioping -c 6 -A /dev/vda1
Sample outputs:

4 KiB from /dev/vda1 (block device 20.0 GiB): request=1 time=458 us
4 KiB from /dev/vda1 (block device 20.0 GiB): request=2 time=226 us
4 KiB from /dev/vda1 (block device 20.0 GiB): request=3 time=477 us
4 KiB from /dev/vda1 (block device 20.0 GiB): request=4 time=412 us
4 KiB from /dev/vda1 (block device 20.0 GiB): request=5 time=381 us
4 KiB from /dev/vda1 (block device 20.0 GiB): request=6 time=356 us

--- /dev/vda1 (block device 20.0 GiB) ioping statistics ---
6 requests completed in 5.00 s, 2.60 k iops, 10.1 MiB/s
min/avg/max/mdev = 226 us / 385 us / 477 us / 82 us

How to use cached I/O (no cache flush/drop)

Pass the -C option:
$ ioping -c 6 -C /dev/sdb1
# ioping -c 6 -C .

Sample outputs from FreeBSD server hosted at Linode:

4 KiB <<< . (zfs zroot/ROOT/default): request=1 time=30.8 us (warmup)
4 KiB <<< . (zfs zroot/ROOT/default): request=2 time=45.3 us
4 KiB <<< . (zfs zroot/ROOT/default): request=3 time=40.9 us
4 KiB <<< . (zfs zroot/ROOT/default): request=4 time=57.1 us
4 KiB <<< . (zfs zroot/ROOT/default): request=5 time=47.5 us
4 KiB <<< . (zfs zroot/ROOT/default): request=6 time=52.8 us
 
--- . (zfs zroot/ROOT/default) ioping statistics ---
5 requests completed in 243.6 us, 20 KiB read, 20.5 k iops, 80.2 MiB/s
generated 6 requests in 5.06 s, 24 KiB, 1 iops, 4.74 KiB/s
min/avg/max/mdev = 40.9 us / 48.7 us / 57.1 us / 5.65 us

How to use direct I/O (O_DIRECT)

Pass the -D option:
# ioping -c 6 -D /tmp
Sample outputs from FreeBSD+ZFS based fs:

4 KiB <<< /tmp (zfs zroot/tmp): request=1 time=19.9 us (warmup)
4 KiB <<< /tmp (zfs zroot/tmp): request=2 time=46.9 us
4 KiB <<< /tmp (zfs zroot/tmp): request=3 time=87.4 us
4 KiB <<< /tmp (zfs zroot/tmp): request=4 time=49.7 us
4 KiB <<< /tmp (zfs zroot/tmp): request=5 time=60.1 us
4 KiB <<< /tmp (zfs zroot/tmp): request=6 time=50.5 us
 
--- /tmp (zfs zroot/tmp) ioping statistics ---
5 requests completed in 294.6 us, 20 KiB read, 17.0 k iops, 66.3 MiB/s
generated 6 requests in 5.06 s, 24 KiB, 1 iops, 4.74 KiB/s
min/avg/max/mdev = 46.9 us / 58.9 us / 87.4 us / 14.9 us

How to use write I/O (please read manpage)

Pass the -W option. From the man page:

Safe for directory target. Write I/O gives more reliable results for systems where non-cached reads are not supported or cached at some level. Might be *DANGEROUS* for file/device: it will shred your data. In this case should be repeated tree times (-WWW).

$ ioping -c 6 -W /tmp
Sample outputs from OpenBSD sever at Linode:

4 KiB >>> /tmp (ffs /dev/sd0d): request=1 time=363.9 us (warmup)
4 KiB >>> /tmp (ffs /dev/sd0d): request=2 time=684.4 us
4 KiB >>> /tmp (ffs /dev/sd0d): request=3 time=6.07 ms
4 KiB >>> /tmp (ffs /dev/sd0d): request=4 time=2.85 ms
4 KiB >>> /tmp (ffs /dev/sd0d): request=5 time=7.98 ms
4 KiB >>> /tmp (ffs /dev/sd0d): request=6 time=1.33 ms
 
--- /tmp (ffs /dev/sd0d) ioping statistics ---
5 requests completed in 18.9 ms, 20 KiB written, 264 iops, 1.03 MiB/s
generated 6 requests in 5.01 s, 24 KiB, 1 iops, 4.79 KiB/s
min/avg/max/mdev = 684.4 us / 3.78 ms / 7.98 ms / 2.80 ms

How to use read-write ping-pong mode

Pass the -G option:
# ioping -c 10 -G /tmp
Sample outputs Google cloud VM:

4 KiB >>> /tmp (ext4 /dev/sda1): request=1 time=614.4 us (warmup)
4 KiB <<< /tmp (ext4 /dev/sda1): request=2 time=2.11 ms
4 KiB >>> /tmp (ext4 /dev/sda1): request=3 time=1.05 ms
4 KiB <<< /tmp (ext4 /dev/sda1): request=4 time=1.89 ms
4 KiB >>> /tmp (ext4 /dev/sda1): request=5 time=965.0 us
4 KiB <<< /tmp (ext4 /dev/sda1): request=6 time=908.3 us
4 KiB >>> /tmp (ext4 /dev/sda1): request=7 time=1.17 ms
4 KiB <<< /tmp (ext4 /dev/sda1): request=8 time=816.1 us (fast)
4 KiB >>> /tmp (ext4 /dev/sda1): request=9 time=1.14 ms
4 KiB <<< /tmp (ext4 /dev/sda1): request=10 time=2.75 ms (slow)
 
--- /tmp (ext4 /dev/sda1) ioping statistics ---
9 requests completed in 12.8 ms, 36 KiB, 703 iops, 2.75 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 816.1 us / 1.42 ms / 2.75 ms / 630.4 us

How to use use sync I/O (O_SYNC)

Pass the -Y option:
# ioping -c 10 -Y /dev/sdb
Sample outputs from Google compute VM:

4 KiB <<< /dev/sdb (block device 200 GiB): request=1 time=237.5 us (warmup)
4 KiB <<< /dev/sdb (block device 200 GiB): request=2 time=245.9 us
4 KiB <<< /dev/sdb (block device 200 GiB): request=3 time=237.1 us
4 KiB <<< /dev/sdb (block device 200 GiB): request=4 time=256.0 us
4 KiB <<< /dev/sdb (block device 200 GiB): request=5 time=251.6 us
4 KiB <<< /dev/sdb (block device 200 GiB): request=6 time=14.5 ms
4 KiB <<< /dev/sdb (block device 200 GiB): request=7 time=252.6 us (fast)
4 KiB <<< /dev/sdb (block device 200 GiB): request=8 time=318.4 us (fast)
4 KiB <<< /dev/sdb (block device 200 GiB): request=9 time=260.5 us (fast)
4 KiB <<< /dev/sdb (block device 200 GiB): request=10 time=277.2 us (fast)
 
--- /dev/sdb (block device 200 GiB) ioping statistics ---
9 requests completed in 16.6 ms, 36 KiB read, 541 iops, 2.12 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/avg/max/mdev = 237.1 us / 1.85 ms / 14.5 ms / 4.48 ms

How to use use data sync I/O (O_DSYNC)

Pass the -y option:
$ ioping -c 10 -y .
Sample outputs from macos:

Fig.02: Use data sync I/O (see O_DSYNC in open(2)).
Fig.02: Use data sync I/O (see O_DSYNC in open(2)).

Getting help

Read man page:
$ man ioping
OR
$ ioping -h
Sample outputs:

      -c <count>      stop after <count> requests
      -i <interval>   interval between requests (1s)
      -l <speed>      speed limit in bytes per second
      -t <time>       minimal valid request time (0us)
      -T <time>       maximum valid request time
      -s <size>       request size (4k)
      -S <wsize>      working set size (1m)
      -o <offset>     working set offset (0)
      -w <deadline>   stop after <deadline> time passed
      -p <period>     print raw statistics for every <period> requests
      -P <period>     print raw statistics for every <period> in time
      -A              use asynchronous I/O
      -C              use cached I/O (no cache flush/drop)
      -B              print final statistics in raw format
      -D              use direct I/O (O_DIRECT)
      -R              seek rate test
      -L              use sequential operations
      -W              use write I/O (please read manpage)
      -G              read-write ping-pong mode
      -Y              use sync I/O (O_SYNC)
      -y              use data sync I/O (O_DSYNC)
      -k              keep and reuse temporary file (ioping.tmp)
      -q              suppress human-readable output
      -h              display this message and exit
      -v              display version and exit

See ioping home page for more info.

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on (or read 1 comments/add one below):

1 comment

Leave a Comment