How to test the network speed/throughput between two Linux servers

Posted on in Categories , , last updated July 25, 2017

I have setup total four servers in cloud based data center. They run either CentOS or Ubuntu Linux. I need to make sure clustered file system performance is reasonable as my servers shared with other users. How do I check the speed between the two Linux servers using command line options for private LAN/VLAN?

You can test the network speed (throughput) between Ubuntu/CentOS/Debian/Fedora or Unix box using the iperf command. Iperf commands show info about bandwidth, delay, jitter, and datagram loss. From the man page:

iperf is a tool for performing network throughput measurements. It can test either TCP or UDP throughput. To perform an iperf test the user must establish both a server (to discard traffic) and a client (to generate traffic).

Here is my sample setup:

Fig.01 Test network speed between two Linux computers
Fig.01 Test network speed between two Linux computers

Step 1 – Installation

You need to install iperf as per your Linux distro version. There are two versions of iperf:

  1. iperf
  2. iperf3

The iperf/iperf3 works on Linux, Unix, Windows, *BSD and other operating systems.

Install iperf on a Ubuntu/Debian Linux server

Type the following apt-get command/[nixmcd name=”apt”] to install iperf on both servers:
$ sudo apt-get update
$ sudo apt-get install iperf

Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  iperf
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 51.0 kB of archives.
After this operation, 184 kB of additional disk space will be used.
Get:1 http://mirrors.linode.com/ubuntu xenial/universe amd64 iperf amd64 2.0.5+dfsg1-2 [51.0 kB]
Fetched 51.0 kB in 0s (4921 kB/s)
Selecting previously unselected package iperf.
(Reading database ... 26108 files and directories currently installed.)
Preparing to unpack .../iperf_2.0.5+dfsg1-2_amd64.deb ...
Unpacking iperf (2.0.5+dfsg1-2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up iperf (2.0.5+dfsg1-2) ...

A note about installing iperf3 on a Debian/Ubuntu Linux

It is also possible to install iperf3 (latest version) as follows:
$ sudo apt-get install iperf3
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libiperf0
The following NEW packages will be installed:
  iperf3 libiperf0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.5 kB of archives.
After this operation, 238 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.linode.com/ubuntu xenial/universe amd64 libiperf0 amd64 3.0.11-1 [50.4 kB]
Get:2 http://mirrors.linode.com/ubuntu xenial/universe amd64 iperf3 amd64 3.0.11-1 [8090 B]
Fetched 58.5 kB in 0s (4796 kB/s)
Selecting previously unselected package libiperf0:amd64.
(Reading database ... 26120 files and directories currently installed.)
Preparing to unpack .../libiperf0_3.0.11-1_amd64.deb ...
Unpacking libiperf0:amd64 (3.0.11-1) ...
Selecting previously unselected package iperf3.
Preparing to unpack .../iperf3_3.0.11-1_amd64.deb ...
Unpacking iperf3 (3.0.11-1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libiperf0:amd64 (3.0.11-1) ...
Setting up iperf3 (3.0.11-1) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...

Install iperf on a Fedora Linux server

Type the following [nixmcd name=”dnf”] to install iperf on both servers:
$ sudo dnf install iperf

A note about installing iperf3 on a CentOS/RHEL/SL/Oracle Linux

Type the following command:
$ sudo dnf install iperf3

Install iperf on a CentOS/RHEL/Oracle/Scientific Linux server

Type the following [nixmcd name=”yum”] to install iperf on both servers (first turn on EPEL repo for CentOS/RHEL):
$ sudo yum install iperf
Sample outputs:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.snu.edu.in
 * epel: mirror01.idc.hinet.net
 * extras: centos.mirror.snu.edu.in
 * updates: centos.mirror.snu.edu.in
Resolving Dependencies
--> Running transaction check
---> Package iperf.x86_64 0:2.0.8-1.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=====================================================================
 Package       Arch           Version             Repository    Size
=====================================================================
Installing:
 iperf         x86_64         2.0.8-1.el7         epel         321 k
 
Transaction Summary
=====================================================================
Install  1 Package
 
Total download size: 321 k
Installed size: 396 k
Is this ok [y/d/N]: y
Downloading packages:
iperf-2.0.8-1.el7.x86_64.rpm                    | 321 kB   00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : iperf-2.0.8-1.el7.x86_64                          1/1 
  Verifying  : iperf-2.0.8-1.el7.x86_64                          1/1 
 
Installed:
  iperf.x86_64 0:2.0.8-1.el7                                         
 
Complete!

A note about installing iperf3 on a CentOS/RHEL/SL/Oracle Linux

Type the following command:
$ sudo yum install iperf3

Step 2 – Start an iperf server on serverA (server mode)

Type the following command to run iperf in server mode:
iperf -s
OR
iperf3 -s
Sample outputs:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------

A note about firewall config

Please note that you must open TCP port 5001 on the serverA. See how to use ufw to open a port on Ubuntu/Debian Linux:
$ sudo ufw allow from 192.168.149.0/24 to 192.168.149.69 port 5001 proto tcp
It is also possible to change the port by passing the -p option as follows (e.g. open and use TCP port 2323):
iperf -s -p 2323

Step 3 – Start an iperf server on serverB (client mode)

Type the following command to run in client mode, connecting to 192.168.149.69:
iperf -c {ip-address-of-serverA-here}
iperf -c {ip-address-of-serverA-here} -p {tcp-port}
iperf -c 192.168.149.69 -p 2323
iperf -c 192.168.149.69

OR
iperf3 -c {ip-address-of-serverA-here}
iperf3 -c {ip-address-of-serverA-here} -p {tcp-port}
iperf3 -c 192.168.149.69 -p 2323
iperf3 -c 192.168.149.69

Sample outputs:

Fig.02: Linux network speed test result
Fig.02: Linux network speed test result

How to use UDP rather than TCP

On the server side:
iperf -s -u
On the client side:
iperf -c 192.168.149.69 -u

See also: How To Install Speedtest-cli On a CentOS / RHEL / Scientific / Fedora Linux To Check Internet Speed From the Bash Shelll

How to set number of parallel client threads to run (client mode options)

Pass the -P option:
iperf -c 192.168.149.69 -P 3
You can also set time in seconds to transmit for (default 10 secs) as follows:
iperf -c 192.168.149.69 -P 3 -t 30
Sample outputs:

------------------------------------------------------------
Client connecting to 192.168.149.69, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.181.40 port 54198 connected with 192.168.149.69 port 5001
[  3] local 192.168.181.40 port 54194 connected with 192.168.149.69 port 5001
[  4] local 192.168.181.40 port 54196 connected with 192.168.149.69 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-30.0 sec  2.26 GBytes   647 Mbits/sec
[  3]  0.0-30.0 sec  2.89 GBytes   828 Mbits/sec
[  4]  0.0-30.5 sec  1.91 GBytes   539 Mbits/sec
[SUM]  0.0-30.5 sec  7.07 GBytes  1.99 Gbits/sec

Here is output from iperf3:
## on the server side ##
iperf3 -s
## on the client side ##
iperf3 -c 192.168.149.69

Sample outputs:

Connecting to host 192.168.149.69, port 5201
[  4] local 192.168.181.40 port 55522 connected to 192.168.149.69 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   235 MBytes  1.97 Gbits/sec   32    609 KBytes       
[  4]   1.00-2.00   sec   230 MBytes  1.93 Gbits/sec   21    670 KBytes       
[  4]   2.00-3.00   sec   159 MBytes  1.33 Gbits/sec   19    635 KBytes       
[  4]   3.00-4.00   sec   235 MBytes  1.97 Gbits/sec    7    686 KBytes       
[  4]   4.00-5.00   sec   229 MBytes  1.92 Gbits/sec   86    757 KBytes       
[  4]   5.00-6.00   sec   206 MBytes  1.73 Gbits/sec    6    537 KBytes       
[  4]   6.00-7.00   sec   210 MBytes  1.76 Gbits/sec    5    556 KBytes       
[  4]   7.00-8.00   sec   235 MBytes  1.97 Gbits/sec   10    617 KBytes       
[  4]   8.00-9.00   sec   245 MBytes  2.06 Gbits/sec   11    670 KBytes       
[  4]   9.00-10.00  sec   234 MBytes  1.96 Gbits/sec   10    713 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.17 GBytes  1.86 Gbits/sec  207             sender
[  4]   0.00-10.00  sec  2.16 GBytes  1.86 Gbits/sec                  receiver

Getting help

Type the following command:
$ man iperf
$ iperf --help

Usage: iperf [-s|-c host] [options]
       iperf [-h|--help] [-v|--version]
 
Client/Server:
  -f, --format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes
  -i, --interval  #        seconds between periodic bandwidth reports
  -l, --len       #[KM]    length of buffer to read or write (default 8 KB)
  -m, --print_mss          print TCP maximum segment size (MTU - TCP/IP header)
  -o, --output    <filename> output the report or error message to this specified file
  -p, --port      #        server port to listen on/connect to
  -u, --udp                use UDP rather than TCP
  -w, --window    #[KM]    TCP window size (socket buffer size)
  -B, --bind      <host>   bind to <host>, an interface or multicast address
  -C, --compatibility      for use with older versions does not sent extra msgs
  -M, --mss       #        set TCP maximum segment size (MTU - 40 bytes)
  -N, --nodelay            set TCP no delay, disabling Nagle's Algorithm
  -V, --IPv6Version        Set the domain to IPv6
 
Server specific:
  -s, --server             run in server mode
  -U, --single_udp         run in single threaded UDP mode
  -D, --daemon             run the server as a daemon
 
Client specific:
  -b, --bandwidth #[KM]    for UDP, bandwidth to send at in bits/sec
                           (default 1 Mbit/sec, implies -u)
  -c, --client    <host>   run in client mode, connecting to <host>
  -d, --dualtest           Do a bidirectional test simultaneously
  -n, --num       #[KM]    number of bytes to transmit (instead of -t)
  -r, --tradeoff           Do a bidirectional test individually
  -t, --time      #        time in seconds to transmit for (default 10 secs)
  -F, --fileinput <name>   input the data to be transmitted from a file
  -I, --stdin              input the data to be transmitted from stdin
  -L, --listenport #       port to receive bidirectional tests back on
  -P, --parallel  #        number of parallel client threads to run
  -T, --ttl       #        time-to-live, for multicast (default 1)
  -Z, --linux-congestion <algo>  set TCP congestion control algorithm (Linux only)
 
Miscellaneous:
  -x, --reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports
  -y, --reportstyle C      report as a Comma-Separated Values
  -h, --help               print this message and quit
  -v, --version            print version information and quit
 
[KM] Indicates options that support a K or M suffix for kilo- or mega-
 
The TCP window size option can be set by the environment variable
TCP_WINDOW_SIZE. Most other options can be set by an environment variable
IPERF_<long option name>, such as IPERF_BANDWIDTH.

See iperf home page for more info.

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on (or read 4 comments/add one below):

4 comment

  1. How about just using nc? On the server:
    nc -vvlnp 5203 >/dev/null
    On the client:
    dd if=/dev/zero bs=1M count=1K | nc -vvn 192.168.149.69 5203
    I see
    Connection to 192.168.149.69 5203 port [tcp/*] succeeded!
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.6777 s, 230 MB/s

    Easy and no need to install iperf tool.

  2. A non-intrusive way of measuring the bandwidth of the link connecting two servers, on symmetric links, is using pings of different sizes, e.g. 64 and 1064 bytes, and measuring the RTT, then divide the difference of ping sizes in bits by the difference of measured RTTs and you get the link bandwidth.

Leave a Comment