Linux Tune Network Stack (Buffers Size) To Increase Networking Performance

I‘ve two servers located in two different data center. Both server deals with a lot of concurrent large file transfers. But network performance is very poor for large files and performance degradation take place with a large files. How do I tune TCP under Linux to solve this problem?

By default the Linux network stack is not configured for high speed large file transfer across WAN links. This is done to save memory resources. You can easily tune Linux network stack by increasing network buffers size for high-speed networks that connect server systems to handle more network packets.

The default maximum Linux TCP buffer sizes are way too small. TCP memory is calculated automatically based on system memory; you can find the actual values by typing the following commands:
$ cat /proc/sys/net/ipv4/tcp_mem
The default and maximum amount for the receive socket memory:
$ cat /proc/sys/net/core/rmem_default
$ cat /proc/sys/net/core/rmem_max

The default and maximum amount for the send socket memory:
$ cat /proc/sys/net/core/wmem_default
$ cat /proc/sys/net/core/wmem_max

The maximum amount of option memory buffers:
$ cat /proc/sys/net/core/optmem_max

Tune values

Set the max OS send buffer size (wmem) and receive buffer size (rmem) to 12 MB for queues on all protocols. In other words set the amount of memory that is allocated for each TCP socket when it is opened or created while transferring files:

WARNING! The default value of rmem_max and wmem_max is about 128 KB in most Linux distributions, which may be enough for a low-latency general purpose network environment or for apps such as DNS / Web server. However, if the latency is large, the default size might be too small. Please note that the following settings going to increase memory usage on your server.

# echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
# echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf

You also need to set minimum size, initial size, and maximum size in bytes:
# echo 'net.ipv4.tcp_rmem= 10240 87380 12582912' >> /etc/sysctl.conf
# echo 'net.ipv4.tcp_wmem= 10240 87380 12582912' >> /etc/sysctl.conf

Turn on window scaling which can be an option to enlarge the transfer window:
# echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
Enable timestamps as defined in RFC1323:
# echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
Enable select acknowledgments:
# echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
By default, TCP saves various connection metrics in the route cache when the connection closes, so that connections established in the near future can use these to set initial conditions. Usually, this increases overall performance, but may sometimes cause performance degradation. If set, TCP will not cache metrics on closing connections.
# echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf
Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them.
# echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
Now reload the changes:
# sysctl -p
Use tcpdump to view changes for eth0:
# tcpdump -ni eth0

Recommend readings:

🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
42 comments… add one
  • Arade May 11, 2012 @ 18:23

    yeah i think it a buffer over bloat , we should not increase the buffer size too much , we should consider the bottleneck on our link while doing such stuff

  • richard May 22, 2012 @ 13:24

    what is TXqueueing actually
    is it recommended on a low latency needed server
    –> ifconfig eth0 txqueuelen 3000 <–

  • Kido Jul 15, 2012 @ 23:54

    after modify,how can i check if the server get better speed?

  • Andrés Chandía Sep 8, 2012 @ 11:52

    Hi there, im not sure if I’m suffering this problem, so not sure if this a suitable solution for me. I better explain my case:
    I have a little office network, where I have a file, printer, download and router server. I mean I use it as storage (NFS). By a web interface users can access to a mldonkey where they look for things they need to download. And all the network goes to the internet through it.
    I recently increased the memory fom 1G to 3G, and added a 2TB hard disk for storage. Since this moment on some workstations, specially the old ones have problems accesing the files stored at the server, if a user needs to watch a video, while watching it, the video stops some times and after a while starts again. The acces to folders with huge amount of data makes the nautilus window to turn gray for a while before showing the content, etc….
    These kind of things didn’t happen with 1Gb RAM. I have to say that I have read that large disks may affect performance of the machine. But I suspect my problem is related to ram addition and the network.
    I hope you can help me.
    Thanks a lot.

  • vlassius Nov 27, 2012 @ 22:05

    Hi, i want to share something that i spent a lot of time to solve and the solution is against any advice found on internet on doing it.

    Ambient:
    Mini network, 3 computers @ 300Mb wifi, where one is a server (proxy and storage).
    To Internet (on server) using wifi direct connected adapter.

    Effect:
    On file transfer between server and computer, when gets about 50% of bandwidth, the connection with internet slows down to zero bytes transfer.
    (The internet adapter is not used to file transfer, its another adapter)
    The load of the server is bellow 20%
    Server: Fedora Linux kernel 3.6.7-4.fc17.x86_64
    Machine 1: Windows 7
    Machine 2: Windows 8
    This issue has about 18 months, i mean, the time that the network was mounted.
    As a experimented developer, I’ve tried really a lot of things and i got really annoyed with this problem and I’ve found a lot of problems related to this found on internet (while searching for a solution). (I was thinking in go to try changes in kernel source, but i was not psychologically prepared to this )

    Solution:
    The solution was simply increase the network parameters to a very hight values (in memory parameters).
    This is NOT proposed to do in a server without a extensive test but, worked like a charm to me. Every network communication had a surprised increase of performance and no more problems.

    So, the working parameters are:

    # adjust memory to 1.677MB – endless memory :-)
    net.core.rmem_max=1677721600
    net.core.rmem_default=167772160
    net.core.wmem_max=1677721600
    net.core.wmem_default=167772160
    net.core.optmem_max= 2048000

    # set minimum size, initial size, and maximum size in bytes
    net.ipv4.tcp_rmem= 1024000 8738000 1677721600
    net.ipv4.tcp_wmem= 1024000 8738000 1677721600
    net.ipv4.tcp_mem= 1024000 8738000 1677721600
    net.ipv4.udp_mem= 1024000 8738000 1677721600

    The server is now allocating about 500MB to network buffer and everything is running very well.

    Regards
    vlassius

  • TT Mar 1, 2013 @ 2:23

    Servers to like RAM for caching whatever rubbish you access on them.
    Be sure to have a UPS and do proper shutdowns every or else you’re looking at serious corruption and data loss.
    Been there, done it, now I have a UPS.

  • TT Mar 1, 2013 @ 2:24

    There’s a typo there.
    Servers to like RAM -> Servers do like RAM

  • Floyd Carp Mar 13, 2013 @ 14:11

    Works Great, THANKS!

  • chekkiligili Jan 28, 2014 @ 5:01

    Hi,

    Is there any way to reduce the network latency? Want to verify a specific issue, when there is low latency we are get into issue. So to complete testing phase we need to simulate that scenario.

    Please advice

  • Anton Feb 16, 2014 @ 8:20

    Thanks for the article. Should I tune all tcp buffer settings on both sides (i.e. client and server)?

  • max Jul 17, 2014 @ 16:42

    Big thanks. It fixes my server performance issues, such as, TCP Dup Ack and TCP out-of-order problems.

  • Sepahrad Salour Mar 1, 2015 @ 16:52

    That was great and simple article… The exact way to calculate TCP Buff size:

    TCP Buffer Size= *
    eg.

    20 (ms) * 100 (Mbps) = 0.02 * 100 / 8 * 1024 = 256 KB

  • Kanagavelu Jun 10, 2015 @ 6:12

    Does this required restart of the linux?

  • Chetan Jul 8, 2015 @ 14:43

    How to do the same process for UDP?
    I am using UDP for trasfering RTSP stream from server to client over a WiFi. I am also getting a stream on my Client laptop but then after 1 minute the whole network is getting collapsed and it says “No buffer space available” on Server PC.
    As there is no network, RTSP stream can not be available.
    Please suggest me some solution.

  • Zohar Sabari Aug 19, 2015 @ 16:50

    There is an error in net.ipv4.tcp_mem, the value is measured in pages sizes (4k) not in single bytes. I haven’t checked for the udp counterpart.

  • Cauliflower Feb 23, 2016 @ 20:26

    Useful looking article. Our situation is that we have 10Gb links to two Linux servers, we want them to talk to each other across a 10Gb network; it’s all within the same AS. Our users want to send large amounts of data across the network periodically (eg every day). The tool they are currently using is rsync, and test results are pretty poor. There seem to be two major factors affecting this (taking contention across the shared network for granted) – OpenSSH’s own windowing (for which we are looking at HPN-SSH as a solution), and spikes in traffic which cause TCP sawtoothing . Would you think your solution here would be suitable for our uses?

    We are looking to address the contention issues on the backbone with QoS rules.

    • Helge Oct 14, 2016 @ 8:18

      Use GridFTP

  • Jan Syren Feb 21, 2017 @ 7:52

    Hello and thanks for a great article.

    I wonder if you have any experience with the tc qdisc netem functionality?
    I am trying to set very specific delays in the microsecond range, and I need the spread around the set time to be quite accurate. But as it is at the moment we have a bit higher percentage within the delay we have set, and then a long falloff curve with longer delays. So it seems as if there is a buffer that doesn’t empty properly. Is there a way to actually monitor the buffers, send out usage statistics to a file etc..

    We are currently using an Emulex oce14102 and the delay is set on a bridge between the two ports of the card. Link speed is 10gbps. If we don’t use netem the bridge itself has good statistics, and when we are sending in just one direction it is also better but you start noticing the falloff curve.

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.