Increase your Linux server Internet speed with TCP BBR congestion control

Posted on in Categories Cloud Computing last updated July 22, 2017

I recently read that TCP BBR has significantly increased throughput and reduced latency for connections on Google’s internal backbone networks and google.com and YouTube Web servers throughput by 4 percent on average globally – and by more than 14 percent in some countries. The TCP BBR patch needs to be applied to the Linux kernel. The first public release of BBR was here, in September 2016. The patch is available to any one to download and install. Another option is using Google Cloud Platform (GCP). GCP by default turned on to use a cutting-edge new congestion control algorithm named TCP BBR.

Requirements

Make sure that your Linux kernel has the following option compiled as either module or inbuilt into the Linux kerne:

  1. CONFIG_TCP_CONG_BBR
  2. CONFIG_NET_SCH_FQ

You must use the Linux kernel version 4.9 or above. On a Debian/Ubuntu Linux type the following grep command/egrep command:
$ grep 'CONFIG_TCP_CONG_BBR' /boot/config-$(uname -r)
$ grep 'CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)
$ egrep 'CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)

Sample outputs:

Fig.01:  Make sure that your Linux kernel has TCP BBR option setup
Fig.01: Make sure that your Linux kernel has TCP BBR option setup

I am using Linux kernel version 4.10.0-28. If above options not found, you need to either compile latest kernel or install the latest version of Linux kernel using the apt-get command/apt command.

How to enable TCP BBR congestion control on Linux

Edit the /etc/sysctl.conf file or create a new file in /etc/sysctl.d/ directory:
$ sudo vi /etc/sysctl.conf
OR
$ sudo vi /etc/sysctl.d/10-custom-kernel-bbr.conf
Append the following two lines:
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr

Save and close the file i.e. exit from the vim/vi text editor by typing :x!. Next you must either reboot the Linux box or reload the changes using the sysctl command:
$ sudo reboot
OR
$ sudo sysctl --system
Sample outputs:

* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-custom.conf ...
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-lxd-inotify.conf ...
fs.inotify.max_user_instances = 1024
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...

You can verify new settings with the following sysctl command. Run:
$ sysctl net.core.default_qdisc
net.core.default_qdisc = fq
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

Test it

In my testing between two long distance Linux server with Gigabit ports connected to the Internet, I was able to bump 250 Mbit/s into 800 Mbit/s. You can use tools such as the wget command to measure bandwidths speed:
$ wget https://your-server-ip/file.iso
I also noticed I was able to push almost 100 Mbit/s for my OpenVPN traffic. Previously I was able to push up to 30-40 Mbit/s only. Overall I am quite satisfied with TCP BBR congestion control option for my Linux box.

References

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on (or read 12 comments/add one below):

12 comment

  1. I have tried that but the download/upload speed is garbage. I was able to download with about 30 MB/sec using kernel 3.x and without TCP BBR activated. Installed kernel 4.x and enabled TCP BBR but the download speed was about 1MB/sec (from the exact same source). So I have decided to remove kernel 4.x and stick to the old 3.x kernel.

      1. Tried on a vps first, CentOS 7, gigabit connection. It might be because of the settings on the dedicated server where the vps was created (for testing purposes only). I will check more anyway.

  2. Is that good for a Raspberry Pi running as a home server with a 60/6 Mbit/s Internet link? Would this improve performance?

  3. So is this server-side only? Will it make any difference if I compiled the kernel on my desktop linux box at hone? Stupid question?

      1. I know you already answered this but, I was reading cerowrt’s article above (the one posted by johnp) and saw the tcp_upload test showing stable bandwidth and more mb/s for bbr vs cubic (the default in several desktop distros)… so wouldn’t this be useful for desktops too?

  4. I am using it with fq_codel instead of fq, isn’t that better?
    Maybe you could try to benchmark with it to see if there is any difference.
    Otherwise, nice article!

Comments are closed.