Linux Configure Jumbo Frames to Boost Network Performance / Throughput

last updated in Categories , , , , , ,

Q. Jumbo frames are Ethernet frames with more than 1500 bytes of payload MTU. Does Linux support jumbo frames? If so how do I set frames to 9000 bytes under Linux operating systems?

A. Most modern Linux distros (read as Linux Kernel 2.6.17+) does support frames larger than 1500 bytes. This can improve the performance. First, make sure your network driver supports custom MTU. Second you need to have a compatible gigabit NIC and switch (such as Cisco Catalyst 4000/4500 Switches with Supervisor III or Supervisor IV ) that is jumbo frame clean. If you are not sure about requirements, please refer to your product documentation.

Jumbo frames can reduce server overhead such as a big decrease in CPU usage when transferring larger file. Also you should see some increase in network throughput.

Advertisements

Configure eth0 interface for Jumbo Frames

WARNING! These examples may not work with your setup as it depends upon a compatible network gear such gigabit Ethernet switches and gigabit Ethernet network interface cards

Simply type the following command at a shell prompt to set new MTU (Jumbo Frames):
# ifconfig eth0 mtu 9000

Make changes permanent

Edit the network configuration file for eth0 interface – for example, /etc/sysconfig/network-script/ifcfg-eth0 (CentOS / RHEL / Fedora Linux):
# vi /etc/sysconfig/network-script/ifcfg-eth0
Append the following configuration directive, which specifies the size of the frame in bytes:
MTU 9000

A note about Debian / Ubuntu Linux user:

Debian / Ubuntu Linux user should add MTU=9000 to /etc/network/interfaces configuration file.

Close and save the file. Restart the Interface eth0:
# /etc/init.d/networking restart

To confirm the MTU used between two specific devices. use ip command as follows:
# ip route get {IP-address}
# ip route get 192.168.1.1
Output:

192.168.1.1 dev eth0  src 192.168.1.100 
    cache  mtu 9000 advmss 1460 hoplimit 64

Application Protocol Tunning

You may need to tune the application / network protocol such as NFS and SMB to take advantage of Jumbo Frames.

Updated for accuracy.

ADVERTISEMENTS

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter.

19 comment

  1. Hi, nice article
    just one (1) error; for debian/ubuntu in /etc/network/interfaces, it’s mtu 9000 (mtu(space)9000) *NOT* mtu=9000 (this is only for redhat linuxes)!

    1. THANK YOU!! You saved the whole point of this article, I was looking specifically for jumbo frames for ubuntu

      1. Actually that only works if your statically setting the IP/etc… for that interface. If you’re using dhcp you have to set the MTU after the card is configured otherwise it will be overwritten by whatever the dhcp server passes it. To get around this I setup mine up like this:

        auto eth0
        iface eth0 inet dhcp
        post-up /sbin/ifconfig eth0 mtu 7000

  2. Thanks for the article. I have a second ethernet card that’s a gigabit card and I tried running ifconfig eth1 mtu 9000 and it gives me an error “SIOCSIFMTU: Invalid argument”. Any idea why this is happening? Also, modifying /etc/network/interfaces didn’t seem to help either.

  3. Chris; I just had the same thing happen on Hardy and a quick Google shows that we are not alone. It must be outside the range that our drivers/hw allow for this chipset, just taking a huge swag. I really need this to work too. FWIW:
    processor : 0
    vendor_id : AuthenticAMD
    cpu family : 15
    model : 104
    model name : AMD Turion(tm) 64 X2 Mobile Technology TL-60
    nVidia MCP67 Ethernet
    Linux jeff-laptop 2.6.24-19-generic #1 SMP Wed Jun 18 14:43:41 UTC 2008 i686 GNU/Linux

    Will get to the bottom of this since I just spent the previous weekend rewiring the house with cat6, upgraded all routers, etc….

    Jeff

  4. Follow-on to my last but just got new information. Chris, open a term and do a ‘sudo ethtool eth0’ or whatever your actual device is. Amongst other useful information there was this (and yes I do have a gigabit ethernet card):
    Settings for eth0:
    Supported ports: [ MII ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 100Mb/s

    Sorry if that came out dodgy but the thing to note is my driver only thinks I can do 100M and as such, I would assume it would be hard for the above command to work. Well, there is a new kernel out too so I might as well grab the drivers and make a party of it…

  5. Jeff,

    Thanks for your reply. It’s strange how you have a gigabit card, yet your limit was only 100Mb/s. Perhaps you had another one, eth1? I know mine correctly has a max limit of 1000Mb/s, but somehow I couldn’t get it to work initially. I found out my card (Netgear GA311) had a max MTU of 7500, but setting that didn’t work. When I did a little bit more googling, I found out that someone who had the same card set the MTU to 7200. Once I did that, it worked for me. So I guess your initial theory was right in that the number was out of range of the max MTU.

  6. Hi Vivek,

    What is the downside for setting higher MTU?
    How would I know the proper value for MTU for my server?
    I am using rhel 4 AS (2.6.9-42) , BroadCom Corporation NetXtreme II BCM5708 Gigabit Ethernet network Card and switch is 3Com® Baseline Switch 2948-SFP Plus.

    Jay

  7. Can’t set mtu > 1500 on Ubuntu 9.10. This sucks! I am trying to find tweaks to fool OS but no luck yet. It works in windows very well > 9000kb jumbo size frames!

    sudo ethtool eth0
    gives

    Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Half 1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Half 1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: g
    Wake-on: g
    Current message level: 0x000000ff (255)
    Link detected: no

  10. Help me!!! :)
    I try change my MTU to 9000, but a receive this error (SIOCSIFMTU: Argumento inválido) . however i can change to 7004.

    My configuration:

    [root@xrafa rafael]# ifconfig eth0
    eth0 Link encap:Ethernet Endereço de HW 00:26:9E:21:F2:D3
    inet end.: 10.10.10.106 Bcast:10.10.10.255 Masc:255.255.255.0
    endereço inet6: fe80::226:9eff:fe21:f2d3/64 Escopo:Link
    UP BROADCASTRUNNING MULTICAST MTU:7004 Métrica:1
    RX packets:21212 errors:0 dropped:0 overruns:0 frame:0
    TX packets:15319 errors:0 dropped:0 overruns:0 carrier:0
    colisões:0 txqueuelen:1000
    RX bytes:24128672 (23.0 MiB) TX bytes:2527859 (2.4 MiB)
    IRQ:31 Endereço de E/S:0xa000

    [root@xrafa rafael]# ethtool eth0
    Settings for eth0:
    Supported ports: [ TP MII ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Half 1000baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    1000baseT/Half 1000baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000033 (51)
    Link detected: yes
    [root@xrafa rafael]# ifconfig eth0 mtu 9000
    SIOCSIFMTU: Argumento inválido

    Thanks.
    Rafael

  11. Rafael, virgoptrex, maybe your driver is not supporting yet big MTUs. You can learn the driver you’re using with:
    ethtool -i XXX

    Example:

    root@random# ethtool -i eth1
    driver: igb
    version: 2.1.1
    firmware-version: 1.7-2
    bus-info: 0000:05:00.1

  13. Hey all, I just noticed that a more helpful message is printed in the system log.

    dmesg:
    e1000: eth0: e1000_change_mtu: Jumbo Frames not supported.

    Strangely the Supermicro board I have has dual Ethernet and the secondary interface supports Jumbo frames, but not the primary!

    lspci:
    04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit Ethernet Controller
    05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller

  14. The path in the article:
    # vi /etc/sysconfig/network-script/ifcfg-eth0

    Should be (scriptS, plural for CentOS/rhe):
    # vi /etc/sysconfig/network-scripts/ifcfg-eth0

  17. I also want to configure two servers with a cross-over Cat6 cable, but still un-successful.
    I am testing by sending big packets, as follows:
    # ping -s 8972 192.168.0.20

    #demsg
    tg3 0000:03:00.1: eth1: Link is up at 1000 Mbps, full duplex
    tg3 0000:03:00.1: eth1: Flow control is off for TX and off for RX
    tg3 0000:03:00.1: eth1: EEE is disabled

    Still, have a question? Get help on our forum!