HOWTO: Optimize Gigabit Networking in Linux

Ethernet Cable FullLinux Logo Half Even if you have a gigabit networking adapter and a gigabit switch capable of jumbo frames, Linux still uses the default MTU size of 1500. To get something better, you need to configure things by hand.

The reason for this is that the IETF has never standardized anything above 1500. You might very well have gigabit ethernet equipment that either does not have jumbo frame support, or may be very disappointed to find out that “jumbo frame” can be used to describe any packet size between 1500 and 9000.

To make matters worse, not every gigabit ethernet switch handles mixed networking the same. You would think a gigabit switch would guarantee a 1gb connection between two computers with 1gb networking adapters, but under various circumstances, this isn’t always the case. Optimally, it would be best to separate your 100mb and 1gb devices onto two different switches, but this isn’t guaranteed to work.

Now that we have all the caveats out of the way, read on if you want to start optimizing.

First, you’ll need to run ifconfig eth[n] mtu [size] on each machine capable of gigabit networking on your networking segment. Start with an MTU of 9000. If you get the error message SIOCSIFMTU: Invalid argument, work your way down in increments of 1000. Once you find a setting that works, work your away around in halves to find the maximum MTU.

For example:

sudo ifconfig eth0 mtu 9000
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 8000
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7000
sudo ifconfig eth0 mtu 7500
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7300
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7200

In my limited experience, max MTUs are measured in increments of 100.

When you have the MTU for all of your machines, you will want to go with the lowest common denominator. On two machines, one preferably being the one with the lowest MTU, run this command:

sudo ifconfig eth[n] mtu [size]

Now that we have our test machines. it’s time to make sure our switch supports the frame size. Run this command on either machine to access any the other one:

ping -s [MTU-28] -M do [ip-address]

We have to account for the 8-byte ping header and 20-byte TCP/IP stack header, hence subtracting 28 from our determined MTU. This runs the ping command using our determined MTU size and forces the frames to refuse fragmentation. If you get responses, you’re in the clear. If not, either the pinged computer is dropping pings or your switch is refusing the frame size. If its the latter, you’ll have to do a bit more discovery on the packet size, in a fashion similar to what you used to discover the lowest max MTU to begin with.

Once you have an MTU size, you can run sudo ifconfig eth[n] mtu [size] on all the machines to have an immediate effect. If you’re using DHCP, add this line under the appropriate stanza for your network card in /etc/network/interfaces to make changes permanent:

pre-up /sbin/ifconfig $IFACE mtu [size]

If you’re using static addressing, you can get away with

mtu [size]

In my experience, this gives me an increase of network speed from 15MB/s (126mb/s) to 38MB/s (318mb/s). And from my research, this is to be expected. No one has ever reached anywhere near the theoretical 119MB/s (1gb/s) limit of gigabit. Whereas we got a little over half the theoretical limit of 100mb/s (7MB/s or 59mb/s), one can only get a little over one-third of the theoretical limit of gigabit.

UPDATE 03/24/2010: I recently upgraded my desktop, and the Windows drivers for the new motherboard’s onboard NIC (Realtek PCIe GBE) gave options for 2KB – 9KB, and those KB turned out to be the binary definition of kilobyte. Well, the binary definition minus 14 bytes. Not sure what ate them up, but the 7KB setting’s MTU was 7154 bytes not 7168 bytes.

UPDATE 04/17/2010: Portuguese University IP = faculty + students + wifi freeloaders. Only the first should be taken as a professional source…that should be focused on work instead of using university resources for personal matters.