HOWTO: Optimize Gigabit Networking in Linux
May 20, 2009 at 3:24 am | In Linux, Networking | 2 Comments
Even if you have a gigabit networking adapter and a gigabit switch capable of jumbo frames, Linux still uses the default MTU size of 1500. To get something better, you need to configure things by hand.
The reason for this is that the IETF has never standardized anything above 1500. You might very well have gigabit ethernet equipment that either does not have jumbo frame support, or may be very disappointed to find out that “jumbo frame” can be used to describe any packet size between 1500 and 9000.
To make matters worse, not every gigabit ethernet switch handles mixed networking the same. You would think a gigabit switch would guarantee a 1gb connection between two computers with 1gb networking adapters, but under various circumstances, this isn’t always the case. Optimally, it would be best to separate your 100mb and 1gb devices onto two different switches, but this isn’t guaranteed to work.
Now that we have all the caveats out of the way, read on if you want to start optimizing.
First, you’ll need to run ifconfig eth[n] mtu [size] on each machine capable of gigabit networking on your networking segment. Start with an MTU of 9000. If you get the error message SIOCSIFMTU: Invalid argument, work your way down in increments of 1000. Once you find a setting that works, work your away around in halves to find the maximum MTU.
For example:
sudo ifconfig eth0 mtu 9000
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 8000
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7000
sudo ifconfig eth0 mtu 7500
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7300
SIOCSIFMTU: Invalid argument
sudo ifconfig eth0 mtu 7200
In my limited experience, max MTUs are measure in increments of 100.
When you have the MTU for all of your machines, you will want to go with the lowest common denominator. On two machines, one preferably being the one with the lowest MTU, run this command:
sudo ifconfig eth[n] mtu [size]
Now that we have our test machines. it’s time to make sure our switch supports the frame size. Run this command on either machine to access any the other one:
ping -s [MTU-28] -M do [ip-address]
We have to account for the 8-byte ping header and 20-byte TCP/IP stack header, hence subtracting 28 from our determined MTU. This runs the ping command using our determined MTU size and forces the frames to refuse fragmentation. If you get responses, you're in the clear. If not, either the pinged computer is dropping pings or your switch is refusing the frame size. If its the latter, you'll have to do a bit more discovery on the packet size, in a fashion similar to what you used to discover the lowest max MTU to begin with.
Once you have an MTU size, you can run sudo ifconfig eth[n] mtu [size] on all the machines to have an immediate effect. If you're using DHCP, add this line under the appropriate stanza for your network card in /etc/network/interfaces to make changes permanent:
pre-up /sbin/ifconfig $IFACE mtu [size]
If you're using static addressing, you can get away with
mtu [size]
In my experience, this gives me an increase of network speed from 15MB/s (126mb/s) to 38MB/s (318mb/s). And from my research, this is to be expected. No one has ever reached anywhere near the theoretical 101MB/s (1gb/s) limit of gigabit. Whereas we got a little over half the theoretical limit of 100mb/s (7MB/s or 59mb/s), one can only get a little over one-third of the theoretical limit of gigabit.
2 Comments »
RSS feed for comments on this post. TrackBack URI
Leave a comment
License
The articles in this blog are licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. Other licenses may be available upon request.-
Recent Comments
brainwreckedtech on Linux SSD Optimizations Heiko on Linux SSD Optimizations Dassin on Review: Netgear WGT624v3 Imran on Review: Belkin F5D7230-4 Wirel… Bill Bartmann on Ubuntu 9.04 “Bug” … Chaco on A Disappointing 7th Gener… doodoo head on Brainwrecked Censorship In Tri… Free Xbox Live on Phantasy Star Universe XBox 36… brainwreckedtech on HOWTO: Update Linksys PAP2 Fir… James S on HOWTO: Update Linksys PAP2 Fir… brainwreckedtech on HOWTO: Update Linksys PAP2 Fir… Peter Beauclerk on HOWTO: Update Linksys PAP2 Fir… brainwreckedtech on HOWTO: Compile Midori From Sou… pR@tz on HOWTO: Compile Midori From Sou… pR@tz on HOWTO: Compile Midori From Sou… XBox 360 Gamer Card
Blog at WordPress.com. | Theme: Pool by Borja Fernandez.
Entries and comments feeds.
why do the MTU’s have to match? a workstation accessing the internet will have several devices between it and the internet, a linksys router running linux being a common one, and it will be a much lower MTU.
the lowest common denominator will be all the gear out at the boundaries, which all the machines and servers will likely have to talk to at some point. so what do you do then?
Comment by strange — June 11, 2009 #
The MTUs don’t have to match unless you enjoy having your LAN speed crippled as your computers break apart packets on their own trying to reach a common denominator. While the advice here is for optimizing the speed that computers communicate on a LAN, not the Internet, keep in mind that computers with bigger MTUs will have no trouble accepting smaller packets from computers with smaller MTUs. Your Internet download speeds won’t be affected, but your upload speeds might. However, most people’s Internet connection speed (in the US, at least) doesn’t even hit 1mb/s upstream. Factor that measly speed with all the latency due to routing, server capacity, etc., and the upload speed degradation from mismatched MTUs with the Internet becomes the least of your problems.
By no means should you adjust the MTU of a machine on your LAN if its sole purpose to upload data to the billions of anonymous users on the net. At the same time, you should consider getting that machine off your private network in the event a security breach. A simple double-NAT will do.
Comment by brainwreckedtech — June 12, 2009 #