WARNING: Your NIC (or Driver) Might Lie About MTU

This is one for the ages. I had two problems creep in at once. The first was shoddy NFS performance. I Googled my way across the internet for troubleshooting and performance tips. However, nothing seemed to improve the situation. While that was going on, I noticed the second problem — one of the hard drives was failing its own SMART test.

I was already up late, and the last thing I wanted to be bothered with was finding out which drive it was, going the manufacturer’s site to download their drive diagnostic boot CD, getting the warranty return code, and starting the RMA process. But I hung in there and pulled through.

After some sleep, I tried tinkering with a few more networking settings. I was still coming up with nothing, so I ordered a Cat6 cross-over cable so I could at least rule either way on the cabling and network switch.

Fortunately, it seems the drive that failed its SMART test was doing so due to a one-time event. It was failing on its internal sector re-allocation count, and the count did not increase at all between the time I noticed the error and the time I received a replacement. Also of good fortune was that the replacement had the exact same number of sectors, so simple dd and I was back in action.

Unfortunately, that was not the cause of the problem. Neither was shoddy cabling or a failing network switch. NFS performance was still not where it should be.

Then it dawned on me. You don’t need to increase the MTU setting to achieve great network performance. It can help, but it’s not required. And wasn’t it a bit weird that I could set the server’s MTU up to point where, though the driver accepted it, it wouldn’t respond to the network at all?

MTU client → server client ← server
6100 40MB/s 6MB/s
1500 110MB/s 120MB/s
3000 55MB/s didn’t bother

I didn’t bother since it was obvious that I was being lied to by either the NIC or the driver.