WARNING: Your NIC (or Driver) Might Lie About MTU
This is one for the ages. I had two problems creep in at once. The first was shoddy NFS performance. I Googled my way across the internet for troubleshooting and performance tips. However, nothing seemed to improve the situation. While that was going on, I noticed the second problem — one of the hard drives was failing its own SMART test.
I was already up late, and the last thing I wanted to be bothered with was finding out which drive it was, going the manufacturer’s site to download their drive diagnostic boot CD, getting the warranty return code, and starting the RMA process. But I hung in there and pulled through.
After some sleep, I tried tinkering with a few more networking settings. I was still coming up with nothing, so I ordered a Cat6 cross-over cable so I could at least rule either way on the cabling and network switch.
Fortunately, it seems the drive that failed its SMART test was doing so due to a one-time event. It was failing on its internal sector re-allocation count, and the count did not increase at all between the time I noticed the error and the time I received a replacement. Also of good fortune was that the replacement had the exact same number of sectors, so simple dd and I was back in action.
Unfortunately, that was not the cause of the problem. Neither was shoddy cabling or a failing network switch. NFS performance was still not where it should be.
Then it dawned on me. You don’t need to increase the MTU setting to achieve great network performance. It can help, but it’s not required. And wasn’t it a bit weird that I could set the server’s MTU up to point where, though the driver accepted it, it wouldn’t respond to the network at all?
|MTU||client → server||client ← server|
I didn’t bother since it was obvious that I was being lied to by either the NIC or the driver.