Ubuntu, RAID, and Some Serious WTF

Ubuntu Logo Just when you think you’re getting to know Linux, something comes up to knock you off your high horse.

I had reached a boiling point with my file server. It was stuck at the horrible initial Suse 10.1 release. I couldn’t update it because there were serious bugs in YOU (YaST Online Update) that made the update process bomb after everything was set up. It’s safe to say the system had been running at for at least a year unpatched. During this time, Novell sold out to Microsoft. Too bad for Novell as I was already warming up to Ubuntu to begin with. I had already switch my little LAMP (Linux Apache MySQL PHP) plaything over to Ubuntu. But I held off on the file server as it was way more complex (software RAID, configuration of Samba, NIS, and NFS).

The Meat of the Story – MDADM

What I didn’t realize going into this is that the program mdadm has an auto-detect and auto-mount features. When you create a software RAID device, mdadm will write a super block at the end of each physical partition that will help assemble the RAID automatically in the future. To help you understand what I mean, let’s say this table below represents the partitions on each of my hard drives, and that the [] represents the super block that contains the information mdadm sets aside for itself:

  /boot swap / /home
/dev/sda   []  []  [] 
/dev/sdb   []  []  [] 

Keep in mind that destroying or over-writing the partition table doesn’t touch any of the data on the hard drive. This means if you re-create any partition on the exact same cylinder boundaries, and set the file system flag for each partition, your data should be in tact.

By default, mdadm will not mount software RAID devices if the physical partitions that make up the RAID vary by more than 1%. Of course this behavior can be overridden, which is what openSUSE did. Ubuntu doesn’t, and hence my original configuration failed. Pro-tip for the future: If you’re going to make RAID devices, even RAID 0, match up the partition sizes.

Also, it seems to me that Ubuntu launches mdadm in some auto-mount mode when you start configuring RAID devices, while openSUSE does not. That sucks, because once mdadm mounts the RAID devices, you can’t make changes to the physical partitions. Even if you don’t have an existing software RAID array, if you screw up in the Ubuntu installer, your stuck with it.

From my research, there are two ways to clear the suberblock:

  1. Use mdadm to zero out the super block:

    mdadm --zero-superblock /dev/sda1
    mdadm --zero-superblock /dev/sda2
    mdadm --zero-superblock /dev/sda3
    mdadm --zero-superblock /dev/sdb1
    mdadm --zero-superblock /dev/sdb2
    mdadm --zero-superblock /dev/sdb3

  2. Zero out the entire hard drive using dd:

    dd if=/dev/zero of=/dev/sda
    dd if=/dev/zero of=/dev/sdb

The Meat of the Story 2 – GRUB and BIOS Boot Orders

If you have IDE and SATA drives in your PC (like I do) and switch around the boot order of the drives (like I did), you may be in for a special treat when you reboot your computer after the installation and GRUB can’t find its configuration file. This can happen when the installer detects the actual order of the hard drives. Normally, installers ask you where you want to install the boot loader, but Ubuntu tries to do so much for you it trips over itself. If you don’t set a partition’s boot flag, the only logical choice is to install to the Master Boot Record.
That makes sense. But when the installer writes GRUB to the MBR of the wrong disk, fun ensues.

The best way to fix this is to boot from a rescue CD like Knoppix and run

sudo grub-install /dev/sda

Then, make sure the other disks’ MBR is wiped out

dd if=/dev/zero of=/dev/hda bs=512 count=1
dd if=/dev/zero of=/dev/sdb bs=512 count=1

How I Went Downhill Fast

I figured everything should be quick and easy. I already had everything set up from openSUSE. All I should have had to do was back up the /home directory, wipe everything, and start again. I started the installer, and when it came to set up the partitions, the installer didn’t like my existing set up. Which was (as pulled from YaST):

┌──────────────────────────────────────────────────┐
│Device   │    Size    │ F │Type           │ Mount │
│/dev/hda │    232.8 GB│   │HDS722525VLAT80│       │
│/dev/hda1│    232.8 GB│   │Linux native   │/backup│
│/dev/sda │    232.8 GB│   │SAMSUNG-SP2504C│       │
│/dev/sda1│     15.6 MB│   │Linux native   │/boot  │
│/dev/sda2│    125.5 MB│   │Linux RAID     │       │
│/dev/sda3│    384.3 MB│   │Linux RAID     │       │
│/dev/sda4│    232.3 GB│   │Linux RAID     │       │
│/dev/sdb │    232.8 GB│   │SAMSUNG-SP2504C│       │
│/dev/sdb1│    133.3 MB│   │Linux RAID     │       │
│/dev/sdb2│    384.3 MB│   │Linux RAID     │       │
│/dev/sdb3│    232.3 GB│   │Linux RAID     │       │
│/dev/md0 │    258.6 MB│   │MD Raid        │swap   │
│/dev/md1 │    768.4 MB│   │MD Raid        │/      │
│/dev/md2 │    464.7 GB│   │MD Raid        │/home  │
└──────────────────────────────────────────────────┘

If you notice, the first RAID’s physical partitions aren’t identical. A small 15.6MB chunk was “taken out” of the first disk’s RAID partition to make room for a non-RAID partition to use for /boot. I did this to get around the fact that GRUB can’t boot from RAID partitions. This is where openSUSE screwed me, because the variance on these two partitions is 6%. Oh, it did warn me that it wasn’t normal, but it let me go ahead and do it.

Now, I can’t remember the details, but I know a lot of funky stuff happened from this point forward. Knowing what I know now, it all makes sense. But without that knowledge, it was inconsistent and frustrating. Sometimes I couldn’t change the actual physical partitions. When I could, messing with the RAID would bomb. All I wanted was:

┌──────────────────────────────────────────────────┐
│Device   │    Size    │ F │Type           │ Mount │
│/dev/hda │    232.8 GB│   │HDS722525VLAT80│       │
│/dev/hda1│    232.8 GB│   │Linux native   │/backup│
│/dev/sda │    232.8 GB│   │SAMSUNG-SP2504C│       │
│/dev/sda1│     23.5 MB│   │Linux native   │/boot  │
│/dev/sda2│    117.7 MB│   │Linux RAID     │       │
│/dev/sda3│    368.7 MB│   │Linux RAID     │       │
│/dev/sda4│    232.4 GB│   │Linux RAID     │       │
│/dev/sdb │    232.8 GB│   │SAMSUNG-SP2504C│       │
│/dev/sdb1│     23.5 MB│   │Linux native   │/boot  │
│/dev/sdd2│    117.7 MB│   │Linux RAID     │       │
│/dev/sdb3│    368.7 MB│   │Linux RAID     │       │
│/dev/sdb4│    235.1 GB│   │Linux RAID     │       │
│/dev/md0 │    737.1 MB│   │MD Raid        │swap   │
│/dev/md1 │    768.4 MB│   │MD Raid        │/      │
│/dev/md2 │    464.8 GB│   │MD Raid        │/home  │
└──────────────────────────────────────────────────┘

Do you know how frustrating it is when you get screens like this:

GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 
GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB GRUB 

Going back to SUSE?

At this point I temporarily gave up on Ubuntu and tried to go back to openSUSE 10.2 using the network install.

Right off the bat, SUSE annoyed me with it’s insistence on using so many directories in its repositories. Seriously, I didn’t have to look up that I was using ftp://suse.mirrors.tds.net/pub/opensuse/distribution/10.2/repo/oss while writing this because I had to correct my mistakes in the directory path so many freakin’ times. OK, actually, some of the blame lies with the mirror — they tacked on the first two directories. But /repo/oss? How about putting this logic into the installer?

I finally get everything typed in correctly, and the installer bombs because I don’t have enough memory. And I can’t activate a swap partition because the swap partition is going RAID! What the hell? It installed before! I look for the system requirements. SuSE 10 requires a whopping 256MB of RAM for the installer. Hey, I have 256MB of RAM! Oh wait, I have on-board graphics. Then how did I get this installed on here before? Oh, that’s right. I ordered the wrong memory when assembling my file server and bummed a 1GB module from my main desktop PC while I waited for the exchange.

So I bum the memory again and we’re back in business. Until I get to the software selection part. Now, if you go back to that system requirements link, notice how they say 500MB for a minimal system? Bullshit. The default minimal system is 730MB. I remove the Novell AppArmor and ZEN Enterprise packages, and then my disk space usage drops to 550MB. However, when I select Accept, YaST resolves dependencies and now has insisted on installing X11 components and fonts thus bumping me back up to 730MB. I am not shitting you. Even worse, when I installed extra packages for a file server (nfs, nis, and samba) the total jumps to over 800MB!

I grit my teeth, go back, and change my partition settings so I’ll have 1GB for home instead of 768MB. I proceed as normal and FINALLY the installer proceeds to install something! I leave the house to pick up something to eat. I come back to something about INIT spawning too many times. This isn’t good. CTRL + ALT + DELETE. Nope, can’t reboot because that’s handled by INIT. I do a hard reboot, and I’m greeted by a flowing screen of error messages from YaST.

Going Back To Ubuntu

Fuck it, I’ll try one more time in Ubuntu to make this work. Well, guess what? It did! Sorta.

GRUB Loading stage1.5.                                                                                                                   
                                                                                
GRUB loading, please wait...                                                    
Error 15                                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                

Turns out Error 15 is a “file not found” error. OK, this should be easy to fix. I pop in my Knoppix CD and run

sudo grub-install /dev/sda

Reboot, and I get:

Probing devices to guess BIOS drives. This may take a long time.                
                                                                                
       [ Minimal BASH-like line editing is supported.   For                     
         the   first   word,  TAB  lists  possible  command                     
         completions.  Anywhere else TAB lists the possible                     
         completions of a device/filename. ]                                    
grub>                                                                           
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 
                                                                                 

OH MY GOD DOES IT HAVE TO BE THIS DIFFICULT?!?!?!

I calm down, and boot back into Knoppix, jot down the appropriate information, reboot, and then enter the commands into GRUB myself.
I go back in and enter the first line, which should have been:

kernel (hd0,3)/boot/vmlinuz-2.6.20-16-generic root=/dev/sda3 ro quiet splash

However, I was lazy and decided to start using tab after kernel (hd0,3)/. All I got though was lost+found. After a while it donned on me what happened. The installer detected my disks IDE-first SATA-second, so when it wrote the devices.map file, hd0 was /dev/hda, hd1 was /dev/sda, and hd2 was /dev/sdb. I changed hd0 to hd1 and then TAB worked as expected.

HOLY SHIT IT WORKED!

Once in Ubuntu, I ran the same code I did under Knoppix to install GRUB. I double-check the devices.map and the menu.lst file to make sure everything is OK.

I FINALLY GOT IT!

Except for one small detail: I’d like to have the extra 256MB SUSE needed for / back into /home.

I boot back into Knoppix and launch qtparted. At this point I knew about the super block, but I didn’t know how to get rid of it. So what I ended up doing is formatting all of the partitions as ext2 and then wiping out the partition tables. This worked well enough to let me set up the partitions as I liked. There was still some auto-detection going on, but I was able to change things to the way I wanted them.

Feedback Welcome

Whew, that was a lot. With something this big, I wouldn’t be surprised if I made an oversight. If you want to nitpick something as minuscule as my “fake” screen shots, go ahead. The more details we have, the less confusion there will be.