The Lights Are On But Nobody Is In
I went into my office this very cold morning and started up my Dell Latitude D630 laptop that runs Ubuntu. The splash screen appeared with the scrolling output of the boot process. It then disappeared and was replaced with, well with nothing. The green power light and the Bluetooth light were the only sign of life. but it really was a case of the lights are on but nobody is in.
It took me ages to get my system back up, but here I’ll leave out the diversions and wrong routes I took. We have a number of computers at home, and I am now looking at Bacula to see if it can help automate my backups. I’m also keen to be able to just reload what I had from scratch, but that’s another story that’s hardly begun.
The Recovery Console
It was a cold day and so I tried rebooting now the laptop had warmed up a bit. This time I stared at the messages and the last one before it went blank was:
Waiting for root file system...
The next time I rebooted I hit the Escape key to bring up the Grub menu. I say reboot but it was actually holding down the power button for 4 seconds until it switched off, then pressing it to start again. Once there I selected the option for the Recovery Console for the latest kernel (2.6.31-16). This time I saw a lot more than the earlier messages. Unfortunately it didn’t get to the recovery console leaving me with the following.
Begin: Waiting for root file system... ... [ 6.804829] scsi 5:0:0:0: Direct-Access WD 2500BEV External 1.05 PQ: 0 ANSI: 4 [ 6.805521] sd 5:0:0:0: Attached scsi generic sg2 type 0 [ 6.807014] sd 5:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB) [ 6.807902] sd 5:0:0:0: [sdb] Write protect is off [ 6.807962] sd 5:0:0:0: [sdb] Assuming drive cache: write through [ 6.809673] sd 5:0:0:0: [sdb] Assuming drive cache: write through [ 6.809712] sdb: sdb1 [ 10.434396] sd 5:0:0:0: [sdb] Assuming drive cache: write through [ 10.434438] sd 5:0:0:0: [sdb] Attached SCSI disk Done. Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules: is /dev/) ALERT! /dev/disk/by-uuid/4eb02332-d390-46b0-8db6-d4af1abd992d does not exist. Dropping to a shell!
BusyBox v1.13.3 (Ubuntu 1:1.13.3-1ubuntu7) built-in shell (ash) Enter 'help' for a list of built-in commands
(initramfs)
I’d never been in the recovery mode and it doesn’t look like I will now. This isn’t an area I would describe as my expertise. I work as a Project Management Consultant helping clients implement systems using tools like Primavera P6 and Deltek Cobra. I happen to do some programming, but this kind of stuff is placed in my mind under magic. There was only one thing to do so I did it. I read through the output to see if any of it made sense, and I think it did.
For some reason I am yet to discover, my laptop had stopped being able to find my hard disk with all the stuff on it. Yesterday it had no problem finding it. Today not a chance. I discovered that later versions of Ubuntu try and find the disk by it’s UUID. That’s the long hex-like number. If you look in the /dev/disk/by-uuid directory you’ll find a list of long numbers each relating to a storage device whether it’s your hard disk or a USB storage device. Now left at the shell prompt and typing ls /dev/disk/by-uuid gave me the UUID for the swap drive and a USB drive that was attached but not my drive.
The Road To Recovery
This was a long road for me, but I’ll shorten it for you. The first thing I should have donw was follow jadcox’s post on the Ubuntu Forums. When I do this I get to login.
Then I should have started up Synaptic Package Manager and upgraded the linux-image. When I did this I got 2.6.31-17 and I could boot once again. I then re-installed my 2.6.31-16 image and now that works.
All I can assume is there was some corruption crept in somewhere and it got me in this mess.
I feel quite vulnerable and so am looking at putting proper backup in and even being able to use Bacula to just reload everything from the last backup. In the meantime, I’m trying to find out how I can check the health of an Ubuntu installation.