A few months ago I hot-added a new disk to an ssh bastion host (a VM on ESX). As these things tend to go, I eventually took a maintenance window and updated firmware/BIOS/OS on the ESX host. When the bastion VM came back online, however, I was presented with an odd error:

[root@bastion ~]: fsck /dev/sdc1
e2fsck 1.39 (29-May-2006)
fsck.ext3: Device or resource busy while trying to open /dev/sdc1
Filesystem mounted or opened exclusively by another program?
[root@oracle1 ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
none /selinux selinuxfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
[root@oracle1 ~]# cat /etc/fstab
/dev/main/root / ext3 defaults 1 1
/dev/sdc1 /home ext3 defaults 1 2
/dev/main/var /var ext3 defaults 1 2
/dev/main/tmp /tmp ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/main/swap swap swap defaults 0 0
# Beginning of the block added by the VMware software
.host:/ /mnt/hgfs vmhgfs defaults,ttl=5 0 0
# End of the block added by the VMware software

So everything in the fstab is how I left it – /dev/sdc1 is the new disk I added that is giving errors mounting. So I thought to check for corruption on the disk, and found the problem:

[root@oracle1 ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 5221 41833260 8e Linux LVM
Disk /dev/sdb: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 5221 41937651 83 Linux
Disk /dev/sdc: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 3917 31457279+ 8e Linux LVM


So, a simple fix – change “/dev/sdc1″ to “/dev/sdb1″ in /etc/fstab (or to VOLUME=home), and boot back up.

It’s not something that’ll probably happen on this server again, but it is something to be aware of, on both VM guests and on physical servers. This is why so many newer Linux OSes are using UUID= or VOLUME= instead of device path for SCSI disks.

Anyways; Workstation 6.5 has been giving me problems.  Because of the newness of KDE4, I initially thought it was a KDE problem, but it turns out it’s something between Workstation 6.5 and Ubuntu 8.10.  I just ran the “adapt –dist-upgrade-devel” command from the Ubuntu wiki to upgrade, and upon reboot, I couldn’t “ctrl-alt-ins” or “ctrl-alt-del” to log into my Windows VM, my “Windows/Start” key on the keyboard wouldn’t respond, and my arrow keys wouldn’t work.  Incredibly, when I’d hit the “down” arrow, I’d get the Windows Start menu pop up!!

Fix is easy, edit /etc/vmware/config and add the line below like:

sudo vim /etc/vmware/config
:$
A (that's vi-command for "go to the end of the file, and start writing a new line")
xkeymap.nokeycodeMap = true


Have to restart your VMs for this change to take effect. Thanks to Duncan Epping for this fix (he posted it in the forums, where I found it).

I was having the problem on a Virtual system in VMWare ESX 3.5, so it was easy to disconnect a disk to get past the error, but downloading and installing the updated RC seemed like the better fix for the first DC in a new test lab.

What apparently is going on is if you have 2 hard drives that have never been partitioned or initialized, then the Setup.exe program gets confused. You can remove one of the disks temporarily, format them with another boot medium (BartPE, anyone?), or just not use Win2k8 RC0. According to the support note, the only fix for Vista is to format the drives.  I bet you can remove one of them and Vista will work, too, but haven’t tested.

Note that the install I linked to is an “Update Package for Red Hat” – this requires RPM to install properly, I’m not sure if that’s to install the script in /etc/rc6.d/ to perform the actual update, or if there’s also a binary driver that needs to be updated as well. To perform this RPM update, rather than building a DOS floppy (or if you don’t have a floppy drive in the server):

1. Download the update to a USB thumb drive or burn it to a CD.
2. Reboot the ESX server.
3. At the boot prompt, you have the choice of “VMware ESX Server”, “VMWare ESX Server (Debug)” or “ESX Troubleshooting mode”. Boot to Troubleshooting mode – this WILL bring you to a shell, eventually.
4. Insert the thumb drive or CD media and mount it.
   For a thumb drive:
mkdir /mnt/usb
mount -t auto /dev/sdb1 /mnt/usb

5. Execute the fix:
/mnt/usb/PE2900_BIOS_LX_2.1.1.BIN
6. Read the notes and hit “q” to finish, then let the server reboot and apply the update.

Server should come up 100% fine after the reboot, and you’re good to go!

For completeness the error I was getting was the same:
/tmp/vmware-config1/vmnet-only/userif.c:630: error: ‘const struct sk_buff’ has no member named ‘h’
when trying to build the VMNet module – VMMon built and inserted perfectly. Downloaded 6.0.1 and installed it, and I’m back and in the game.