totalnetsolutions.net

Because not enough information exists in easy-to-find searches: as a simple reminder – SCSI device IDs can and will change.

A few months ago I hot-added a new disk to an ssh bastion host (a VM on ESX). As these things tend to go, I eventually took a maintenance window and updated firmware/BIOS/OS on the ESX host. When the bastion VM came back online, however, I was presented with an odd error:

[root@bastion ~]: fsck /dev/sdc1
e2fsck 1.39 (29-May-2006)
fsck.ext3: Device or resource busy while trying to open /dev/sdc1
Filesystem mounted or opened exclusively by another program?
[root@oracle1 ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
none /selinux selinuxfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
[root@oracle1 ~]# cat /etc/fstab
/dev/main/root / ext3 defaults 1 1
/dev/sdc1 /home ext3 defaults 1 2
/dev/main/var /var ext3 defaults 1 2
/dev/main/tmp /tmp ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/main/swap swap swap defaults 0 0
# Beginning of the block added by the VMware software
.host:/ /mnt/hgfs vmhgfs defaults,ttl=5 0 0
# End of the block added by the VMware software

So everything in the fstab is how I left it – /dev/sdc1 is the new disk I added that is giving errors mounting. So I thought to check for corruption on the disk, and found the problem:

[root@oracle1 ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 5221 41833260 8e Linux LVM
Disk /dev/sdb: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 5221 41937651 83 Linux
Disk /dev/sdc: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 3917 31457279+ 8e Linux LVM


So, a simple fix – change “/dev/sdc1″ to “/dev/sdb1″ in /etc/fstab (or to VOLUME=home), and boot back up.

It’s not something that’ll probably happen on this server again, but it is something to be aware of, on both VM guests and on physical servers. This is why so many newer Linux OSes are using UUID= or VOLUME= instead of device path for SCSI disks.

Upgrading software – always required to keep things secure.  Windows, Wordpress, Mac OSx, Linux, Office, Firefox, etc.  So I just finished upgrading TotalNetSolutions.net again.  Hopefully I’ll be able to be better about this, now that Wordpress does the automatic upgrades now.

I’ve been doing the automatic upgrades on one of my other sites since they came out.  They’re easy, fast, and even more painless than the 3-step upgrade that works so well. So now, I should be able to keep TNS much further away from the “cobbler’s kids” syndrome so many small company’s systems suffer with.

I’ve been using TweetDeck as a Twitter client for a while now, and love it. I recently updated my main laptop from Kubuntu 8.04 to Kubuntu 9.04, and even though the ATI x1300 3d driver is broken, I’m finally getting all the little annoyances fixed up.

The biggest issue I had, though, was after the upgrade, TweetDeck was just showing a blank window, with only the “Tweet/All Friends/Replies/Direct Messages/Groups/Favorites” etc. buttons on the top. There are a lot of posts about how to run Adobe Air and TweetDeck on x64 Ubuntu, but this is a 32-bit machine, so those fixes weren’t working.

The fix is surprisingly simple: make sure kwallet is running with an open wallet. Full details in a nice concise post from Peter Upfold, here. One of his commenters mentioned an issue, but I believe it turned out to be a “no wallets were open” problem. Thanks Peter!

It used to be that you could edit /etc/modprobe.d/blacklist and add “blacklist pcspkr” to turn off the console beeps entirely on Ubuntu / Kubuntu.  As of 9.04, the module is now called “snd_pcsp”.

So, to turn off console (not X terminal, but tty) beeps, you can do one of the following:
1) (This is my preference)

echo blacklist snd_pcsp >> /etc/modprobe.d/blacklist.conf


2) (I’ve done this, but it doesn’t affect all software)

for i in 1 2 3 4 5 6
do
setterm -blength 0 > /dev/tty$i
done

3) (Only works per shell if ~/.inputrc is included)

echo set bell-style visible >> ~/.inputrc


Enjoy more-sane editing from ttyX in the future!

The past week has given me major troubles.  I was tasked with performing a restore of a large database from our offsite storage.  Upon getting the tapes back I found that their indexes were no longer available and I would need to read them in from the tapes…there were only 107 tapes.  Not knowing the software well enough to accomplish this quickly I contacted support, where things began to get more…”interesting”.

After four hours on the phone I was able to determine the two tapes that would be needed to recover the 79Gb database file and started reading in the specific saveset that was needed.  Two hours later I was able to start a restore, which failed.  2Gb of the restore file was missing.  After another two hours on the phone with support I was told “Let’s reposition the tape.  It could take a while, on newer technology I’ve seen it take an hour, on LTO1 and LTO2 drives I’ve seen it take 8 hours.”  You guessed it, I have LTO2 drives.  Fortunately I have a multitude of drives to reposition the tapes with so it won’t impact backups, unfortunately I have a time limit on the restore that’s fast approaching.

So what do you do when you backup your file systems?  Do you simply believe that the software you backup with validates your tapes or do you test them regularly?  Are you satisfied with seeing an email at the end of a backup routine stating “SUCCESS”?  Then answer is simply NO.  Your backups are only as good as your ability to restore from them. Keeping that in mind and all the different technologies and services available what do you choose?

For us the answer is simple.  We require low cost, reliable, offsite secure data storage as do most companies nowadays.  TAPE.  We’ve looked into collocated services and replicated SANs with virtual tape backup but the cost far exceeds it’s benefits.  Tape technology has been proven over and over for decades.  There is no cost effective replacement for a good old fashioned tape, even taking into consideration the troubles it can give you.  Our entire datacenter can be put onto 6 tapes costing $25 each.  4.8TB for $150.

Any good backup initiative should be followed up with an equally adequate restore plan.  So next time you recommend a backup solution plan a regular restore plan to test because there’s nothing worse than spending an entire week restoring one file.