A few months ago I hot-added a new disk to an ssh bastion host (a VM on ESX). As these things tend to go, I eventually took a maintenance window and updated firmware/BIOS/OS on the ESX host. When the bastion VM came back online, however, I was presented with an odd error:

[root@bastion ~]: fsck /dev/sdc1
e2fsck 1.39 (29-May-2006)
fsck.ext3: Device or resource busy while trying to open /dev/sdc1
Filesystem mounted or opened exclusively by another program?
[root@oracle1 ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
none /selinux selinuxfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
[root@oracle1 ~]# cat /etc/fstab
/dev/main/root / ext3 defaults 1 1
/dev/sdc1 /home ext3 defaults 1 2
/dev/main/var /var ext3 defaults 1 2
/dev/main/tmp /tmp ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/main/swap swap swap defaults 0 0
# Beginning of the block added by the VMware software
.host:/ /mnt/hgfs vmhgfs defaults,ttl=5 0 0
# End of the block added by the VMware software

So everything in the fstab is how I left it – /dev/sdc1 is the new disk I added that is giving errors mounting. So I thought to check for corruption on the disk, and found the problem:

[root@oracle1 ~]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 5221 41833260 8e Linux LVM
Disk /dev/sdb: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 5221 41937651 83 Linux
Disk /dev/sdc: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 3917 31457279+ 8e Linux LVM


So, a simple fix – change “/dev/sdc1″ to “/dev/sdb1″ in /etc/fstab (or to VOLUME=home), and boot back up.

It’s not something that’ll probably happen on this server again, but it is something to be aware of, on both VM guests and on physical servers. This is why so many newer Linux OSes are using UUID= or VOLUME= instead of device path for SCSI disks.

So, to turn off console (not X terminal, but tty) beeps, you can do one of the following:
1) (This is my preference)

echo blacklist snd_pcsp >> /etc/modprobe.d/blacklist.conf


2) (I’ve done this, but it doesn’t affect all software)

for i in 1 2 3 4 5 6
do
setterm -blength 0 > /dev/tty$i
done

3) (Only works per shell if ~/.inputrc is included)

echo set bell-style visible >> ~/.inputrc


Enjoy more-sane editing from ttyX in the future!

Anyways; Workstation 6.5 has been giving me problems.  Because of the newness of KDE4, I initially thought it was a KDE problem, but it turns out it’s something between Workstation 6.5 and Ubuntu 8.10.  I just ran the “adapt –dist-upgrade-devel” command from the Ubuntu wiki to upgrade, and upon reboot, I couldn’t “ctrl-alt-ins” or “ctrl-alt-del” to log into my Windows VM, my “Windows/Start” key on the keyboard wouldn’t respond, and my arrow keys wouldn’t work.  Incredibly, when I’d hit the “down” arrow, I’d get the Windows Start menu pop up!!

Fix is easy, edit /etc/vmware/config and add the line below like:

sudo vim /etc/vmware/config
:$
A (that's vi-command for "go to the end of the file, and start writing a new line")
xkeymap.nokeycodeMap = true


Have to restart your VMs for this change to take effect. Thanks to Duncan Epping for this fix (he posted it in the forums, where I found it).

For the Unix/Linux world, there doesn’t seem to be as much available.

If you have a pure HP-UX shop, there is HP Systems Insight Manager (SIM) with plug-ins available for software deployment, and I believe IBM Tivoli has a function or sub-product which does the same thing if you have all AIX systems. Red Hat Network has a feature to allow commands to be run on your servers, but only whenever they check in with the RHN or your internal Satellite Server (much like Group Policy, except GPO doesn’t allow “in the middle of the day” script creation without GP-Preferences). So what’s available that’s like SCCM or DesktopAuthority – a “click now and do this thing” tool?

A bunch of my customers just have various levels of logging and processing that come down to being a big for loop that ssh’s into a server and runs a command:
for i in `cat server-list.txt` ; do scp scriptname $i:/root/; ssh $i "/root/scriptname" | tee logfile-$i.log; done;
While it works great for smaller commands. if you have a mixed environment, the “scriptname” script has to be intelligent enough to know what it’s running against, or your “server-list.txt” has to be broken up by class of system. In either case, if you have 200 systems in the list, and the task takes 5 minutes per server, a single install will run for 16-17 hours.

Software like Likewise Enterprise which allow Group Policy management to remote computers is great, because you can have guaranteed delivery and execution of your script or command in (by default) 30 minutes, but my problem is how to get it there in the first place?

So, administrators out there in companies with 1000, 4000, 10000+ servers (or even Desktops), what mutli-threaded or multi-process tool are you using to tackle this timing/resouce problem? Please post below!

• AIX: iptrace: /usr/sbin/iptrace [ -a ] [ -b ][ -e ] [ -u ] [ -PProtocol_list ] [ -iInterface ] [ -pPort_list ] [ -sHost [ -b ] ] [ -dHost ] [ -L Log_size ] [ -B ] [ -T ] [ -S snap_length] LogFile
• FreeBSD: tcpdump (I think): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• HP-UX: nettl: nettl requires a daemon start, and other setup: /usr/sbin/nettl -traceon kind… -entity subsystem… [-card dev_name...] [-file tracename] [-m bytes] [-size portsize] [-tracemax maxsize] [-n num_files] [-mem init_mem [max_mem]] [-bind cpu_id] [-timer timer_value]
• Linux 2.4 and higher:
  • tcpdump (some distros): tcpdump [ -AdDefKlLnNOpqRStuUvxX ] [ -c count ] [ -C file_size ] [ -G rotate_seconds ] [ -F file ] [ -i interface ] [ -m module ] [ -M secret ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ] [ -E spi@ipaddr algo:secret,... ] [ -y datalinktype ] [ -z postrotate-command ] [ -Z user ] [ expression ]
  • wireshark (some distros, used to be called “ethereal”): GUI-config, no command-line, use tethereal (now tshark) for that
  • tshark: tshark [ -a <capture autostop condition> ] … [ -b <capture ring buffer option>] … [ -B <capture buffer size (Win32 only)> ]  [ -c <capture packet count> ] [ -C <configuration profile> ] [ -d <layer type>==<selector>,<decode-as protocol> ] [ -D ] [ -e <field> ] [ -E <field print option> ] [ -f <capture filter> ] [ -F <file format> ] [ -h ] [ -i <capture interface>|- ] [ -l ] [ -L ] [ -n ] [ -N <name resolving flags> ] [ -o <preference setting> ] … [ -p ] [ -q ] [ -r <infile> ] [ -R <read (display) filter> ] [ -s <capture snaplen> ] [ -S ] [ -t ad|a|r|d|e ] [ -T pdml|psml|ps|text|fields ] [ -v ] [ -V ] [ -w <outfile>|- ] [ -x ] [ -X <eXtension option>] [ -y <capture link type> ] [ -z <statistics> ] [ <capture filter> ]
• Mac OSX: tcpdump (among others): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• Solaris: snoop: snoop [ -aPDSvVNC ] [ -d device ] [ -s snaplen ] [ -c maxcount ] [ -i filename ] [ -o filename ] [ -n filename ] [ -t [ r | a | d ] ] [ -p first [ , last ] ] [ -x offset [ , length ] ] [ expression ]
• Windows 2000, XP, 2003, Vista, 2008 and beyond:
  • netmon (not installed by default): GUI config, filter creation info here
  • wireshark (available by download): GUI-config, no command-line

Any others anyone wants added (or corrected), just comment or email and I’ll update this.
(Edit 7/29/08 – change tcpdump link)
(Edit 10/13/08 – add tshark info, thanks Jefferson!, and wireshark on Windows)
(Edit 12/27/09 – update IBM iptrace man page link)

However, this new PHP webserver caused a few glitches. I feel a little silly for not catching this as being an SELinux problem earlier, but since it’s caused 0 issues in 9 months of use in production, I didn’t even consider it initially.

What we initially saw was 0 errors from PHP – all the pages would run without error. PHP.ini has the following lines:

sendmail_from = from@domain.com
sendmail_path = /usr/sbin/sendmail -t -i

and testing cat mail.txt | /usr/sbin/sendmail -t -i as a non-root user delivered mail properly as well. Combine that with /var/log/maillog being completely empty for every test page loaded, and it was sure that the mail wasn’t getting TO postfix (our preferred localhost MTA).

So, I looked at the /var/log/httpd/error_log for apache and found:

sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied

But I knew that non-root users could access sendmail as defined in php.ini, so I finally decided to tail /var/log/messages and saw:

Nov 2 11:05:41 $(servername) setroubleshoot: SELinux is preventing the sh from using potentially mislabeled files sendmail.postfix (sendmail_exec_t). For complete SELinux messages. run sealert -l c9001c48-5d48-4b7c-9fd7-8400544daa8f

So now to fix it…
This is surprisingly simple, actually. The sad part is, we had this problem, fixed it, forgot about it, had it again, and I blogged it… and lost the post. so this has been sitting in my “drafts” folder for about 10 months now:
setsebool httpd_can_sendmail=true
service httpd restart
service postfix restart
And retry sending mail. There’s a few posts about sendmail and having to change permissions on home directories or on “main.cf”, but I use postfix, and not sendmail, so I don’t know how effective or necessary those changes are.

With my recent upgrade of that same laptop to Kubuntu Gutsy Gibbon, I kept the laptop-mode.conf file a bit closer to the maintaner’s version. There are some changes to the “dirty-writeback-centiseconds” and the “dirty-background-ratio” values from what I posted, and my issue seems to have gone away. I’ve been able to go back to running my Windows 2003 SBS server with a Centrify DirectControl lab environment and a RHEL 4 Oracle 10g server attached at the same time.

The configuration files that work MUCH better are attached here:

laptop-mode.conf

cpufreqd.conf

For completeness the error I was getting was the same:
/tmp/vmware-config1/vmnet-only/userif.c:630: error: ‘const struct sk_buff’ has no member named ‘h’
when trying to build the VMNet module – VMMon built and inserted perfectly. Downloaded 6.0.1 and installed it, and I’m back and in the game.

However, because I mostly design Windows networks, I also run VMWare Workstation 6 with the following VMs: Windows XP Pro joined to primary domain (full workstation with Office 2003, Visio, Outlook (and EMC EmailExtender plugins), Windows Admin Pack, Resource Kit, SQL Enterprise Manager, and Exchange Admin Pack), Windows Vista Enterprise, Windows 2003 DC, Exchange, SQL server, Windows XP Pro joined to VM domain, RHEL 4, RHEL 5, and a Live-CD system that I often use to test bootable CDs. And a sysprep’d Windows install.

All that, plus Evolution caching my email and Firefox being disk-happy (did you ever “strace -p $(pgrep firefox) -e trace=open,close,read,write”? It’s busier than Evolution during an offline mail check!), means my 7200 RPM 160GB SATA disk gets hammered. Me also using laptop-mode-tools changing my vm.dirty_writeback_* settings and read/write cache isn’t helping either, I’m sure.

Today and yesterday I ran into an issue where my disk would begin a sync that would last 10-20 minutes, leaving me unable to work the entire time that was happening. Hunting down WHAT was causing this, however, was even more frusterating than it happening (If I shut down any of the 3 above-mentioned programs, the problem went away – it only happened with all 3 open). In Windows, you can open Task Manager, go to the “Process” tab, click “View-> Columns” and add “IO Write Bytes” and “IO Read Bytes” and watch the numbers count up. Or you can use Perfmon and look at IO reads/writes/bytes per second or total, and know immediately what’s causing all your disk IO pain. I still don’t know how to do this in Linux.

First, any hunt for “disk utilization” and “Linux” on Google directs you to hundreds of sites, forums, and blogs evangelizing the wonders of “df” for disk utilization. Yes, it’s really nice to know how much free space I have on my hard drives- that’s why I have SuperKaramba to tell me. But when a problem hits and leaves me unable to work, it’s useless.

“iostat -k 1″ is great – you’ll know immediately which disk is being used, and how hard. But on a laptop with a single disk, you already know.

“top” sorted by process-state will show you what’s in “waiting on IO” state, but not what’s CAUSING the IO that’s causing everything else to wait.

“sar” seems to be the only tool that can provide per-process IO stats, but it has to be pre-set up to write to a log. And I can’t begin to guess how well that will work when my disk is at 100% utilization (peaked at 120tps today).

So if anyone knows of any way to know what’s causing disk IO in a “right now” fashion, please comment or email me. And if you’re curious more about my problem:

1. Only happens when VMWare (with a guest), firefox, and Evolution are all running.
2. VMWare with multiple guests runs fine, and never has this issue.
3. rauch@lt00-bofh:~$ free
total used free shared buffers cached
Mem: 3348960 1099216 2249744 0 68668 533124
-/+ buffers/cache: 497424 2851536
Swap: 6000268 0 6000268
4. Happens with Laptop_mode disabled or enabled, on AC or on battery
5. “sync” causes the exact same symptoms, leading me to believe that somehow I’m getting a LOT more dirty pages than my parameters are set at.
6. dirty_background_ratio
1
dirty_expire_centisecs
60003
dirty_ratio
60
dirty_writeback_centisecs
60003
7. For now, I just close Firefox when I have VMWare open, which means I spend a lot more time in IE than I want to.

As a final note, I’ve updated my Linux EVDO post here with my new built-in card’s info.