• AIX: iptrace: /usr/sbin/iptrace [ -a ] [ -b ][ -e ] [ -u ] [ -PProtocol_list ] [ -iInterface ] [ -pPort_list ] [ -sHost [ -b ] ] [ -dHost ] [ -L Log_size ] [ -B ] [ -T ] [ -S snap_length] LogFile
• FreeBSD: tcpdump (I think): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• HP-UX: nettl: nettl requires a daemon start, and other setup: /usr/sbin/nettl -traceon kind… -entity subsystem… [-card dev_name...] [-file tracename] [-m bytes] [-size portsize] [-tracemax maxsize] [-n num_files] [-mem init_mem [max_mem]] [-bind cpu_id] [-timer timer_value]
• Linux 2.4 and higher:
  • tcpdump (some distros): tcpdump [ -AdDefKlLnNOpqRStuUvxX ] [ -c count ] [ -C file_size ] [ -G rotate_seconds ] [ -F file ] [ -i interface ] [ -m module ] [ -M secret ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ] [ -E spi@ipaddr algo:secret,... ] [ -y datalinktype ] [ -z postrotate-command ] [ -Z user ] [ expression ]
  • wireshark (some distros, used to be called “ethereal”): GUI-config, no command-line, use tethereal (now tshark) for that
  • tshark: tshark [ -a <capture autostop condition> ] … [ -b <capture ring buffer option>] … [ -B <capture buffer size (Win32 only)> ]  [ -c <capture packet count> ] [ -C <configuration profile> ] [ -d <layer type>==<selector>,<decode-as protocol> ] [ -D ] [ -e <field> ] [ -E <field print option> ] [ -f <capture filter> ] [ -F <file format> ] [ -h ] [ -i <capture interface>|- ] [ -l ] [ -L ] [ -n ] [ -N <name resolving flags> ] [ -o <preference setting> ] … [ -p ] [ -q ] [ -r <infile> ] [ -R <read (display) filter> ] [ -s <capture snaplen> ] [ -S ] [ -t ad|a|r|d|e ] [ -T pdml|psml|ps|text|fields ] [ -v ] [ -V ] [ -w <outfile>|- ] [ -x ] [ -X <eXtension option>] [ -y <capture link type> ] [ -z <statistics> ] [ <capture filter> ]
• Mac OSX: tcpdump (among others): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• Solaris: snoop: snoop [ -aPDSvVNC ] [ -d device ] [ -s snaplen ] [ -c maxcount ] [ -i filename ] [ -o filename ] [ -n filename ] [ -t [ r | a | d ] ] [ -p first [ , last ] ] [ -x offset [ , length ] ] [ expression ]
• Windows 2000, XP, 2003, Vista, 2008 and beyond:
  • netmon (not installed by default): GUI config, filter creation info here
  • wireshark (available by download): GUI-config, no command-line

Any others anyone wants added (or corrected), just comment or email and I’ll update this.
(Edit 7/29/08 – change tcpdump link)
(Edit 10/13/08 – add tshark info, thanks Jefferson!, and wireshark on Windows)
(Edit 12/27/09 – update IBM iptrace man page link)

We haev this running ever 3 minutes from 3 systems: 1 Windows 2003, 1 Fedora Core 8, and 1 Kubuntu Gutsy Gibbon. Requires Net::SMTP, Config::INIFiles, LWP::UserAgent, and HTTP::Request. The only one that I’ve needed to download and install is Config::INIFiles on any of those 3 systems. But I do have LWP::Simple on all systems, so I’m not sure if you’ll need the last 2. This is my first published code other than 3 line bash scripts, so be kind in comments.

Feel free to take and use / improve / update this – I’d just appreciate if you’d let me know so I can update this version here. The parseIni() function checks that all “URL”s are in http://www.google.com format or http://64.233.167.99 format (it checks for http:// followed by text followed by what appears to be a valid TLD format, or it checks for http:// followed by an IP address). I have yet to add in the regex to look for a valid full URI, because I didn’t need that yet.

This is upgraded over responser.pl in that:

1. It will send to any number of SMTP recipients (comma-separated)
2. It will silence its alerting if *all* checked addresses are down. If the monitoring system gets unplugged from the network, it won’t attempt to send hundreds of alerts upon regaining access. Or if you’re testing from a DSL line, you won’t get alerts because the DSL line went down, but the actual target was up. The next version will have this as an option in the INI file.
3. It uses standard INI file formatting, rather than a parsed text file.
4. it runs out of the box (so to speak) on Windows (ActivePerl) or Linux (Fedora and Ubuntu both tested).
5. It has better inline documentation.

The major problem is that a minimum of 2 URLs are needed in the INI file for the full logic to work. You can get around this for small networks by adding in the DNS domain for one, and the IP address for the other.

Thanks, and please share any concerns or problems.

chk-web.pl

We’ve seen some issues over the past approximately 2 months, particularly with MS SQL 2000 clusters (1 Exchange 2003 cluster), where the cluster group fails on one node, and the other node (or nodes) fails to pick up the group, leaving the complete cluster group offline. In each of the cases (on both HP and Dell hardware) the first striking piece of evidence in the logs is that all nodes that fail to bring up the cluster report that the Cluster IP Address resource couldn’t be brought online, because of an IP address conflict on the network

Making this issue particularly fun is that most of the information we used to solve the problem, is a lack of information.  In particular, there is absolutely nothing interesting at all in any nodes’ cluster.log file.  You see the disks negotiate from node to node, but nothing that makes the failover look any different than if you had right-clicked the group and chosen “Move Group” from Cluster Administrator.

What starts the problem off is Event ID 1228 from source “ClusNet”, which says that the “ClusNet driver couldn’t communicate with the ClusSvc for 60 seconds, the Cluster service is being terminated.” Most of the time, you might even miss that this event is there, because it causes so many Event Source Tcpip, ID 4199; Source ftdisk, ID 57; and Source ntfs event ID 50 events, that it’s easy to look over 1 little error. Especially when monitoring systems like Microsoft Operations Manager (MOM), or Idera SQLDiagnostics Manager (SQLDiag) or HP Systems Insight Manager (SIM) all report the cluster as having issues 30-60 seconds after the CluNet 1228 event is written (timing which corresponds exactly to the Tcpip 4199 events (IP address conflict) or the ftdisk 57 events (failed to flush transaction data).  So, here’s what happens, based on conversations with Microsoft, training with Microsoft and HP, and a LOT of reading.

First, we have to discuss the layout of Microsoft Clustering.  There is the well-known “ClusSvc” – the “Cluster” service that you find in the “Services” MMC console.  This is what’s known as a user-mode process (as opposed to kernel-mode), so it runs in the same memory and processor spaces as Exchange and SQL.  There are 2 other “services” – the Cluster Network driver (ClusNet) and the Cluster Disk driver (ClusDisk).  If you were ever an NT4 MCSE, you might remember the old Control panel “Drivers” extension.  These services would show up in there, if it still existed, along with a lot of other driver/services – they haven’t gone away, just the GUI for editing them.  Anyways, because ClusSvc runs in the same processor context as SQL or Exchange, and because it shares memory with SQL or Exchange, it can be starved of resources by SQL or Exchange, or any other user-mode process running on your server.  To keep track of whether this has happened or not, Microsoft wrote the ClusNet driver to not only keep track of the private and public networks, and the group IP address resources currently owned by the node, they wrote it to also check a heartbeat (default of 60 seconds) back with ClusSvc.  The best explanation I’ve found is here on Microsoft’s site.

So, the ClusNet event 1228 means that the ClusSvc failed to heartbeat within the 60 second timeout window, so ClusNet killed the Cluster.exe process (ClusSvc).  Generally the OS is set to auto-restart the Cluster service if it terminates, so the node will recover… but what about the cluster as a whole?  Some interesting things happen when cluster.exe is forcibly terminated.

1. Because it’s terminated, cluster.exe can’t tell ClusNet to drop registration of the group IP address resources owned by the node, so ClusNet keeps the IP addresses advertised (arp’d) on the network.
2. Cluster.exe also can’t tell ClusDisk to flush ntfs logs to disk in preparation for the disks being grabbed by another node.
3. Because the Cluster service isn’t running, the node can’t heartbeat to the other nodes in the cluster.
4. Therefore, the other node(s) reach a quorum, decide that the first node is unavailable, and begin the process of a failover.
5. First, disks are negotiated on the SCSI/FibreChannel bus, eventually forcibly taken from the “offline” node (causing the ftdisk 57 and ntfs 50 errors).
6. Simultaneously (unless you have IP address resources having a requirement on your disks), the preferred failover node (or next in the round-robin process) attempts to advertise that its MAC address is now the IP address of the cluster group.
7. And things get ugly.

Because the original node still hasn’t restarted its Cluster Service (services are normally restarted after 60 seconds / 1 minute, unless you changed your cluster), the ClusNet driver on the original node is still arping the group IP address.  This causes the tcpip 4199 errors on the node that’s attempting to bring up the group.  These errors cause the group to fail to come up on the 2nd node, so the cluster group moves to the next node in the round-robin (or preferred owners) list, untill all nodes have failed to bring up the group.  In my experience, 4 nodes all fail to bring up the group in under 60 seconds, so the cluster group stays offline failed, even after the cluster service restarts on the original node.  Also, the disks have been moved to every node in the cluster, so you can probably gather where the disk errors come from.

So how can you fix or avoid this problem?  I am taking the advice written between the lines of the previously mentioned Microsoft article.  The default action of a failure of the heartbeat is to restart the cluster service, which causes the issues mentioned above.  However, there are 2 other options available:  Do Nothing (and log or not), and Bugcheck.

If you set the mentioned registry key to “3″, the heartbeat failure will cause a bugcheck of the server, dumping out a kernel memory dump (or minidump or full dump, depending on your settings, but the default is a kernel dump) of exactly what was going on when the heartbeat failed.  Then, because the failed node is now Blue Screened while it writes out the memory dump, it will no longer fight the disk arbitration OR advertise the cluster group IP addresses that it had owned.  The next node in the list will pick up the group, restart the resources, and your cluster is once-again highly available.

And you now have more troubleshooting information than you would if you just let the service restart.

Now, we had some discussions with clients about this setting, and they were concerned that bugchecking a server could cause data corruption, especially on a busy SQL server.  This is true.  However, restarting the cluster service causes a forced unload of running cluster group processes anyways, so bugcheck or cluster service restart causes the exact same data corruption issues, but a bugcheck gives the technician looking at the problem more data, to know how to avoid the problem in the future.  Also, if no action is taken, you could have the cluster in a resource starvation situation, in which case the technician has to manually kill processes, if he can even get into the server to kill processes.  The resource starvation may even force someone to physically reset power to the server node.  And while that’s happening, the cluster may not fail over to the working nodes, causing an actual business outage.  And isn’t that why you’re building clusters in the first place?

1. Same as before: why are you changing IPs? In larger environments, I do this because of a physical move of just one site. If the networking team doesn’t have the new subnet up and routing, don’t start!
2. Make sure the new site (if required) is set up in AD. If I’m moving DCs from one physical location to another, I will build a new site, rather than re-using the old one, because the new site often has better connectivity, so the site link costs are changing.
3. Add the new IP to the DC you’re moving (DC01 for this). Same as before: don’t remove the old one, just add the new.
4. On DC01, do the following to verify registration worked:
   ipconfig /registerdns
   Wait a few minutes.
   nslookup
server DC01
set type=A
DC01.foobar.local
foobar.local
server DC02
DC01.foobar.local
foobar.local
   The answers from DC01 and DC02 should be the same, with possibly different orders. The important thing is that the new IP address and the old IP address show up for both queries on both servers.
5. Shut down DC01, pack it up and move it. (Or just plug it into the new network.)
6. Boot up, verify that DC01 has network connectivity, and that other systems can see that it has the new IP.
7. If you haven’t, make the new IP primary (change order in Network settings), make sure the DNS and WINS servers are correct and reachable (Remember that Windows 2003 DNS should point to itself).
8. Once verifying that AD is replicating across sites properly (up to 15 minutes in my experience), remove the old IP, ipconfig /registerdns, and reboot.
9. When it comes back up re-verify that AD is still replicating, and you should be set.

I would point out that when doing a change this big to your environment, reviewing your AD replication, DNS forwarding, and WINS topology is a good idea.

1. Why are you changing the subnet? Most often, I’ve had to do this for customers because a business partner’s subnet has conflicts that are coming up during VPN tunnel creation. I’ve had other reasons, and you might too – but that’s the most common I’ve seen – 2 sites that are both 192.168.1.0/24 trying to build a VPN tunnel to each other.
2. Now that you have a good reason to go through the pain, determine your layout – I’m going to write this for a single DC environment and point out a few changes for a double-DC environment. If you have more, you should be able to extrapolate the requirements from there, but you can also leave some comments and I can write another post if required. I’ll also be writing cmd scripts for most changes, rather than attaching huge (pixel-size) images.
3. Determine the new subnet.
4. Add the reverse lookup zone to DNS for your new network. In this case, in the DNS wizard, you’d make the reverse zone “172.31.2″ (filling in all boxes). This makes the zone name “2.31.172.in-addr.arpa” or “172.31.2.x” in the DNS console.
5. Open Active Directory Sites and Services. Right-Click “Subnets” and add in the new subnet (172.31.2.0 with a Subnet Mask of 255.255.255.0). You’ll see the subnet listed as CIDR notation in the box (172.31.2.0/24) for verification. Pick the site that the new subnet belongs in (probably Default-First-Site-Name), and click “OK”. This guarnantees that AD will recognize the new site properly.
6. Determine the new IP for the DC, and *add* it to the DC’s adapter. Do NOT remove the existing IP, yet. Don’t add the new gateway, yet. Just add the new IP, and save the settings.
   netsh interface ip add address "local area connection" 172.31.2.2 255.255.255.0
   (This command assumes that your DC has a static IP address. I haven’t seen a site use DHCP for their DCs yet, but it’s a possibility. This command on a DHCP address will set only a SINGLE address with no gateway, and could leave you with 0 remote access to the server.)
7. Verify that the new IP is showing up on the server.
   ipconfig
8. Get the server to register its new info in DNS
   ipconfig /registerdns
9. Wait and watch the Application and System event logs for DNS related errors. Also check that the DNS server is publishing the new IP address, not just as the server name, but also as the domain name, and the DCs (This can take up to, but shouldn’t take longer than, 15 minutes):
   nslookup foobar.local
   Also, open the DNS console and look inside “_msdcs.foobar.local” to see that the GUID of the server is listed with both IPs.
10. Once the server is advertising the new IP, you can swap the system to use the new IP range completely. It’s not time to remove the old IP yet though. This is the time to change the gateway, DNS servers, and WINS server. Since the server is listening on, and advertising on, the old and new IPs, DNS shouldn’t have any issues.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.1 10
netsh interface ip add address "local area connection" 192.168.1.2 255.255.255.0
netsh interface ip set dns "local area connection" static 172.31.2.2
netsh interface ip set wins "local area connection" static 172.31.2.2
    This can, like all other steps, also be done in the GUI quite easily, by just shifting the orders of some things (IPs), and replacing others (DNS/WINS/Gateway).
11. Now re-register the server with itself, looking for error messages in the Application and System logs.
    ipconfig /registerdns
12. Remember that changes to DNS can take up to 15 minutes to appear, as you watch the logs for errors, and check dns.
    nslookup foobar.local
    What you’re looking for in the nslookup is to see the address of every domain controller in your domain. If this is a single server (like a Small Business Server 2003 network), you should, at this point, see 2 or 3 addresses (depending on how you set up your public network, for SBS2003).
13. If everything looks good here, this is a good time to test some logons to make sure things are working properly. This is the point in the project where I normally create the new DHCP scope, deactivate (not delete) the old scope, and change the LAN settings on the router. This is also a good time to reboot some of the client PCs to make sure that they can boot up properly, get IP addresses in the new subnet, login without errors (remember to check that Application log!), and get online without issues.
    Because we haven’t removed the old IP from the server yet, the biggest issue you *should* run into is a client who gets an IP from the old subnet, or is statically set. They’ll log in ok, but won’t be able to get to the internet (unless you’ve got SBS2003, and the server is also your router). At this point in time, it’ll be easy to figure out if the new IP is working – clients that can log in and get access to resources are logging in to the IP address for the DC that’s local to them – if they’re in the new subnet, then your new IP is working. You can now reboot your DC as a final test (or act of faith, as your experience may prompt).
14. When the DC comes back up, log in, and remove the old IP address – this is easiest in the GUI, but if you’re doing it with netsh, I prefer to just reset the DC settings completely.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.2 10
ipconfig /registerdns
nslookup foobar.local
    Now it’s cleanup time.
15. Open Active Directory Sites and Services. Delete the old subnet.
16. Open up your DNS server and make sure your forwarders are correct – lots of small offices skip this step – your forwarders should be the IP addresses of the DNS servers that your ISP gave you. Don’t put these into your DHCP scope DNS servers list, or even in the list of DNS servers on your server – they go in the “forwarders” section here.
17. That’s it. Your clients are set up and ready to go, your server is healthy, and now you get to tackle the problem that made you have to change the IP in the first place!

For those who are having issues, because they’re finding this after attempting the change, or for my friend who prompted me to write this, here are some suggestions:

1. Add the old IP back to the server, run <code>ipconfig /registerdns</code> and wait for the old IP to take. Now verify that the server’s logging in properly, and not giving error messages.
2. Do you have all the zones in DNS, so that registration can take? Many small sites forget to put in the reverse zones into the DNS server.
3. Go back and check Active Directory Sites and Services to verify that both the old and new subnets are listed. While you’re in transition, all subnets should be listed in all locations.
4. If your server is screwed up badly enough that you can’t even log in, boot into Active Directory Services Restore Mode, make sure that your networking changes are set properly, and that your DNS server has it’s forwarders set up right. This is a good way to check the base level of your server’s health, then bring it back online to try to log into AD.
5. There’s no reason to need to rebuild the server, but if it’s a small enough location (AD doesn’t change very often, or if it does, the changes are minor), and you have good backups, take a look at how long this is taking you, and how much longer you’ll spend attempting to fix the problem. Going back to the backup from the previous night and restoring ONLY the system state in Directory Services Restore Mode might be the fastest and best solution. Then you can follow the steps in here in order, and grow fewer ulcers.

Next week sometime I’ll do a similar version for multi-DC, multi-Site AD networks – it’s a lot shorter and easier.

Documentation on [LB::server name] has been updated. It used to say that [LB::server name] would return the name of the pool member that the client was attached to. I had to open a ticket saying it doesn’t work, and ask why not. Their answer was: yeah, the documentation is wrong. So how to get the feature I want – the pool member name in the title, rather than it’s private IP?

Right now, the only way to get the name of the pool member that the client is pinned to, is [NS::lookup [LB::server addr]]. Depending on utilitzation, however, since you’re rewriting every single page hit, and Linux doesn’t have a client-side caching resolver like Windows does, you might have to dedicate a DNS server just to the F5 for this (I would have to). Or use /etc/hosts. I am NOT a fan of putting anything in /etc/hosts, but for now, I may have to for this system.

Update coming when I get a better fix.

Technically, a co-worker was doing the upgrade, so I don’t have full details (I did review the CR and steps), but he was doing the “local network install” via crossover cable into the management interface, so there should have been no issues, since the documentation states that’s OK. But we still completely ruined the box. Thank God no impact to the business, due to the HA pair working properly.

Ran the upgrade on the replacement box today with the “console cable” trick from the support guys, saw an error, fixed it, and everything went smoothly. Now to failover from 9.2.3 to 9.4.1 and reap the benefits of administrative domains! (And teach the other IT groups how to put their own stuff in maintenance/disabled mode.)

Edit: due to several searches ending up here:
Connect with a Null Modem cable, set: 9600,8n1. Make sure that your laptop’s serial port is actually working, by testing with a Cisco device or similar – we have 2 laptops that didn’t get any data on the 9.2.3 OS, but did on 9.1.2, changing laptops (or OS in my case) got us back online.