totalnetsolutions.net

I ran into a problem today where I couldn’t remember the native packet capture tool for Solaris and couldn’t install tcpdump, so i thought I’d put down as many as many native packet capture commands as I knew, by OS, in a single place.  I’ll update this as I find more, since there’s hundreds of Operating systems out there.

• AIX: iptrace: /usr/sbin/iptrace [ -a ] [ -b ][ -e ] [ -u ] [ -PProtocol_list ] [ -iInterface ] [ -pPort_list ] [ -sHost [ -b ] ] [ -dHost ] [ -L Log_size ] [ -B ] [ -T ] [ -S snap_length] LogFile
• FreeBSD: tcpdump (I think): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• HP-UX: nettl: nettl requires a daemon start, and other setup: /usr/sbin/nettl -traceon kind… -entity subsystem… [-card dev_name...] [-file tracename] [-m bytes] [-size portsize] [-tracemax maxsize] [-n num_files] [-mem init_mem [max_mem]] [-bind cpu_id] [-timer timer_value]
• Linux 2.4 and higher:
  • tcpdump (some distros): tcpdump [ -AdDefKlLnNOpqRStuUvxX ] [ -c count ] [ -C file_size ] [ -G rotate_seconds ] [ -F file ] [ -i interface ] [ -m module ] [ -M secret ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ] [ -E spi@ipaddr algo:secret,... ] [ -y datalinktype ] [ -z postrotate-command ] [ -Z user ] [ expression ]
  • wireshark (some distros, used to be called “ethereal”): GUI-config, no command-line, use tethereal (now tshark) for that
  • tshark: tshark [ -a <capture autostop condition> ] … [ -b <capture ring buffer option>] … [ -B <capture buffer size (Win32 only)> ]  [ -c <capture packet count> ] [ -C <configuration profile> ] [ -d <layer type>==<selector>,<decode-as protocol> ] [ -D ] [ -e <field> ] [ -E <field print option> ] [ -f <capture filter> ] [ -F <file format> ] [ -h ] [ -i <capture interface>|- ] [ -l ] [ -L ] [ -n ] [ -N <name resolving flags> ] [ -o <preference setting> ] … [ -p ] [ -q ] [ -r <infile> ] [ -R <read (display) filter> ] [ -s <capture snaplen> ] [ -S ] [ -t ad|a|r|d|e ] [ -T pdml|psml|ps|text|fields ] [ -v ] [ -V ] [ -w <outfile>|- ] [ -x ] [ -X <eXtension option>] [ -y <capture link type> ] [ -z <statistics> ] [ <capture filter> ]
• Mac OSX: tcpdump (among others): tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -E algo:secret ] [ expression ]
• Solaris: snoop: snoop [ -aPDSvVNC ] [ -d device ] [ -s snaplen ] [ -c maxcount ] [ -i filename ] [ -o filename ] [ -n filename ] [ -t [ r | a | d ] ] [ -p first [ , last ] ] [ -x offset [ , length ] ] [ expression ]
• Windows 2000, XP, 2003, Vista, 2008 and beyond:
  • netmon (not installed by default): GUI config, filter creation info here
  • wireshark (available by download): GUI-config, no command-line

Any others anyone wants added (or corrected), just comment or email and I’ll update this.
(Edit 7/29/08 – change tcpdump link)
(Edit 10/13/08 – add tshark info, thanks Jefferson!, and wireshark on Windows)
(Edit 12/27/09 – update IBM iptrace man page link)

We’ve been having some server uptime/stability issues, and aren’t getting alerts from HP Systems Insight Manager (HP SIM) that the services are down (cause they’re not, they’re just not answering on HTTP). So I took a copy of “responder.pl” and put it into something I wrote for totalnetsolutions.net. What came out is actually pretty nice, easily configurable, and so far this week, very stable.

We haev this running ever 3 minutes from 3 systems: 1 Windows 2003, 1 Fedora Core 8, and 1 Kubuntu Gutsy Gibbon. Requires Net::SMTP, Config::INIFiles, LWP::UserAgent, and HTTP::Request. The only one that I’ve needed to download and install is Config::INIFiles on any of those 3 systems. But I do have LWP::Simple on all systems, so I’m not sure if you’ll need the last 2. This is my first published code other than 3 line bash scripts, so be kind in comments.

Feel free to take and use / improve / update this – I’d just appreciate if you’d let me know so I can update this version here. The parseIni() function checks that all “URL”s are in http://www.google.com format or http://64.233.167.99 format (it checks for http:// followed by text followed by what appears to be a valid TLD format, or it checks for http:// followed by an IP address). I have yet to add in the regex to look for a valid full URI, because I didn’t need that yet.

This is upgraded over responser.pl in that:

1. It will send to any number of SMTP recipients (comma-separated)
2. It will silence its alerting if *all* checked addresses are down. If the monitoring system gets unplugged from the network, it won’t attempt to send hundreds of alerts upon regaining access. Or if you’re testing from a DSL line, you won’t get alerts because the DSL line went down, but the actual target was up. The next version will have this as an option in the INI file.
3. It uses standard INI file formatting, rather than a parsed text file.
4. it runs out of the box (so to speak) on Windows (ActivePerl) or Linux (Fedora and Ubuntu both tested).
5. It has better inline documentation.

The major problem is that a minimum of 2 URLs are needed in the INI file for the full logic to work. You can get around this for small networks by adding in the DNS domain for one, and the IP address for the other.

Thanks, and please share any concerns or problems.

chk-web.pl

I’ve been very busy with clients over the past 2 weeks, troubleshooting Clustering problems, Exchange issues, and planning a new trust relationship, on top of normal maintenance and design. As I solve each issue, I’ll be posting what I can about them. This week we were able to solve the odd clustering problem…

We’ve seen some issues over the past approximately 2 months, particularly with MS SQL 2000 clusters (1 Exchange 2003 cluster), where the cluster group fails on one node, and the other node (or nodes) fails to pick up the group, leaving the complete cluster group offline. In each of the cases (on both HP and Dell hardware) the first striking piece of evidence in the logs is that all nodes that fail to bring up the cluster report that the Cluster IP Address resource couldn’t be brought online, because of an IP address conflict on the network

Making this issue particularly fun is that most of the information we used to solve the problem, is a lack of information.  In particular, there is absolutely nothing interesting at all in any nodes’ cluster.log file.  You see the disks negotiate from node to node, but nothing that makes the failover look any different than if you had right-clicked the group and chosen “Move Group” from Cluster Administrator.

What starts the problem off is Event ID 1228 from source “ClusNet”, which says that the “ClusNet driver couldn’t communicate with the ClusSvc for 60 seconds, the Cluster service is being terminated.” Most of the time, you might even miss that this event is there, because it causes so many Event Source Tcpip, ID 4199; Source ftdisk, ID 57; and Source ntfs event ID 50 events, that it’s easy to look over 1 little error. Especially when monitoring systems like Microsoft Operations Manager (MOM), or Idera SQLDiagnostics Manager (SQLDiag) or HP Systems Insight Manager (SIM) all report the cluster as having issues 30-60 seconds after the CluNet 1228 event is written (timing which corresponds exactly to the Tcpip 4199 events (IP address conflict) or the ftdisk 57 events (failed to flush transaction data).  So, here’s what happens, based on conversations with Microsoft, training with Microsoft and HP, and a LOT of reading. (more…)

First, reference back to my first post on Domain Controller IP/Subnet changes. The nice thing about changing IP addresses on DCs in a larger environment, is that it’s actually easier. I have to keep this one quick for now, but will expand based on comments, which you all seem pretty good at leaving (and thank you!). Please, PLEASE refer back to the first post – this one is only an expansion on that one.

1. Same as before: why are you changing IPs? In larger environments, I do this because of a physical move of just one site. If the networking team doesn’t have the new subnet up and routing, don’t start!
2. Make sure the new site (if required) is set up in AD. If I’m moving DCs from one physical location to another, I will build a new site, rather than re-using the old one, because the new site often has better connectivity, so the site link costs are changing.
3. Add the new IP to the DC you’re moving (DC01 for this). Same as before: don’t remove the old one, just add the new.
4. On DC01, do the following to verify registration worked:
   ipconfig /registerdns
   Wait a few minutes.
   nslookup
server DC01
set type=A
DC01.foobar.local
foobar.local
server DC02
DC01.foobar.local
foobar.local
   The answers from DC01 and DC02 should be the same, with possibly different orders. The important thing is that the new IP address and the old IP address show up for both queries on both servers.
5. Shut down DC01, pack it up and move it. (Or just plug it into the new network.)
6. Boot up, verify that DC01 has network connectivity, and that other systems can see that it has the new IP.
7. If you haven’t, make the new IP primary (change order in Network settings), make sure the DNS and WINS servers are correct and reachable (Remember that Windows 2003 DNS should point to itself).
8. Once verifying that AD is replicating across sites properly (up to 15 minutes in my experience), remove the old IP, ipconfig /registerdns, and reboot.
9. When it comes back up re-verify that AD is still replicating, and you should be set.

I would point out that when doing a change this big to your environment, reviewing your AD replication, DNS forwarding, and WINS topology is a good idea.

A friend came to me this week with an issue he had earlier – he had to change the subnet of a client’s network, and issues with the Windows 2003 Domain Controller appeared. Here’s how to avoid (or fix) the problem. For this HowTo, I’ll be using a current network of 192.168.1.2/24, with the server being it’s own DNS and WINS, and a gateway of 192.168.1.1. We’ll be changing the server to 172.31.2.2/24, and using the domain name “foobar.local”.

1. Why are you changing the subnet? Most often, I’ve had to do this for customers because a business partner’s subnet has conflicts that are coming up during VPN tunnel creation. I’ve had other reasons, and you might too – but that’s the most common I’ve seen – 2 sites that are both 192.168.1.0/24 trying to build a VPN tunnel to each other.
2. Now that you have a good reason to go through the pain, determine your layout – I’m going to write this for a single DC environment and point out a few changes for a double-DC environment. If you have more, you should be able to extrapolate the requirements from there, but you can also leave some comments and I can write another post if required. I’ll also be writing cmd scripts for most changes, rather than attaching huge (pixel-size) images.
3. Determine the new subnet.
4. Add the reverse lookup zone to DNS for your new network. In this case, in the DNS wizard, you’d make the reverse zone “172.31.2″ (filling in all boxes). This makes the zone name “2.31.172.in-addr.arpa” or “172.31.2.x” in the DNS console.
5. Open Active Directory Sites and Services. Right-Click “Subnets” and add in the new subnet (172.31.2.0 with a Subnet Mask of 255.255.255.0). You’ll see the subnet listed as CIDR notation in the box (172.31.2.0/24) for verification. Pick the site that the new subnet belongs in (probably Default-First-Site-Name), and click “OK”. This guarnantees that AD will recognize the new site properly.
6. Determine the new IP for the DC, and *add* it to the DC’s adapter. Do NOT remove the existing IP, yet. Don’t add the new gateway, yet. Just add the new IP, and save the settings.
   netsh interface ip add address "local area connection" 172.31.2.2 255.255.255.0
   (This command assumes that your DC has a static IP address. I haven’t seen a site use DHCP for their DCs yet, but it’s a possibility. This command on a DHCP address will set only a SINGLE address with no gateway, and could leave you with 0 remote access to the server.)
7. Verify that the new IP is showing up on the server.
   ipconfig
8. Get the server to register its new info in DNS
   ipconfig /registerdns
9. Wait and watch the Application and System event logs for DNS related errors. Also check that the DNS server is publishing the new IP address, not just as the server name, but also as the domain name, and the DCs (This can take up to, but shouldn’t take longer than, 15 minutes):
   nslookup foobar.local
   Also, open the DNS console and look inside “_msdcs.foobar.local” to see that the GUID of the server is listed with both IPs.
10. Once the server is advertising the new IP, you can swap the system to use the new IP range completely. It’s not time to remove the old IP yet though. This is the time to change the gateway, DNS servers, and WINS server. Since the server is listening on, and advertising on, the old and new IPs, DNS shouldn’t have any issues.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.1 10
netsh interface ip add address "local area connection" 192.168.1.2 255.255.255.0
netsh interface ip set dns "local area connection" static 172.31.2.2
netsh interface ip set wins "local area connection" static 172.31.2.2
    This can, like all other steps, also be done in the GUI quite easily, by just shifting the orders of some things (IPs), and replacing others (DNS/WINS/Gateway).
11. Now re-register the server with itself, looking for error messages in the Application and System logs.
    ipconfig /registerdns
12. Remember that changes to DNS can take up to 15 minutes to appear, as you watch the logs for errors, and check dns.
    nslookup foobar.local
    What you’re looking for in the nslookup is to see the address of every domain controller in your domain. If this is a single server (like a Small Business Server 2003 network), you should, at this point, see 2 or 3 addresses (depending on how you set up your public network, for SBS2003).
13. If everything looks good here, this is a good time to test some logons to make sure things are working properly. This is the point in the project where I normally create the new DHCP scope, deactivate (not delete) the old scope, and change the LAN settings on the router. This is also a good time to reboot some of the client PCs to make sure that they can boot up properly, get IP addresses in the new subnet, login without errors (remember to check that Application log!), and get online without issues.
    Because we haven’t removed the old IP from the server yet, the biggest issue you *should* run into is a client who gets an IP from the old subnet, or is statically set. They’ll log in ok, but won’t be able to get to the internet (unless you’ve got SBS2003, and the server is also your router). At this point in time, it’ll be easy to figure out if the new IP is working – clients that can log in and get access to resources are logging in to the IP address for the DC that’s local to them – if they’re in the new subnet, then your new IP is working. You can now reboot your DC as a final test (or act of faith, as your experience may prompt).
14. When the DC comes back up, log in, and remove the old IP address – this is easiest in the GUI, but if you’re doing it with netsh, I prefer to just reset the DC settings completely.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.2 10
ipconfig /registerdns
nslookup foobar.local
    Now it’s cleanup time.
15. Open Active Directory Sites and Services. Delete the old subnet.
16. Open up your DNS server and make sure your forwarders are correct – lots of small offices skip this step – your forwarders should be the IP addresses of the DNS servers that your ISP gave you. Don’t put these into your DHCP scope DNS servers list, or even in the list of DNS servers on your server – they go in the “forwarders” section here.
17. That’s it. Your clients are set up and ready to go, your server is healthy, and now you get to tackle the problem that made you have to change the IP in the first place!

For those who are having issues, because they’re finding this after attempting the change, or for my friend who prompted me to write this, here are some suggestions:

1. Add the old IP back to the server, run <code>ipconfig /registerdns</code> and wait for the old IP to take. Now verify that the server’s logging in properly, and not giving error messages.
2. Do you have all the zones in DNS, so that registration can take? Many small sites forget to put in the reverse zones into the DNS server.
3. Go back and check Active Directory Sites and Services to verify that both the old and new subnets are listed. While you’re in transition, all subnets should be listed in all locations.
4. If your server is screwed up badly enough that you can’t even log in, boot into Active Directory Services Restore Mode, make sure that your networking changes are set properly, and that your DNS server has it’s forwarders set up right. This is a good way to check the base level of your server’s health, then bring it back online to try to log into AD.
5. There’s no reason to need to rebuild the server, but if it’s a small enough location (AD doesn’t change very often, or if it does, the changes are minor), and you have good backups, take a look at how long this is taking you, and how much longer you’ll spend attempting to fix the problem. Going back to the backup from the previous night and restoring ONLY the system state in Directory Services Restore Mode might be the fastest and best solution. Then you can follow the steps in here in order, and grow fewer ulcers.

Next week sometime I’ll do a similar version for multi-DC, multi-Site AD networks – it’s a lot shorter and easier.