Networking


A friend came to me this week with an issue he had earlier – he had to change the subnet of a client’s network, and issues with the Windows 2003 Domain Controller appeared. Here’s how to avoid (or fix) the problem. For this HowTo, I’ll be using a current network of 192.168.1.2/24, with the server being it’s own DNS and WINS, and a gateway of 192.168.1.1. We’ll be changing the server to 172.31.2.2/24, and using the domain name “foobar.local”.

  1. Why are you changing the subnet? Most often, I’ve had to do this for customers because a business partner’s subnet has conflicts that are coming up during VPN tunnel creation. I’ve had other reasons, and you might too – but that’s the most common I’ve seen – 2 sites that are both 192.168.1.0/24 trying to build a VPN tunnel to each other.
  2. Now that you have a good reason to go through the pain, determine your layout – I’m going to write this for a single DC environment and point out a few changes for a double-DC environment. If you have more, you should be able to extrapolate the requirements from there, but you can also leave some comments and I can write another post if required. I’ll also be writing cmd scripts for most changes, rather than attaching huge (pixel-size) images.
  3. Determine the new subnet.
  4. Add the reverse lookup zone to DNS for your new network. In this case, in the DNS wizard, you’d make the reverse zone “172.31.2? (filling in all boxes). This makes the zone name “2.31.172.in-addr.arpa” or “172.31.2.x” in the DNS console.
  5. Open Active Directory Sites and Services. Right-Click “Subnets” and add in the new subnet (172.31.2.0 with a Subnet Mask of 255.255.255.0). You’ll see the subnet listed as CIDR notation in the box (172.31.2.0/24) for verification. Pick the site that the new subnet belongs in (probably Default-First-Site-Name), and click “OK”. This guarnantees that AD will recognize the new site properly.
  6. Determine the new IP for the DC, and *add* it to the DC’s adapter. Do NOT remove the existing IP, yet. Don’t add the new gateway, yet. Just add the new IP, and save the settings.
    netsh interface ip add address "local area connection" 172.31.2.2 255.255.255.0
    (This command assumes that your DC has a static IP address. I haven’t seen a site use DHCP for their DCs yet, but it’s a possibility. This command on a DHCP address will set only a SINGLE address with no gateway, and could leave you with 0 remote access to the server.)
  7. Verify that the new IP is showing up on the server.
    ipconfig
  8. Get the server to register its new info in DNS
    ipconfig /registerdns
  9. Wait and watch the Application and System event logs for DNS related errors. Also check that the DNS server is publishing the new IP address, not just as the server name, but also as the domain name, and the DCs (This can take up to, but shouldn’t take longer than, 15 minutes):
    nslookup foobar.local
    Also, open the DNS console and look inside “_msdcs.foobar.local” to see that the GUID of the server is listed with both IPs.
  10. Once the server is advertising the new IP, you can swap the system to use the new IP range completely. It’s not time to remove the old IP yet though. This is the time to change the gateway, DNS servers, and WINS server. Since the server is listening on, and advertising on, the old and new IPs, DNS shouldn’t have any issues.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.1 10
    netsh interface ip add address "local area connection" 192.168.1.2 255.255.255.0
    netsh interface ip set dns "local area connection" static 172.31.2.2
    netsh interface ip set wins "local area connection" static 172.31.2.2

    This can, like all other steps, also be done in the GUI quite easily, by just shifting the orders of some things (IPs), and replacing others (DNS/WINS/Gateway).
  11. Now re-register the server with itself, looking for error messages in the Application and System logs.
    ipconfig /registerdns
  12. Remember that changes to DNS can take up to 15 minutes to appear, as you watch the logs for errors, and check dns.
    nslookup foobar.local
    What you’re looking for in the nslookup is to see the address of every domain controller in your domain. If this is a single server (like a Small Business Server 2003 network), you should, at this point, see 2 or 3 addresses (depending on how you set up your public network, for SBS2003).
  13. If everything looks good here, this is a good time to test some logons to make sure things are working properly. This is the point in the project where I normally create the new DHCP scope, deactivate (not delete) the old scope, and change the LAN settings on the router. This is also a good time to reboot some of the client PCs to make sure that they can boot up properly, get IP addresses in the new subnet, login without errors (remember to check that Application log!), and get online without issues.
    Because we haven’t removed the old IP from the server yet, the biggest issue you *should* run into is a client who gets an IP from the old subnet, or is statically set. They’ll log in ok, but won’t be able to get to the internet (unless you’ve got SBS2003, and the server is also your router). At this point in time, it’ll be easy to figure out if the new IP is working – clients that can log in and get access to resources are logging in to the IP address for the DC that’s local to them – if they’re in the new subnet, then your new IP is working. You can now reboot your DC as a final test (or act of faith, as your experience may prompt).
  14. When the DC comes back up, log in, and remove the old IP address – this is easiest in the GUI, but if you’re doing it with netsh, I prefer to just reset the DC settings completely.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.2 10
    ipconfig /registerdns
    nslookup foobar.local

    Now it’s cleanup time.
  15. Open Active Directory Sites and Services. Delete the old subnet.
  16. Open up your DNS server and make sure your forwarders are correct – lots of small offices skip this step – your forwarders should be the IP addresses of the DNS servers that your ISP gave you. Don’t put these into your DHCP scope DNS servers list, or even in the list of DNS servers on your server – they go in the “forwarders” section here.
  17. That’s it. Your clients are set up and ready to go, your server is healthy, and now you get to tackle the problem that made you have to change the IP in the first place!

For those who are having issues, because they’re finding this after attempting the change, or for my friend who prompted me to write this, here are some suggestions:

  1. Add the old IP back to the server, run <code>ipconfig /registerdns</code> and wait for the old IP to take. Now verify that the server’s logging in properly, and not giving error messages.
  2. Do you have all the zones in DNS, so that registration can take? Many small sites forget to put in the reverse zones into the DNS server.
  3. Go back and check Active Directory Sites and Services to verify that both the old and new subnets are listed. While you’re in transition, all subnets should be listed in all locations.
  4. If your server is screwed up badly enough that you can’t even log in, boot into Active Directory Services Restore Mode, make sure that your networking changes are set properly, and that your DNS server has it’s forwarders set up right. This is a good way to check the base level of your server’s health, then bring it back online to try to log into AD.
  5. There’s no reason to need to rebuild the server, but if it’s a small enough location (AD doesn’t change very often, or if it does, the changes are minor), and you have good backups, take a look at how long this is taking you, and how much longer you’ll spend attempting to fix the problem. Going back to the backup from the previous night and restoring ONLY the system state in Directory Services Restore Mode might be the fastest and best solution. Then you can follow the steps in here in order, and grow fewer ulcers.

Next week sometime I’ll do a similar version for multi-DC, multi-Site AD networks – it’s a lot shorter and easier.

I have a new rule in my F5:

when HTTP_RESPONSE {
if { [regsub -all "<title>" [HTTP::payload] "<title>[LB::server addr] -" newdata] } {
HTTP::payload replace 0 [HTTP::payload length] $newdata
}
}

This is tagged to a virtual server with “Rechunk” enabled. Most of the rules I’ve seen on http://devcentral.f5.com are forcing clients to HTTP/1.0 and disabling chunking, but simply “rechunk”ing the stream seems to allow the F5 to read the data, and not break the app. This is one of 2 rules I have using this feature. Anyways, to the problem:

Documentation on [LB::server name] has been updated. It used to say that [LB::server name] would return the name of the pool member that the client was attached to. I had to open a ticket saying it doesn’t work, and ask why not. Their answer was: yeah, the documentation is wrong. So how to get the feature I want – the pool member name in the title, rather than it’s private IP?

Right now, the only way to get the name of the pool member that the client is pinned to, is [NS::lookup [LB::server addr]]. Depending on utilitzation, however, since you’re rewriting every single page hit, and Linux doesn’t have a client-side caching resolver like Windows does, you might have to dedicate a DNS server just to the F5 for this (I would have to). Or use /etc/hosts. I am NOT a fan of putting anything in /etc/hosts, but for now, I may have to for this system.

Update coming when I get a better fix.

Despite what the documentation may or may not say, do not attempt to upgrade your BIGIP without a working, directly connected console cable. We bricked an LTM 6400 by performing the upgrade exactly as the steps stated, but without using a console cable to watch the output.

Technically, a co-worker was doing the upgrade, so I don’t have full details (I did review the CR and steps), but he was doing the “local network install” via crossover cable into the management interface, so there should have been no issues, since the documentation states that’s OK. But we still completely ruined the box. Thank God no impact to the business, due to the HA pair working properly.

Ran the upgrade on the replacement box today with the “console cable” trick from the support guys, saw an error, fixed it, and everything went smoothly. Now to failover from 9.2.3 to 9.4.1 and reap the benefits of administrative domains! (And teach the other IT groups how to put their own stuff in maintenance/disabled mode.)

Edit: due to several searches ending up here:
Connect with a Null Modem cable, set: 9600,8n1. Make sure that your laptop’s serial port is actually working, by testing with a Cisco device or similar – we have 2 laptops that didn’t get any data on the 9.2.3 OS, but did on 9.1.2, changing laptops (or OS in my case) got us back online.

« Previous Page