July 2007


Recently my neighbors came to me with performance problems on their new notebook computer.  I told them I would be happy to look at their computer and resolve the poor performance (figuring it was loaded with ’special’ software from the manufacture and spyware).  What I found was quite disturbing to me.

I powered up the computer and was pretty happy with the boot time.  I was at the login screen (XP Home) in under a minute.  Then I clicked on a username.  After 12 minutes the desktop had not completely loaded, and preloaded software was popping up everywhere.  I proceeded to kill unnecessary apps and removing them from the startup routine.

After we were in business I started disk management to defragment the hard drive.  To my amazement the disk was zero percent fragmented (I almost fell off my chair).  I then downloaded and installed a reputable spyware removal tool and updated the definitions.  After an hour of the scan, and only getting through about one third of the scan I stopped it and noticed that the only items found were not spyware, tracking cookies, trojans, or adware, but were MRU items (gee, do people open documents on their computer?  Of course, that’s what ALL of these items were).  I cleaned up the MRU items and reboot the computer.

An almost identical boot process followed, including the 12 minutes to start the user session.  Thinking I was crazy, I reboot and choose the other username that was on the machine.  Another 12 minutes later (and a snifter of brandy) I started looking at the little sticker on the wrist rest and became angry.  1.7GHz Intel CPU, 80 GB Hard drive, 256MB Memory and a 64MB video card.  It doesn’t sound that bad…HOWEVER, remember this is a notebook computer that I have spent already two to three hours on which after looking closer only has 192MB of memory after the video card takes it’s 64MB.  With all of the apps that started the physical memory that was ‘available’ was right around 24MB…the memory used was around 400MB.  This poor computer (and consumer) was paging ALL of the memory.  The page file recommended size was 213, and was automatically resized to 527 (wow).

I am amazed that computers are available with this little memory considering what Microsoft has put out for an Operating System.  Not to mention the fact that the video card was consuming one fourth of the system memory.  Most users that have a notebook don’t care about the video functionality; they care about portability…especially if they only buy a computer with 256 MB memory.

My temporary resolution was to change the video card down to 32Mb, which is much more responsive, but still incredibly slow.  Hopefully they will heed my advice and drop in a 1GB module for a mere $45.

Paul

A friend came to me this week with an issue he had earlier – he had to change the subnet of a client’s network, and issues with the Windows 2003 Domain Controller appeared. Here’s how to avoid (or fix) the problem. For this HowTo, I’ll be using a current network of 192.168.1.2/24, with the server being it’s own DNS and WINS, and a gateway of 192.168.1.1. We’ll be changing the server to 172.31.2.2/24, and using the domain name “foobar.local”.

  1. Why are you changing the subnet? Most often, I’ve had to do this for customers because a business partner’s subnet has conflicts that are coming up during VPN tunnel creation. I’ve had other reasons, and you might too – but that’s the most common I’ve seen – 2 sites that are both 192.168.1.0/24 trying to build a VPN tunnel to each other.
  2. Now that you have a good reason to go through the pain, determine your layout – I’m going to write this for a single DC environment and point out a few changes for a double-DC environment. If you have more, you should be able to extrapolate the requirements from there, but you can also leave some comments and I can write another post if required. I’ll also be writing cmd scripts for most changes, rather than attaching huge (pixel-size) images.
  3. Determine the new subnet.
  4. Add the reverse lookup zone to DNS for your new network. In this case, in the DNS wizard, you’d make the reverse zone “172.31.2″ (filling in all boxes). This makes the zone name “2.31.172.in-addr.arpa” or “172.31.2.x” in the DNS console.
  5. Open Active Directory Sites and Services. Right-Click “Subnets” and add in the new subnet (172.31.2.0 with a Subnet Mask of 255.255.255.0). You’ll see the subnet listed as CIDR notation in the box (172.31.2.0/24) for verification. Pick the site that the new subnet belongs in (probably Default-First-Site-Name), and click “OK”. This guarnantees that AD will recognize the new site properly.
  6. Determine the new IP for the DC, and *add* it to the DC’s adapter. Do NOT remove the existing IP, yet. Don’t add the new gateway, yet. Just add the new IP, and save the settings.
    netsh interface ip add address "local area connection" 172.31.2.2 255.255.255.0
    (This command assumes that your DC has a static IP address. I haven’t seen a site use DHCP for their DCs yet, but it’s a possibility. This command on a DHCP address will set only a SINGLE address with no gateway, and could leave you with 0 remote access to the server.)
  7. Verify that the new IP is showing up on the server.
    ipconfig
  8. Get the server to register its new info in DNS
    ipconfig /registerdns
  9. Wait and watch the Application and System event logs for DNS related errors. Also check that the DNS server is publishing the new IP address, not just as the server name, but also as the domain name, and the DCs (This can take up to, but shouldn’t take longer than, 15 minutes):
    nslookup foobar.local
    Also, open the DNS console and look inside “_msdcs.foobar.local” to see that the GUID of the server is listed with both IPs.
  10. Once the server is advertising the new IP, you can swap the system to use the new IP range completely. It’s not time to remove the old IP yet though. This is the time to change the gateway, DNS servers, and WINS server. Since the server is listening on, and advertising on, the old and new IPs, DNS shouldn’t have any issues.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.1 10
    netsh interface ip add address "local area connection" 192.168.1.2 255.255.255.0
    netsh interface ip set dns "local area connection" static 172.31.2.2
    netsh interface ip set wins "local area connection" static 172.31.2.2

    This can, like all other steps, also be done in the GUI quite easily, by just shifting the orders of some things (IPs), and replacing others (DNS/WINS/Gateway).
  11. Now re-register the server with itself, looking for error messages in the Application and System logs.
    ipconfig /registerdns
  12. Remember that changes to DNS can take up to 15 minutes to appear, as you watch the logs for errors, and check dns.
    nslookup foobar.local
    What you’re looking for in the nslookup is to see the address of every domain controller in your domain. If this is a single server (like a Small Business Server 2003 network), you should, at this point, see 2 or 3 addresses (depending on how you set up your public network, for SBS2003).
  13. If everything looks good here, this is a good time to test some logons to make sure things are working properly. This is the point in the project where I normally create the new DHCP scope, deactivate (not delete) the old scope, and change the LAN settings on the router. This is also a good time to reboot some of the client PCs to make sure that they can boot up properly, get IP addresses in the new subnet, login without errors (remember to check that Application log!), and get online without issues.
    Because we haven’t removed the old IP from the server yet, the biggest issue you *should* run into is a client who gets an IP from the old subnet, or is statically set. They’ll log in ok, but won’t be able to get to the internet (unless you’ve got SBS2003, and the server is also your router). At this point in time, it’ll be easy to figure out if the new IP is working – clients that can log in and get access to resources are logging in to the IP address for the DC that’s local to them – if they’re in the new subnet, then your new IP is working. You can now reboot your DC as a final test (or act of faith, as your experience may prompt).
  14. When the DC comes back up, log in, and remove the old IP address – this is easiest in the GUI, but if you’re doing it with netsh, I prefer to just reset the DC settings completely.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.2 10
    ipconfig /registerdns
    nslookup foobar.local

    Now it’s cleanup time.
  15. Open Active Directory Sites and Services. Delete the old subnet.
  16. Open up your DNS server and make sure your forwarders are correct – lots of small offices skip this step – your forwarders should be the IP addresses of the DNS servers that your ISP gave you. Don’t put these into your DHCP scope DNS servers list, or even in the list of DNS servers on your server – they go in the “forwarders” section here.
  17. That’s it. Your clients are set up and ready to go, your server is healthy, and now you get to tackle the problem that made you have to change the IP in the first place!

For those who are having issues, because they’re finding this after attempting the change, or for my friend who prompted me to write this, here are some suggestions:

  1. Add the old IP back to the server, run <code>ipconfig /registerdns</code> and wait for the old IP to take. Now verify that the server’s logging in properly, and not giving error messages.
  2. Do you have all the zones in DNS, so that registration can take? Many small sites forget to put in the reverse zones into the DNS server.
  3. Go back and check Active Directory Sites and Services to verify that both the old and new subnets are listed. While you’re in transition, all subnets should be listed in all locations.
  4. If your server is screwed up badly enough that you can’t even log in, boot into Active Directory Services Restore Mode, make sure that your networking changes are set properly, and that your DNS server has it’s forwarders set up right. This is a good way to check the base level of your server’s health, then bring it back online to try to log into AD.
  5. There’s no reason to need to rebuild the server, but if it’s a small enough location (AD doesn’t change very often, or if it does, the changes are minor), and you have good backups, take a look at how long this is taking you, and how much longer you’ll spend attempting to fix the problem. Going back to the backup from the previous night and restoring ONLY the system state in Directory Services Restore Mode might be the fastest and best solution. Then you can follow the steps in here in order, and grow fewer ulcers.

Next week sometime I’ll do a similar version for multi-DC, multi-Site AD networks – it’s a lot shorter and easier.

We had to emergency move some SQL Server service account mailboxes to a new Exchange server tonight, and dump all the mail in the mailboxes. Since doing that, some of the servers have been randomly failing SOME of their jobs – it looked exactly like the issue described in FIX: SQL Mail Procedures May Intermittently Fail with Error 0×8004010f , but we’re running SQL 2000 SP4.

We fixed it by having the DBA run:
xp_startmail
xp_stopmail

Then re-running the jobs. It seems to have worked, but I’m curious how we got hit by a fixed bug. My only thought is that it must have something to do with SQL thinking we essentially truncated the mailbox.

On anonther note, I’m writing up full instructions for how to change the IP on a DC, particularly for small offices with only 1 or 2 DCs.  It’ll post by end of the weekend.

Today I had a continuation of a call with tech support.  The guy on the phone seems decent, so I ask, “based on what you see, what’s the best way to fix this problem?”

“Manually migrate all the data out of the system, then manually migrate it back into a new one, finally reconnect all the users to the new system.  It should only take a week.”

Sadly he’s right.  I’m still going to go one more shot at the “fix the corruption during an outage window” option first, since we’re moving OFF of this system by the end of the summer.

I have a new rule in my F5:

when HTTP_RESPONSE {
if { [regsub -all "<title>" [HTTP::payload] "<title>[LB::server addr] -" newdata] } {
HTTP::payload replace 0 [HTTP::payload length] $newdata
}
}

This is tagged to a virtual server with “Rechunk” enabled. Most of the rules I’ve seen on http://devcentral.f5.com are forcing clients to HTTP/1.0 and disabling chunking, but simply “rechunk”ing the stream seems to allow the F5 to read the data, and not break the app. This is one of 2 rules I have using this feature. Anyways, to the problem:

Documentation on [LB::server name] has been updated. It used to say that [LB::server name] would return the name of the pool member that the client was attached to. I had to open a ticket saying it doesn’t work, and ask why not. Their answer was: yeah, the documentation is wrong. So how to get the feature I want – the pool member name in the title, rather than it’s private IP?

Right now, the only way to get the name of the pool member that the client is pinned to, is [NS::lookup [LB::server addr]]. Depending on utilitzation, however, since you’re rewriting every single page hit, and Linux doesn’t have a client-side caching resolver like Windows does, you might have to dedicate a DNS server just to the F5 for this (I would have to). Or use /etc/hosts. I am NOT a fan of putting anything in /etc/hosts, but for now, I may have to for this system.

Update coming when I get a better fix.

Next Page »