HowTo


Previously I mentioned some issues I had been having on Kubuntu Feisty Fawn with disk utilization seemingly caused by unflushed disk buffers. I alluded to believing that my “laptop-mode.conf” parameters were at fault.

With my recent upgrade of that same laptop to Kubuntu Gutsy Gibbon, I kept the laptop-mode.conf file a bit closer to the maintaner’s version. There are some changes to the “dirty-writeback-centiseconds” and the “dirty-background-ratio” values from what I posted, and my issue seems to have gone away. I’ve been able to go back to running my Windows 2003 SBS server with a Centrify DirectControl lab environment and a RHEL 4 Oracle 10g server attached at the same time.

The configuration files that work MUCH better are attached here:

laptop-mode.conf

cpufreqd.conf

First, reference back to my first post on Domain Controller IP/Subnet changes. The nice thing about changing IP addresses on DCs in a larger environment, is that it’s actually easier. I have to keep this one quick for now, but will expand based on comments, which you all seem pretty good at leaving (and thank you!). Please, PLEASE refer back to the first post – this one is only an expansion on that one.

  1. Same as before: why are you changing IPs? In larger environments, I do this because of a physical move of just one site. If the networking team doesn’t have the new subnet up and routing, don’t start!
  2. Make sure the new site (if required) is set up in AD. If I’m moving DCs from one physical location to another, I will build a new site, rather than re-using the old one, because the new site often has better connectivity, so the site link costs are changing.
  3. Add the new IP to the DC you’re moving (DC01 for this). Same as before: don’t remove the old one, just add the new.
  4. On DC01, do the following to verify registration worked:
    ipconfig /registerdns
    Wait a few minutes.
    nslookup
    server DC01
    set type=A
    DC01.foobar.local
    foobar.local
    server DC02
    DC01.foobar.local
    foobar.local

    The answers from DC01 and DC02 should be the same, with possibly different orders. The important thing is that the new IP address and the old IP address show up for both queries on both servers.
  5. Shut down DC01, pack it up and move it. (Or just plug it into the new network.)
  6. Boot up, verify that DC01 has network connectivity, and that other systems can see that it has the new IP.
  7. If you haven’t, make the new IP primary (change order in Network settings), make sure the DNS and WINS servers are correct and reachable (Remember that Windows 2003 DNS should point to itself).
  8. Once verifying that AD is replicating across sites properly (up to 15 minutes in my experience), remove the old IP, ipconfig /registerdns, and reboot.
  9. When it comes back up re-verify that AD is still replicating, and you should be set.

I would point out that when doing a change this big to your environment, reviewing your AD replication, DNS forwarding, and WINS topology is a good idea.

On my Ubuntu Dell D620 laptop (which dual-boots into Windows XP occasionally), I run some pretty demanding software. Sometimes. As a systems architect, I spend a lot of time in standard “Information Worker” tools – email, office suite, web browser for white papers and product reviews. For me, OpenOffice.org, Evolution, and Firefox work great. I even like Evolution’s Exchange plugin better than Outlook – everything updates on my Windows Mobile phone just like Outlook, it’s faster than Outlook, and it threads my messages. I love message threading for all the reasons that it was invented.

However, because I mostly design Windows networks, I also run VMWare Workstation 6 with the following VMs: Windows XP Pro joined to primary domain (full workstation with Office 2003, Visio, Outlook (and EMC EmailExtender plugins), Windows Admin Pack, Resource Kit, SQL Enterprise Manager, and Exchange Admin Pack), Windows Vista Enterprise, Windows 2003 DC, Exchange, SQL server, Windows XP Pro joined to VM domain, RHEL 4, RHEL 5, and a Live-CD system that I often use to test bootable CDs. And a sysprep’d Windows install.

All that, plus Evolution caching my email and Firefox being disk-happy (did you ever “strace -p $(pgrep firefox) -e trace=open,close,read,write”? It’s busier than Evolution during an offline mail check!), means my 7200 RPM 160GB SATA disk gets hammered. Me also using laptop-mode-tools changing my vm.dirty_writeback_* settings and read/write cache isn’t helping either, I’m sure.

Today and yesterday I ran into an issue where my disk would begin a sync that would last 10-20 minutes, leaving me unable to work the entire time that was happening. Hunting down WHAT was causing this, however, was even more frusterating than it happening (If I shut down any of the 3 above-mentioned programs, the problem went away – it only happened with all 3 open). In Windows, you can open Task Manager, go to the “Process” tab, click “View-> Columns” and add “IO Write Bytes” and “IO Read Bytes” and watch the numbers count up. Or you can use Perfmon and look at IO reads/writes/bytes per second or total, and know immediately what’s causing all your disk IO pain. I still don’t know how to do this in Linux.

First, any hunt for “disk utilization” and “Linux” on Google directs you to hundreds of sites, forums, and blogs evangelizing the wonders of “df” for disk utilization. Yes, it’s really nice to know how much free space I have on my hard drives- that’s why I have SuperKaramba to tell me. But when a problem hits and leaves me unable to work, it’s useless.

“iostat -k 1″ is great – you’ll know immediately which disk is being used, and how hard. But on a laptop with a single disk, you already know.

“top” sorted by process-state will show you what’s in “waiting on IO” state, but not what’s CAUSING the IO that’s causing everything else to wait.

“sar” seems to be the only tool that can provide per-process IO stats, but it has to be pre-set up to write to a log. And I can’t begin to guess how well that will work when my disk is at 100% utilization (peaked at 120tps today).

So if anyone knows of any way to know what’s causing disk IO in a “right now” fashion, please comment or email me. And if you’re curious more about my problem:

  1. Only happens when VMWare (with a guest), firefox, and Evolution are all running.
  2. VMWare with multiple guests runs fine, and never has this issue.
  3. rauch@lt00-bofh:~$ free
    total used free shared buffers cached
    Mem: 3348960 1099216 2249744 0 68668 533124
    -/+ buffers/cache: 497424 2851536
    Swap: 6000268 0 6000268
  4. Happens with Laptop_mode disabled or enabled, on AC or on battery
  5. “sync” causes the exact same symptoms, leading me to believe that somehow I’m getting a LOT more dirty pages than my parameters are set at.
  6. dirty_background_ratio
    1
    dirty_expire_centisecs
    60003
    dirty_ratio
    60
    dirty_writeback_centisecs
    60003
  7. For now, I just close Firefox when I have VMWare open, which means I spend a lot more time in IE than I want to.

As a final note, I’ve updated my Linux EVDO post here with my new built-in card’s info.

I’ve used this configuration, with minor tweaks on 3 different laptops, with 3 different OSes (if not 4) with great success. There’s so little good Linux info on evdoforums.com (at least I have a hard time finding it), and the posts I made link to a site that’s non-existant now, so I realized I had to repost this info. I’ve had both the Merlin S620 Sprint PCS EVDO card, and the Sierra Mobile AirCard 575, and now have a Dell built-in EVDO modem.

For all 3 of them, the only difference was the modprobe line, for vendor and product ID, as noted below.
Merlin S620:
modprobe usbserial vendor=0x1410 product=0x1110
Sierra 575:
modprobe usbserial vendor=0x1199 product=0x0019
Dell Sprint 5720 PCI-Express Modem
modprobe usbserial vendor=0x413c product=0x8134

I saved the appropriate PPP files at /etc/ppp/peers/1xevdo and at /etc/chatscripts/1xevdo_chat. In this post, I’ve blanked out the parts that are particular to my install (phone number), but it should be pretty easy to recreate your settings.

/etc/ppp/peers/1xevdo:

-detach
ttyUSB0
115200
debug
noauth
defaultroute
usepeerdns
user $(full-phone-number)@sprintpcs.com
show-password
crtscts
lock
lcp-echo-failure 4
lcp-echo-interval 65535
connect '/usr/sbin/chat -v -t3 -f /etc/chatscripts/1xevdo_chat'

/etc/chatscripts/1xevdo_chat:

'' 'AT'
'OK' 'ATZ'
'OK' 'ATE0V1&F&D2&C1&C2S0=0'
'OK' 'ATE0V1'
'OK' 'ATS7=60'
'OK' 'ATDT#777'
CONNECT CLIENT

Since I do several bits of work through the console, including accessing my Cisco VPN, and in some cases naim and tmsnc (console AOL and MSN chat) inside a screen session, scripts for these setups work great for me. I wrapped the whole thing up inside a bash script called $HOME/bin/evdo.sh – and I just call that when I want to get online, after inserting the card.
~/bin/evdo.sh:

sudo /sbin/modprobe usbserial vendor=0x1199 product=0x0019
sleep 5
sudo /usr/sbin/pppd call 1xevdo

The sleep statement helps make sure that the modprobe has completed, scanned the device, and settled before calling PPP.

My Dell D620 arrived, and I was able to quickly determine the changes for an “always there” card, vs. a pluggable PCMCIA card.
First I created /etc/modprobe.d/usbserial with the line:
options usbserial vendor=0x413c product=0x8134
and added “usbserial” to the end of /etc/modules so that the card would always come up at boot (sudo lspci -v | less to find the exact product ID and vendor – I only have 2 “Dell” devices on my laptop). I set my radio kill switch to affect only my EVDO and bluetooth radios, letting the software (~/bin/rfkill.sh in Linux and the Dell software in Windows) handle the WiFi – I use WiFi all the time, but only want the battery-draining EVDO in a few specific instances. So I added “cat 0 > /sys/bus/pci/devices/0000\:03\:00.0\rf_kill” to my “evdo.sh” file to kill the wireless when I wanted to use EVDO – no need to ever have them both on.

A friend came to me this week with an issue he had earlier – he had to change the subnet of a client’s network, and issues with the Windows 2003 Domain Controller appeared. Here’s how to avoid (or fix) the problem. For this HowTo, I’ll be using a current network of 192.168.1.2/24, with the server being it’s own DNS and WINS, and a gateway of 192.168.1.1. We’ll be changing the server to 172.31.2.2/24, and using the domain name “foobar.local”.

  1. Why are you changing the subnet? Most often, I’ve had to do this for customers because a business partner’s subnet has conflicts that are coming up during VPN tunnel creation. I’ve had other reasons, and you might too – but that’s the most common I’ve seen – 2 sites that are both 192.168.1.0/24 trying to build a VPN tunnel to each other.
  2. Now that you have a good reason to go through the pain, determine your layout – I’m going to write this for a single DC environment and point out a few changes for a double-DC environment. If you have more, you should be able to extrapolate the requirements from there, but you can also leave some comments and I can write another post if required. I’ll also be writing cmd scripts for most changes, rather than attaching huge (pixel-size) images.
  3. Determine the new subnet.
  4. Add the reverse lookup zone to DNS for your new network. In this case, in the DNS wizard, you’d make the reverse zone “172.31.2″ (filling in all boxes). This makes the zone name “2.31.172.in-addr.arpa” or “172.31.2.x” in the DNS console.
  5. Open Active Directory Sites and Services. Right-Click “Subnets” and add in the new subnet (172.31.2.0 with a Subnet Mask of 255.255.255.0). You’ll see the subnet listed as CIDR notation in the box (172.31.2.0/24) for verification. Pick the site that the new subnet belongs in (probably Default-First-Site-Name), and click “OK”. This guarnantees that AD will recognize the new site properly.
  6. Determine the new IP for the DC, and *add* it to the DC’s adapter. Do NOT remove the existing IP, yet. Don’t add the new gateway, yet. Just add the new IP, and save the settings.
    netsh interface ip add address "local area connection" 172.31.2.2 255.255.255.0
    (This command assumes that your DC has a static IP address. I haven’t seen a site use DHCP for their DCs yet, but it’s a possibility. This command on a DHCP address will set only a SINGLE address with no gateway, and could leave you with 0 remote access to the server.)
  7. Verify that the new IP is showing up on the server.
    ipconfig
  8. Get the server to register its new info in DNS
    ipconfig /registerdns
  9. Wait and watch the Application and System event logs for DNS related errors. Also check that the DNS server is publishing the new IP address, not just as the server name, but also as the domain name, and the DCs (This can take up to, but shouldn’t take longer than, 15 minutes):
    nslookup foobar.local
    Also, open the DNS console and look inside “_msdcs.foobar.local” to see that the GUID of the server is listed with both IPs.
  10. Once the server is advertising the new IP, you can swap the system to use the new IP range completely. It’s not time to remove the old IP yet though. This is the time to change the gateway, DNS servers, and WINS server. Since the server is listening on, and advertising on, the old and new IPs, DNS shouldn’t have any issues.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.1 10
    netsh interface ip add address "local area connection" 192.168.1.2 255.255.255.0
    netsh interface ip set dns "local area connection" static 172.31.2.2
    netsh interface ip set wins "local area connection" static 172.31.2.2

    This can, like all other steps, also be done in the GUI quite easily, by just shifting the orders of some things (IPs), and replacing others (DNS/WINS/Gateway).
  11. Now re-register the server with itself, looking for error messages in the Application and System logs.
    ipconfig /registerdns
  12. Remember that changes to DNS can take up to 15 minutes to appear, as you watch the logs for errors, and check dns.
    nslookup foobar.local
    What you’re looking for in the nslookup is to see the address of every domain controller in your domain. If this is a single server (like a Small Business Server 2003 network), you should, at this point, see 2 or 3 addresses (depending on how you set up your public network, for SBS2003).
  13. If everything looks good here, this is a good time to test some logons to make sure things are working properly. This is the point in the project where I normally create the new DHCP scope, deactivate (not delete) the old scope, and change the LAN settings on the router. This is also a good time to reboot some of the client PCs to make sure that they can boot up properly, get IP addresses in the new subnet, login without errors (remember to check that Application log!), and get online without issues.
    Because we haven’t removed the old IP from the server yet, the biggest issue you *should* run into is a client who gets an IP from the old subnet, or is statically set. They’ll log in ok, but won’t be able to get to the internet (unless you’ve got SBS2003, and the server is also your router). At this point in time, it’ll be easy to figure out if the new IP is working – clients that can log in and get access to resources are logging in to the IP address for the DC that’s local to them – if they’re in the new subnet, then your new IP is working. You can now reboot your DC as a final test (or act of faith, as your experience may prompt).
  14. When the DC comes back up, log in, and remove the old IP address – this is easiest in the GUI, but if you’re doing it with netsh, I prefer to just reset the DC settings completely.
    netsh interface ip set address "local area connection" static 172.31.2.2 255.255.255.0 172.31.2.2 10
    ipconfig /registerdns
    nslookup foobar.local

    Now it’s cleanup time.
  15. Open Active Directory Sites and Services. Delete the old subnet.
  16. Open up your DNS server and make sure your forwarders are correct – lots of small offices skip this step – your forwarders should be the IP addresses of the DNS servers that your ISP gave you. Don’t put these into your DHCP scope DNS servers list, or even in the list of DNS servers on your server – they go in the “forwarders” section here.
  17. That’s it. Your clients are set up and ready to go, your server is healthy, and now you get to tackle the problem that made you have to change the IP in the first place!

For those who are having issues, because they’re finding this after attempting the change, or for my friend who prompted me to write this, here are some suggestions:

  1. Add the old IP back to the server, run <code>ipconfig /registerdns</code> and wait for the old IP to take. Now verify that the server’s logging in properly, and not giving error messages.
  2. Do you have all the zones in DNS, so that registration can take? Many small sites forget to put in the reverse zones into the DNS server.
  3. Go back and check Active Directory Sites and Services to verify that both the old and new subnets are listed. While you’re in transition, all subnets should be listed in all locations.
  4. If your server is screwed up badly enough that you can’t even log in, boot into Active Directory Services Restore Mode, make sure that your networking changes are set properly, and that your DNS server has it’s forwarders set up right. This is a good way to check the base level of your server’s health, then bring it back online to try to log into AD.
  5. There’s no reason to need to rebuild the server, but if it’s a small enough location (AD doesn’t change very often, or if it does, the changes are minor), and you have good backups, take a look at how long this is taking you, and how much longer you’ll spend attempting to fix the problem. Going back to the backup from the previous night and restoring ONLY the system state in Directory Services Restore Mode might be the fastest and best solution. Then you can follow the steps in here in order, and grow fewer ulcers.

Next week sometime I’ll do a similar version for multi-DC, multi-Site AD networks – it’s a lot shorter and easier.

« Previous Page