2007-10-08

How To: Change a Domain Controller IP address: Multi-DCs

Posted by Robert Auch under Domain Controllers, HowTo, Networking
[3] Comments 

First, reference back to my first post on Domain Controller IP/Subnet changes. The nice thing about changing IP addresses on DCs in a larger environment, is that it’s actually easier. I have to keep this one quick for now, but will expand based on comments, which you all seem pretty good at leaving (and thank you!). Please, PLEASE refer back to the first post – this one is only an expansion on that one.

  1. Same as before: why are you changing IPs? In larger environments, I do this because of a physical move of just one site. If the networking team doesn’t have the new subnet up and routing, don’t start!
  2. Make sure the new site (if required) is set up in AD. If I’m moving DCs from one physical location to another, I will build a new site, rather than re-using the old one, because the new site often has better connectivity, so the site link costs are changing.
  3. Add the new IP to the DC you’re moving (DC01 for this). Same as before: don’t remove the old one, just add the new.
  4. On DC01, do the following to verify registration worked:
    ipconfig /registerdns
    Wait a few minutes.
    nslookup
    server DC01
    set type=A
    DC01.foobar.local
    foobar.local
    server DC02
    DC01.foobar.local
    foobar.local
    The answers from DC01 and DC02 should be the same, with possibly different orders. The important thing is that the new IP address and the old IP address show up for both queries on both servers.
  5. Shut down DC01, pack it up and move it. (Or just plug it into the new network.)
  6. Boot up, verify that DC01 has network connectivity, and that other systems can see that it has the new IP.
  7. If you haven’t, make the new IP primary (change order in Network settings), make sure the DNS and WINS servers are correct and reachable (Remember that Windows 2003 DNS should point to itself).
  8. Once verifying that AD is replicating across sites properly (up to 15 minutes in my experience), remove the old IP, ipconfig /registerdns, and reboot.
  9. When it comes back up re-verify that AD is still replicating, and you should be set.

I would point out that when doing a change this big to your environment, reviewing your AD replication, DNS forwarding, and WINS topology is a good idea.

 

2007-10-05

Linux disk utilization

Posted by Robert Auch under HowTo, Linux
Leave a Comment 

On my Ubuntu Dell D620 laptop (which dual-boots into Windows XP occasionally), I run some pretty demanding software. Sometimes. As a systems architect, I spend a lot of time in standard “Information Worker” tools – email, office suite, web browser for white papers and product reviews. For me, OpenOffice.org, Evolution, and Firefox work great. I even like Evolution’s Exchange plugin better than Outlook – everything updates on my Windows Mobile phone just like Outlook, it’s faster than Outlook, and it threads my messages. I love message threading for all the reasons that it was invented.

However, because I mostly design Windows networks, I also run VMWare Workstation 6 with the following VMs: Windows XP Pro joined to primary domain (full workstation with Office 2003, Visio, Outlook (and EMC EmailExtender plugins), Windows Admin Pack, Resource Kit, SQL Enterprise Manager, and Exchange Admin Pack), Windows Vista Enterprise, Windows 2003 DC, Exchange, SQL server, Windows XP Pro joined to VM domain, RHEL 4, RHEL 5, and a Live-CD system that I often use to test bootable CDs. And a sysprep’d Windows install.

All that, plus Evolution caching my email and Firefox being disk-happy (did you ever “strace -p $(pgrep firefox) -e trace=open,close,read,write”? It’s busier than Evolution during an offline mail check!), means my 7200 RPM 160GB SATA disk gets hammered. Me also using laptop-mode-tools changing my vm.dirty_writeback_* settings and read/write cache isn’t helping either, I’m sure.

Today and yesterday I ran into an issue where my disk would begin a sync that would last 10-20 minutes, leaving me unable to work the entire time that was happening. Hunting down WHAT was causing this, however, was even more frusterating than it happening (If I shut down any of the 3 above-mentioned programs, the problem went away – it only happened with all 3 open). In Windows, you can open Task Manager, go to the “Process” tab, click “View-> Columns” and add “IO Write Bytes” and “IO Read Bytes” and watch the numbers count up. Or you can use Perfmon and look at IO reads/writes/bytes per second or total, and know immediately what’s causing all your disk IO pain. I still don’t know how to do this in Linux.

First, any hunt for “disk utilization” and “Linux” on Google directs you to hundreds of sites, forums, and blogs evangelizing the wonders of “df” for disk utilization. Yes, it’s really nice to know how much free space I have on my hard drives- that’s why I have SuperKaramba to tell me. But when a problem hits and leaves me unable to work, it’s useless.

“iostat -k 1″ is great – you’ll know immediately which disk is being used, and how hard. But on a laptop with a single disk, you already know.

“top” sorted by process-state will show you what’s in “waiting on IO” state, but not what’s CAUSING the IO that’s causing everything else to wait.

“sar” seems to be the only tool that can provide per-process IO stats, but it has to be pre-set up to write to a log. And I can’t begin to guess how well that will work when my disk is at 100% utilization (peaked at 120tps today).

So if anyone knows of any way to know what’s causing disk IO in a “right now” fashion, please comment or email me. And if you’re curious more about my problem:

  1. Only happens when VMWare (with a guest), firefox, and Evolution are all running.
  2. VMWare with multiple guests runs fine, and never has this issue.
  3. rauch@lt00-bofh:~$ free
    total used free shared buffers cached
    Mem: 3348960 1099216 2249744 0 68668 533124
    -/+ buffers/cache: 497424 2851536
    Swap: 6000268 0 6000268
  4. Happens with Laptop_mode disabled or enabled, on AC or on battery
  5. “sync” causes the exact same symptoms, leading me to believe that somehow I’m getting a LOT more dirty pages than my parameters are set at.
  6. dirty_background_ratio
    1
    dirty_expire_centisecs
    60003
    dirty_ratio
    60
    dirty_writeback_centisecs
    60003
  7. For now, I just close Firefox when I have VMWare open, which means I spend a lot more time in IE than I want to.

As a final note, I’ve updated my Linux EVDO post here with my new built-in card’s info.

 

2007-09-23

Australia and World Heritage Sites

Posted by Robert Auch under Personal
Leave a Comment 

Back in Sydney after visiting Port Douglas (dove the Great Barrier Reef and visited Daintree Rainforest National Park), and Uluru and Kata Tjuta National Park.  That makes 3 UNESCO (United Nations Education, Science, and Cultural Organization) World Heritage sites in 3 days. With the Greater Blue Mountains area near Sydney and the Sydney Opera House tomorrow, that’ll be 5 in just this vacation. I’ve only visited 3 in the US so far (Yellowstone, Statue of Liberty, and Redwood National Park).

Back in another week!

 

2007-09-12

Vacation Time

Posted by Robert Auch under Business, Personal
Leave a Comment 

Today is my last day on the job before starting a 2 week long vacation to Australia, visiting Sydney, Port Douglas, and Uluru.  I’ve been asked several times if I got a GSM phone to be able to take calls there in case something went horribly wrong at the office, and as a follow up, if Sprint has service out there (when they hear I’m not taking a GSM phone).

I make it a point in my work to make sure someone else can effectively back me up on all aspects of my work.  There are some people I’ve worked with who seem to think that if they are indispensable, then the company can’t fire them.  However, it also means, to me, that they can’t be promoted, can’t go on vacation, and can’t even have an evening at home with family.  So, when I design things, or fix something that broke, or make changes to make something work better, I make sure to include as many team members as I can, so that I can do things like take my wife out, and not be tied to my phone, worried that it may ring, even when I’m not officially on-call.

To that end, I spent a lot of time over the past few weeks giving a lot of history to the newer members of our team, so that they understand the decision making process that led us to the system state we’re at now.  Why do we have to reboot Terminal Servers every weekend?  Because of a memory leak in Windows 2000 that our application and settings trigger fast enough to require it.  Not just “which servers do we have to have up 24/7″, but why those servers, and not others, even if they’re in the same priority group.   This has been tremendously helpful to them in their day-to-day work, evidenced by the lower volume of questions they’re asking to other members of the team.

So, after all this work, how are things set?  Does everyone in the team have the exact same skillset at the same level as me?  No, because we’re different people.  Will it maybe take them a few minutes more to solve <insert particular problem here>?  Maybe, because I may be the most knowledgeable person on that application, but that doesn’t mean that they can’t fix it quickly.  So I spent half of the day today re-iterating those facts to people who are worried that the company will fail if I’m not here (it surely won’t – I’m not that important).

Now, then, off to vacation – I’ll write a blurb about it in 2 weeks, then a few days later about how busy I am catching up!

 

2007-09-08

Software Discovery: Centrify DirectControl

Posted by Robert Auch under Domain Controllers, Linux
Leave a Comment 

I just finished evaluating an excellent piece of software for Windows / Linux hybrid shops: Centrify Corporation’s DirectControl Suite. This is a fantastically well executed integration suite which allows administrators to bring their GNU/Linux and Unix boxes into the Windows ActiveDirectory domain. This brings centralized control of UID/GID (like NIS), the mutual authentication of Kerberos, and centralized Group Policy control to Linux/Unix.

First off, I’d like to mention that the software installs first on a Windows “console” system. That install has the option of extending the schema, but it is not required (the extensions allows administrators to use the Centrify Profile tab for users and computers without installing the Centrify Console locally).  All required pieces work with the standard out-of-the-box Windows 2003 AD schema.  Although the view extensions are well worth it, if you can get them approved by your AD administrative team.

I installed this on a Debian Etch system and a Red Hat Enterprise Linux 4 box.  They ship RPM and DEB installers, so installation is a snap, and shows up in your package manager.  Restarting the systems was not required, but a few systems may not pick up the new PAM settings without at least a reload (OpenSSH did fine).

One of the best parts of this software, however, is in their updated version of OpenSSH to support Windows Kerberos tickets for authentication of users.  Single-signon to any Linux box from Linux or Windows (customized Putty for the same reason) without having to copy RSA keys across your network every time you build a box.  Now my Oracle admins can log into the 10g databases seamlessly (yes, they support Oracle authenticating through AD as well).

Of course, no solution that integrates into AD would be complete without support for Group Policy.  As a huge user of Group Policy (I have 8 GPOs on my home domain), this is key for me.  The thing that makes it so spectacular, is that they just install new ADM files to your console system.  That’s it – no new trees needed, just new ADM files with settings specific to Linux like “SuDoers entries” and “SSH settings”.  Just like GPO on Windows, they’re applied every 90+-30 minutes, and when you remove the system from the policy, the settings get pulled.  For the Sudoers settings, they are appended to the end of the existing file.  Also, many of your security settings for Windows boxes are read directly by the Centrify systems as well, including password expiration notices, lockout policy handling, etc.

There are so many other little features that show how well thought-out the system is.  The client can be configured to cache logons similar to Windows, so you can control your Linux laptops, and still enable the users to log in when they’re on the road. There are several scripts and other tools to help “suck” the users out of /etc/passwd and NIS into AD, to help keep your UIDs in check if you’re installing the client into existing servers.

And that’s just the operating system.  JBoss, WebSphere, Apache and other applications and middleware can be AD-enabled, and anything that uses PAM is automatically AD-enabled, giving you the ability to set up true single sign-on everywhere in your network, if you so choose.

Needless to say, we purchased it, and I’ll be integrating this into all my deployments from this point forward.

 

« Previous Page — Next Page »