I’ve been doing the automatic upgrades on one of my other sites since they came out.  They’re easy, fast, and even more painless than the 3-step upgrade that works so well. So now, I should be able to keep TNS much further away from the “cobbler’s kids” syndrome so many small company’s systems suffer with.

After four hours on the phone I was able to determine the two tapes that would be needed to recover the 79Gb database file and started reading in the specific saveset that was needed.  Two hours later I was able to start a restore, which failed.  2Gb of the restore file was missing.  After another two hours on the phone with support I was told “Let’s reposition the tape.  It could take a while, on newer technology I’ve seen it take an hour, on LTO1 and LTO2 drives I’ve seen it take 8 hours.”  You guessed it, I have LTO2 drives.  Fortunately I have a multitude of drives to reposition the tapes with so it won’t impact backups, unfortunately I have a time limit on the restore that’s fast approaching.

So what do you do when you backup your file systems?  Do you simply believe that the software you backup with validates your tapes or do you test them regularly?  Are you satisfied with seeing an email at the end of a backup routine stating “SUCCESS”?  Then answer is simply NO.  Your backups are only as good as your ability to restore from them. Keeping that in mind and all the different technologies and services available what do you choose?

For us the answer is simple.  We require low cost, reliable, offsite secure data storage as do most companies nowadays.  TAPE.  We’ve looked into collocated services and replicated SANs with virtual tape backup but the cost far exceeds it’s benefits.  Tape technology has been proven over and over for decades.  There is no cost effective replacement for a good old fashioned tape, even taking into consideration the troubles it can give you.  Our entire datacenter can be put onto 6 tapes costing $25 each.  4.8TB for $150.

Any good backup initiative should be followed up with an equally adequate restore plan.  So next time you recommend a backup solution plan a regular restore plan to test because there’s nothing worse than spending an entire week restoring one file.

Don Jones had the next session, but I had to skip out on it to prepare for my session. I did hear some great feedback, but was disappointed, since his was one of the sessions I was most looking forward to prior to the conference start.

I spoke at 11am about how to integrate Linux/Unix systems with Active Directory. Download the deck here. It was a great experience, and the bit of feedback I’ve heard so far has been really positive.  It sounds like several attendees have moved their integration projects forward with information I presented, so I think it was successful.

After lunch, the Microsoft Windows and Active Directory product teams had a chalk talk about what’s next with AD where they solicited a LOT of suggestions from the attendees.  I was surprised by the number of people who are using “Prune and Graft” techniques for domain migrations.  Microsoft was very clear, however: do not EVER prune and graft domains.

And I’ll leave it at that.  In all, in was a great experience, and I learned so much.  I’m going to go back again!

Kevin Sullivan talked about the new “GP Preferences” and client side extensions available for WinXP and higher OSes, that will be available with the new Remote Server Administration Tool (RSAT). I’ve been looking for that download every day since they mentioned it. Still waiting…  but with some new security initiatives at one of my smaller clients, I’ll be using these soon as well!

Mark Foust from Microsoft had a great discussion on security audits and the most common problems found – they’re amazingly simple ones to fix, too, like cleaning up members of built-in groups such as “Schema Admins”, that even large enterprises miss.  A great reminder of where to start with security house cleaning.

We then had several “Birds of a Feather” sessions – I attended the Group Policy one lead by Kevin and Darren.  We had some great back-and-forth about how other companies are using Group Policy, how to do upgrades to the new Client-Side Extensions, and other GPO subjects.  There were some requests for better reporting than the RSoP provides, and a request for a way to dump RSoP reports into something that can be audited against, like with System Center 2007.  It’s not something I had noticed as missing in my environments, but the idea was mentioned, I realized how great it would be.  Hopefully Kevin has taken the idea back to Redmond for further review.

The evening was spent mostly in the Centrify hospitality suite, talking Linux/Windows interoperability with several other attendees, and an early trip home to rest up for the morning!

Sunday March 2nd was only registration and the reception for me. I just used the brief time downtown to meet with the Centrify and Likewise teams who worked so hard over the previous month to help me prepare my presentation for Wednesday. I met a bunch of great new contacts as well – not a conversation passed that I didn’t learn something new.

Monday the 3rd included Gil Kirkpatrick’s discussion on AD administrators vs. software developers, Jerry Camel and Brad Turner’s overview of proper architecture for ILM “2″, how Microsoft is using Windows Server 2008 (Brian Puhl), an amazingly indepth look into AD with Dean Wells and Joe Richards, and a discussion about how Centrify DirectControl works (in Centrify’s vendor track).

Gil Kirkpatrick covered things like mistakes that developers often make because they’re taught how to write well-constructed SQL queries, but not well-constructed LDAP queries. He discussed at great length 11 tips to help ensure that directory-integrated software performs as it should, without killing domain controller performance. The most interesting part, however, was his suggestions on how to talk with software developers so that both halves of the IT team can create a well-rounded product.

After lunch, Brian Puhl with Microsoft IT spoke at length about the rollout of Windows Server 2008 within Microsoft. He talked about the problems they encountered running a release candidate OS, and how their rollout process works, from the test domain to the “pre-production” forest of 5000 real users, to the “real” production forest. That they’re able to run in 2008 Forest mode already is impressive to me. The discussion of using RODCs (Read Only Domain Controllers) in DMZs and remote offices was also very cool.

Dean Wells and Joe Richards – if you ever get a chance to see them speak, take it. Not only do they know things about AD that nobody in the audience knew (and the attendees at DEC are *smart*), but they present really well – personality, humor, and great new info. They covered things like exactly what AdminSDHolder does, and how precisely the Infrastructure Master role works (down to the changes inside the DIT itself). They also had a few things to say about the Second City itself.

I spoke in Centrify’s vendor track about their DirectControl product. We had a decent turnout, considering it was a vendor-specific talk. Likewise Software, NetPro software, and OptimalIDM threw some pretty great parties after hours. It was interesting meeting people like Mark Foust, Mike Dube, and Stuart Kwan from Microsoft, Manny Vellon from Likewise Software, David McNeely from Centrify, and John Serban from WaMu, and talking to them about work and other things.

I’ll follow up on Days 2 and 3, including my presentation, in the next few posts.

Of note is a response for T. Colin Dodd regaring his short and sweet post regarding Red Hat Flaws according to Secunia. In short, Mr. Dodd (please correct me if the address is wrong), yes, Red Hat should be proud of what they’ve accomplished, but…

Well, that’s 2 pages of text that’s not yet finished.

1. We have restored from most recent backup, but are missing a single post, “PHP, mail(), Apache, and SELinux (FC7)”, which even google.com’s cache didn’t catch in full. I apologize to the readers who were using the instructions in that post whom we met through their comments.
2. We haven’t yet restored the “comments” table. I haven’t yet decided if we will.
3. I have fixed the problem of storing backups for the company in 3 different locations, based on system type. Now we only have 2 – onsite and offsite.
4. The extremely popular How to Change a DC IP address article was restored first. (That page drives over half of our traffic.)

We did a standard forensics review of what happened, and it appears as though a perfect storm of issues hit us – a weekend outage, a hardware failure, and failure to keep publicly exposed software fully up-to-date. The saying often goes, “The cobbler’s kids are the ones without shoes” or something similar to that, and here we failed to follow our own advice, preferring to keep our customers’ systems running smoothly. I know I’ll be spending a few extra hours a week the rest of this year reviewing our internal systems for best practices.

In any case, things are fixed and running great again.

I read both of these blogs, and I’m frankly disgusted by the way both sides are treating the data. I understand that statistics are often more useful for what they hide, than what they show. In this case, the 2 competing ideas seem to be: “We fix more bugs, which means we’re working harder to protect you”, vs. “we fix fewer bugs because we have fewer bugs, so we’re working harder to protect you”. I think both of these arguments are invalid, so I hope both sides see this and pay attention.

1. Jeff Jones: Jeff does a very interesting quarterly (or so) patch report – what OS’s have had the most patches applied in “xx” time frame (past quarter, past year, etc.). I get a lot of out this report, and he does very good trending. Find them on his blog and read them.To that end, he does a very good job selling Microsoft as a security company. By purely counting “number of patches submitted”, Microsoft will automatically look better, simply because “Windows (XP and 2003 combined)” has fewer features than “Red Hat Enterprise Linux” or “SUSE Enterprise Linux” or “Ubuntu Desktop Edition”.Jeff makes a point that Microsoft has only released patches for 649 security vulnerabilities across all Microsoft products in 7 years, but…What Windows does have that the GNU/Linux variants don’t have: .NET Framework, which is a HUGE project, but when it’s updated, you get a single update, so it counts as “1″ in Jeff’s analysis. Also, Microsoft doesn’t have conflicting software product lines – they have the Office team which has swallowed the “Works” team, but there are at least 3 “Office” suites in any GNU/Linux distro (OOo, koffice for KDE, and the suite including ABIWord for gnome).

   Then we can discuss kernels – when there is a driver update for a 3rd party product (Intel i810/845/945 motherboard, for example), it’s a module in the kernel, which requires an updated kernel package from the GNU/Linux distributors, but when there’s a driver update for a 3rd party application, Microsoft doesn’t even have to count it, since it’s “3rd party.” And on the subject of kernels, I don’t recall ever seeing an actual “kernel” update for Windows that wasn’t included in a service pack, or a box on a shelf.

2. Truth Happens writers: Selling “look how many bugs we fix” to a corporation is a pretty crappy way of doing business, in my opinion. That I can put an appointment in my calendar for 3pm the 2nd Tuesday of each month to review patches, test them that afternoon, and start rolling them out to QA the next morning, is a fantastic way to work. When Red Hat comes out with an update, it’s at a random time, and I have to review each one individually against what I may have installed on my systems.Now, this isn’t a dig against any GNU/Linux distribution out there – free (Ubuntu) or enterprise (Novell / Red Hat) – they are forced into this disclosure/fix model by the fact that these packages are not maintained solely by the companies that are pushing the fixes. In fact, in these cases, the patches have to be done on a “per-report” basis because of how most open-source software vulnerabilities are reported.This is a great time to ask: why is OOo included in a server distro? There *has* to be some GPL or package management reason behind it, but I’d be really interested to know.

So here we see 2 points of view: MS’s (Jeff Jones’) “we’re great because we don’t have a lot of patches, which means we’re more secure;” and RH’s (Truth Happens’) “we’re great because we’ve patched all of the bugs that have been found, no matter how small.” In truth, I think the real point should be that they are 2 completely different companies with huge differences in their offerings in the “Operating System” category. To have both representatives of both companies post what amount to “nyah nyah, we’re better than you are” blogs, keeps the entire discourse of security at a childish level that helps nobody.

So, to both Jeff and the writers of “Truth Happens”: please, out of respect for your readers, look deeper into the numbers and provide some insight, don’t just knock your competition.

I make it a point in my work to make sure someone else can effectively back me up on all aspects of my work.  There are some people I’ve worked with who seem to think that if they are indispensable, then the company can’t fire them.  However, it also means, to me, that they can’t be promoted, can’t go on vacation, and can’t even have an evening at home with family.  So, when I design things, or fix something that broke, or make changes to make something work better, I make sure to include as many team members as I can, so that I can do things like take my wife out, and not be tied to my phone, worried that it may ring, even when I’m not officially on-call.

To that end, I spent a lot of time over the past few weeks giving a lot of history to the newer members of our team, so that they understand the decision making process that led us to the system state we’re at now.  Why do we have to reboot Terminal Servers every weekend?  Because of a memory leak in Windows 2000 that our application and settings trigger fast enough to require it.  Not just “which servers do we have to have up 24/7″, but why those servers, and not others, even if they’re in the same priority group.   This has been tremendously helpful to them in their day-to-day work, evidenced by the lower volume of questions they’re asking to other members of the team.

So, after all this work, how are things set?  Does everyone in the team have the exact same skillset at the same level as me?  No, because we’re different people.  Will it maybe take them a few minutes more to solve <insert particular problem here>?  Maybe, because I may be the most knowledgeable person on that application, but that doesn’t mean that they can’t fix it quickly.  So I spent half of the day today re-iterating those facts to people who are worried that the company will fail if I’m not here (it surely won’t – I’m not that important).

Now, then, off to vacation – I’ll write a blurb about it in 2 weeks, then a few days later about how busy I am catching up!