I have been invited to present at the Directory Experts Conference in Chicago in March, hosted by NetPro Computing, Inc.. I’ll be discussing how we recently integrated dozens of Linux servers into our 300+ server Windows 2000 Native Mode forest. I’m excited, but it’s taking away from the time to update a few things here I have in “unpublished” state.

Of note is a response for T. Colin Dodd regaring his short and sweet post regarding Red Hat Flaws according to Secunia. In short, Mr. Dodd (please correct me if the address is wrong), yes, Red Hat should be proud of what they’ve accomplished, but…

Well, that’s 2 pages of text that’s not yet finished.

Now that I have the system back online, I thought I’d post a quick “where we are” update for any regular readers:

  1. We have restored from most recent backup, but are missing a single post, “PHP, mail(), Apache, and SELinux (FC7)”, which even’s cache didn’t catch in full. I apologize to the readers who were using the instructions in that post whom we met through their comments.
  2. We haven’t yet restored the “comments” table. I haven’t yet decided if we will.
  3. I have fixed the problem of storing backups for the company in 3 different locations, based on system type. Now we only have 2 – onsite and offsite.
  4. The extremely popular How to Change a DC IP address article was restored first. (That page drives over half of our traffic.)

We did a standard forensics review of what happened, and it appears as though a perfect storm of issues hit us – a weekend outage, a hardware failure, and failure to keep publicly exposed software fully up-to-date. The saying often goes, “The cobbler’s kids are the ones without shoes” or something similar to that, and here we failed to follow our own advice, preferring to keep our customers’ systems running smoothly. I know I’ll be spending a few extra hours a week the rest of this year reviewing our internal systems for best practices.

In any case, things are fixed and running great again.

As I recently rebuilt my primary system (my laptop), I couldn’t find the backups for the site – it appears that I may have skipped them when backing up the laptop. However, I have all the content, it will just be a day or 2 to get everything re-imported. The short story is that the site got hacked and *all* content and code were wiped.

I saw this post from Jeff Jones over at Microsoft today. He mentions that Red Hat Enterprise Linux 4 recently patched their 1000th vulnerability, and provides a quote from Truth Happens(direct link to post), which is a Red Hat blog. I suggest you at least read Jeff’s post, since he quotes the relevant point of the Truth article.

I read both of these blogs, and I’m frankly disgusted by the way both sides are treating the data. I understand that statistics are often more useful for what they hide, than what they show. In this case, the 2 competing ideas seem to be: “We fix more bugs, which means we’re working harder to protect you”, vs. “we fix fewer bugs because we have fewer bugs, so we’re working harder to protect you”. I think both of these arguments are invalid, so I hope both sides see this and pay attention.

  1. Jeff Jones: Jeff does a very interesting quarterly (or so) patch report – what OS’s have had the most patches applied in “xx” time frame (past quarter, past year, etc.). I get a lot of out this report, and he does very good trending. Find them on his blog and read them.To that end, he does a very good job selling Microsoft as a security company. By purely counting “number of patches submitted”, Microsoft will automatically look better, simply because “Windows (XP and 2003 combined)” has fewer features than “Red Hat Enterprise Linux” or “SUSE Enterprise Linux” or “Ubuntu Desktop Edition”.Jeff makes a point that Microsoft has only released patches for 649 security vulnerabilities across all Microsoft products in 7 years, but…What Windows does have that the GNU/Linux variants don’t have: .NET Framework, which is a HUGE project, but when it’s updated, you get a single update, so it counts as “1? in Jeff’s analysis. Also, Microsoft doesn’t have conflicting software product lines – they have the Office team which has swallowed the “Works” team, but there are at least 3 “Office” suites in any GNU/Linux distro (OOo, koffice for KDE, and the suite including ABIWord for gnome).

    Then we can discuss kernels – when there is a driver update for a 3rd party product (Intel i810/845/945 motherboard, for example), it’s a module in the kernel, which requires an updated kernel package from the GNU/Linux distributors, but when there’s a driver update for a 3rd party application, Microsoft doesn’t even have to count it, since it’s “3rd party.” And on the subject of kernels, I don’t recall ever seeing an actual “kernel” update for Windows that wasn’t included in a service pack, or a box on a shelf.

  2. Truth Happens writers: Selling “look how many bugs we fix” to a corporation is a pretty crappy way of doing business, in my opinion. That I can put an appointment in my calendar for 3pm the 2nd Tuesday of each month to review patches, test them that afternoon, and start rolling them out to QA the next morning, is a fantastic way to work. When Red Hat comes out with an update, it’s at a random time, and I have to review each one individually against what I may have installed on my systems.Now, this isn’t a dig against any GNU/Linux distribution out there – free (Ubuntu) or enterprise (Novell / Red Hat) – they are forced into this disclosure/fix model by the fact that these packages are not maintained solely by the companies that are pushing the fixes. In fact, in these cases, the patches have to be done on a “per-report” basis because of how most open-source software vulnerabilities are reported.This is a great time to ask: why is OOo included in a server distro? There *has* to be some GPL or package management reason behind it, but I’d be really interested to know.

So here we see 2 points of view: MS’s (Jeff Jones’) “we’re great because we don’t have a lot of patches, which means we’re more secure;” and RH’s (Truth Happens’) “we’re great because we’ve patched all of the bugs that have been found, no matter how small.” In truth, I think the real point should be that they are 2 completely different companies with huge differences in their offerings in the “Operating System” category. To have both representatives of both companies post what amount to “nyah nyah, we’re better than you are” blogs, keeps the entire discourse of security at a childish level that helps nobody.

So, to both Jeff and the writers of “Truth Happens”: please, out of respect for your readers, look deeper into the numbers and provide some insight, don’t just knock your competition.

Today is my last day on the job before starting a 2 week long vacation to Australia, visiting Sydney, Port Douglas, and Uluru.  I’ve been asked several times if I got a GSM phone to be able to take calls there in case something went horribly wrong at the office, and as a follow up, if Sprint has service out there (when they hear I’m not taking a GSM phone).

I make it a point in my work to make sure someone else can effectively back me up on all aspects of my work.  There are some people I’ve worked with who seem to think that if they are indispensable, then the company can’t fire them.  However, it also means, to me, that they can’t be promoted, can’t go on vacation, and can’t even have an evening at home with family.  So, when I design things, or fix something that broke, or make changes to make something work better, I make sure to include as many team members as I can, so that I can do things like take my wife out, and not be tied to my phone, worried that it may ring, even when I’m not officially on-call.

To that end, I spent a lot of time over the past few weeks giving a lot of history to the newer members of our team, so that they understand the decision making process that led us to the system state we’re at now.  Why do we have to reboot Terminal Servers every weekend?  Because of a memory leak in Windows 2000 that our application and settings trigger fast enough to require it.  Not just “which servers do we have to have up 24/7?, but why those servers, and not others, even if they’re in the same priority group.   This has been tremendously helpful to them in their day-to-day work, evidenced by the lower volume of questions they’re asking to other members of the team.

So, after all this work, how are things set?  Does everyone in the team have the exact same skillset at the same level as me?  No, because we’re different people.  Will it maybe take them a few minutes more to solve <insert particular problem here>?  Maybe, because I may be the most knowledgeable person on that application, but that doesn’t mean that they can’t fix it quickly.  So I spent half of the day today re-iterating those facts to people who are worried that the company will fail if I’m not here (it surely won’t – I’m not that important).

Now, then, off to vacation – I’ll write a blurb about it in 2 weeks, then a few days later about how busy I am catching up!

« Previous PageNext Page »