Now that I have the system back online, I thought I’d post a quick “where we are” update for any regular readers:

  1. We have restored from most recent backup, but are missing a single post, “PHP, mail(), Apache, and SELinux (FC7)”, which even’s cache didn’t catch in full. I apologize to the readers who were using the instructions in that post whom we met through their comments.
  2. We haven’t yet restored the “comments” table. I haven’t yet decided if we will.
  3. I have fixed the problem of storing backups for the company in 3 different locations, based on system type. Now we only have 2 – onsite and offsite.
  4. The extremely popular How to Change a DC IP address article was restored first. (That page drives over half of our traffic.)

We did a standard forensics review of what happened, and it appears as though a perfect storm of issues hit us – a weekend outage, a hardware failure, and failure to keep publicly exposed software fully up-to-date. The saying often goes, “The cobbler’s kids are the ones without shoes” or something similar to that, and here we failed to follow our own advice, preferring to keep our customers’ systems running smoothly. I know I’ll be spending a few extra hours a week the rest of this year reviewing our internal systems for best practices.

In any case, things are fixed and running great again.

I�m hunting down an issue on Fedora Core 7 where PHP5 can�t send mail using sendmail or postfix. In /var/log/httpd/access_log we are getting sh: /usr/sbin/sendmail: Permission denied every time the mail() function is accessed, and postfix never sees any connection. This is being caused by SELinux blocking Apache from transitioning from the �httpd� role to the �mta� role – I�m just not sure what the *best* way to fix it is yet. I haven�t seen many posts about this, so stay tuned – I expect to have a fix tomorrow afternoon.

« Previous Page