totalnetsolutions.net

Now that I have the system back online, I thought I’d post a quick “where we are” update for any regular readers:

1. We have restored from most recent backup, but are missing a single post, “PHP, mail(), Apache, and SELinux (FC7)”, which even google.com’s cache didn’t catch in full. I apologize to the readers who were using the instructions in that post whom we met through their comments.
2. We haven’t yet restored the “comments” table. I haven’t yet decided if we will.
3. I have fixed the problem of storing backups for the company in 3 different locations, based on system type. Now we only have 2 - onsite and offsite.
4. The extremely popular How to Change a DC IP address article was restored first. (That page drives over half of our traffic.)

We did a standard forensics review of what happened, and it appears as though a perfect storm of issues hit us - a weekend outage, a hardware failure, and failure to keep publicly exposed software fully up-to-date. The saying often goes, “The cobbler’s kids are the ones without shoes” or something similar to that, and here we failed to follow our own advice, preferring to keep our customers’ systems running smoothly. I know I’ll be spending a few extra hours a week the rest of this year reviewing our internal systems for best practices.

In any case, things are fixed and running great again.

As I recently rebuilt my primary system (my laptop), I couldn’t find the backups for the site - it appears that I may have skipped them when backing up the laptop. However, I have all the content, it will just be a day or 2 to get everything re-imported. The short story is that the site got hacked and *all* content and code were wiped.

(Originally drafted November 2nd, 2007, finally finished and posted much later)
As I posted last night, we built a new Fedora Core 7 box last night for PHP testing. Whenever at all possible, I leave SELinux enabled on new systems in Enforcing mode. Oracle 10g hasn’t had any issues with it, Oracle 11i EBusiness Suite hasn’t had any issues with it, and my NFS and FTP servers run without at hitch. The Oracle systems are RHEL4 (Red Hat Enterprise Linux 4), and the NFS and FTP servers are RHEL5.

However, this new PHP webserver caused a few glitches. I feel a little silly for not catching this as being an SELinux problem earlier, but since it’s caused 0 issues in 9 months of use in production, I didn’t even consider it initially.

What we initially saw was 0 errors from PHP - all the pages would run without error. PHP.ini has the following lines:

sendmail_from = from@domain.com
sendmail_path = /usr/sbin/sendmail -t -i

and testing cat mail.txt | /usr/sbin/sendmail -t -i as a non-root user delivered mail properly as well. Combine that with /var/log/maillog being completely empty for every test page loaded, and it was sure that the mail wasn’t getting TO postfix (our preferred localhost MTA).

So, I looked at the /var/log/httpd/error_log for apache and found:

sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied
sh: /usr/sbin/sendmail: Permission denied

But I knew that non-root users could access sendmail as defined in php.ini, so I finally decided to tail /var/log/messages and saw:

Nov 2 11:05:41 $(servername) setroubleshoot: SELinux is preventing the sh from using potentially mislabeled files sendmail.postfix (sendmail_exec_t). For complete SELinux messages. run sealert -l c9001c48-5d48-4b7c-9fd7-8400544daa8f

So now to fix it…
This is surprisingly simple, actually. The sad part is, we had this problem, fixed it, forgot about it, had it again, and I blogged it… and lost the post. so this has been sitting in my “drafts” folder for about 10 months now:
setsebool httpd_can_sendmail=true
service httpd restart
service postfix restart
And retry sending mail. There’s a few posts about sendmail and having to change permissions on home directories or on “main.cf”, but I use postfix, and not sendmail, so I don’t know how effective or necessary those changes are.

I’m hunting down an issue on Fedora Core 7 where PHP5 can’t send mail using sendmail or postfix. In /var/log/httpd/access_log we are getting sh: /usr/sbin/sendmail: Permission denied every time the mail() function is accessed, and postfix never sees any connection. This is being caused by SELinux blocking Apache from transitioning from the “httpd” role to the “mta” role - I’m just not sure what the *best* way to fix it is yet. I haven’t seen many posts about this, so stay tuned - I expect to have a fix tomorrow afternoon.