totalnetsolutions.net

Upgrading software – always required to keep things secure.  Windows, Wordpress, Mac OSx, Linux, Office, Firefox, etc.  So I just finished upgrading TotalNetSolutions.net again.  Hopefully I’ll be able to be better about this, now that Wordpress does the automatic upgrades now.

I’ve been doing the automatic upgrades on one of my other sites since they came out.  They’re easy, fast, and even more painless than the 3-step upgrade that works so well. So now, I should be able to keep TNS much further away from the “cobbler’s kids” syndrome so many small company’s systems suffer with.

In the Windows world, tools like Group Policy, System Center Configuration Manager, and DesktopAuthority, among others, have been around for 8 or more years to allow fast simple deployment of software and updates to remote computers, or force tasks to be run on remote computers.

For the Unix/Linux world, there doesn’t seem to be as much available.

If you have a pure HP-UX shop, there is HP Systems Insight Manager (SIM) with plug-ins available for software deployment, and I believe IBM Tivoli has a function or sub-product which does the same thing if you have all AIX systems. Red Hat Network has a feature to allow commands to be run on your servers, but only whenever they check in with the RHN or your internal Satellite Server (much like Group Policy, except GPO doesn’t allow “in the middle of the day” script creation without GP-Preferences). So what’s available that’s like SCCM or DesktopAuthority – a “click now and do this thing” tool?

A bunch of my customers just have various levels of logging and processing that come down to being a big for loop that ssh’s into a server and runs a command:
for i in `cat server-list.txt` ; do scp scriptname $i:/root/; ssh $i "/root/scriptname" | tee logfile-$i.log; done;
While it works great for smaller commands. if you have a mixed environment, the “scriptname” script has to be intelligent enough to know what it’s running against, or your “server-list.txt” has to be broken up by class of system. In either case, if you have 200 systems in the list, and the task takes 5 minutes per server, a single install will run for 16-17 hours.

Software like Likewise Enterprise which allow Group Policy management to remote computers is great, because you can have guaranteed delivery and execution of your script or command in (by default) 30 minutes, but my problem is how to get it there in the first place?

So, administrators out there in companies with 1000, 4000, 10000+ servers (or even Desktops), what mutli-threaded or multi-process tool are you using to tackle this timing/resouce problem? Please post below!

As of today:

“Akismet has caught 347 spam for you since you first installed it.”

That’s since 11/29/2007. Akismet has YET to miscategorize a comment as spam, and it has missed a single spam comment. All I had to do was click “this is spam” and it’s cleaned up.

The only other anti-spam product I’ve seen to perform this well is the IronPort mail system at a client. 130,000 or so attempts / day, 1 spam / day in the entire company queue, and no users complaining about spam in 5 months.

Akismet, Ironport, my hat is off to you both.

If you are setting up a cross-forest trust with selective authentication (which requires a Windows Server 2003 Native mode level forest and domain), don’t forget to grant the “Allowed to Authenticate” right to the users from the trusted domain to the servers they’ll need access to in your domain. The error messages you’ll get back (replicated here in my test VM domains) don’t really say much helpful.

System Error 317 has occurred. The system cannot find message text for message number 0x*** in the message file for ***.

Further information about adding the “Allowed to Authenticate” right to the trusted users is available at Microsoft TechNet. If you have the opportunity to raise your forest and domain functional levels to take advantage of this, I highly recommend it. But I recommend also (even more strongly) documenting precisely what you set.

Now that I have the system back online, I thought I’d post a quick “where we are” update for any regular readers:

1. We have restored from most recent backup, but are missing a single post, “PHP, mail(), Apache, and SELinux (FC7)”, which even google.com’s cache didn’t catch in full. I apologize to the readers who were using the instructions in that post whom we met through their comments.
2. We haven’t yet restored the “comments” table. I haven’t yet decided if we will.
3. I have fixed the problem of storing backups for the company in 3 different locations, based on system type. Now we only have 2 – onsite and offsite.
4. The extremely popular How to Change a DC IP address article was restored first. (That page drives over half of our traffic.)

We did a standard forensics review of what happened, and it appears as though a perfect storm of issues hit us – a weekend outage, a hardware failure, and failure to keep publicly exposed software fully up-to-date. The saying often goes, “The cobbler’s kids are the ones without shoes” or something similar to that, and here we failed to follow our own advice, preferring to keep our customers’ systems running smoothly. I know I’ll be spending a few extra hours a week the rest of this year reviewing our internal systems for best practices.

In any case, things are fixed and running great again.