First step is to determine which servers are responsible for mail delivery inbound for the domain you’re sending to. You do this by looking in DNS for the “MX” type records. These are provided in the format “priority servername.domain.” Priority is reverse-ordered. The easiest way to remember priority order is that it’s the order in which servers are attempted.

rob@rob-kubuntu3:~$ dig MX totalnetsolutions.net +short
10 docsmooth.isa-geek.net.
rob@rob-kubuntu3:~$ dig MX likewise.com +short
10 server1.inboundmx.com.
20 server2.inboundmx.com.

This tells you the servers, in order, that *all* mail will be sent to for the domain listed. So, anything to my likewise.com address will go to server1.inboundmx.com. The higher priorities are only used if the lower priorities fail to answer. If no server answers, the mail is held by the sender and retried, generally every 1 or 4 hours for up to 4 days, but this retry is configured on the *sending* server. That means, your own email admin (or you, if you’re the mail admin).

Next thing to check is: does the server work, and is it your sender, or their receiver? Check with telnet!
Stuff I type is in red:

rob@rob-kubuntu3:~$ telnet docsmooth.isa-geek.net 25
Trying 99.29.179.119...
Connected to docsmooth.isa-geek.net.
Escape character is '^]'.
220 totalnetsolutions.net Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at Tue, 31 May 2011 08:43:08 -0500
HELO
250 totalnetsolutions.net Hello [12.130.116.175]
MAIL FROM: me@me.com
250 2.1.0 me@me.com....Sender OK
RCPT TO:you@you.net
250 2.1.5 you@you.net
DATA
354 Start mail input; end with .
from:me@me.com
to:you@you.net
subject:test manually
test
test
.
250 2.6.0 Queued mail for delivery
quit
221 2.0.0 totalnetsolutions.net Service closing transmission channel
Connection closed by foreign host.

The last “.” is SUPER important – it tells the mail server when you’re done sending that email. You could use that channel to send other messages, rather than sending “QUIT” if you’d like. You might notice that I entered the “From” and “To” lines twice. The first entries are for the SMTP header (analogy would be the message envelope), and the second entries are for the MIME headings (analogy would be the return address header in a formal postal letter, if anyone sends those). The MIME headings are what most mail programs display, and actually don’t technically need to match the SMTP header (but if MIME and SMTP don’t match many anti-spam programs will throw out the message).

The MIME header is pretty complex, but not order-dependant, although I prefer to enter it in order, so that I can be sure I don’t miss anything.
If you want to add an attachment, just base64 encode it first with:
perl -e 'use MIME::Base64; qw(encode_base64); print encode_base64("@ARGV");' cat attachment-to-send.zip
Then you can just paste it into the email. In the MIME heading (right after the subject), just add (with the appropriate mime coding, probably application/octet-string:

------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: application/zip;
name="attachment-to-send.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="attachment-to-send.zip"
<paste your base64 encoded attachment/ >
Content-Type: multipart/mixed;
boundary="----=_NextPart_000_000D_01CC1C41.21F38080"
------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

Now that you know *how* to send an email message by hand, you can use the returned error codes to troubleshoot where the message may be disappearing. Remember, that this just gives you transport troubleshooting between yourself and the initial destination mail server. Many large (and even medium-sized) organizations will have a perimeter mail server which then forwards the message to one or more internal servers. If the mail is being dropped at that point, you’ll have to contact the reciever with the proof that their server is accepting your messages.

In our lab we have an empty forest root, as per the old (Windows 2000-era) Microsoft recommendations, to match several large customer environments. Because it’s a lab, and no clients connect to it, we only have a single DC. I snapshotted it as a backup, and went through the procedure to rename a domain controller, also well documented by Microsoft, this time at TechNet.

For review, the procedure we planned to run was:
netdom computername dc04 /add:dc01.lwtest.corp
netdom computername dc04 /makeprimary:dc01.lwtest.corp
shutdown -r -t 0
netdom computername dc01 /enum
netdom computername dc01 /verify
netdom computername dc01 /rem:dc04.lwtest.corp

I’m still not sure what caused it, but in this case, this command failed:
netdom computername dc04 /makeprimary:dc01.tns.lab
At this point, I couldn’t make the old name primary again (I would get an “Access Denied” error), so I rebooted to see which name had taken. And that’s where things went bad.

When the DC came up, we were getting this error: . Source: NETLOGON, EventID: 5602, Data: “An internal error occurred while accessing the computer’s local or network security database.”

Because the DC rename hadn’t completed successfully, the computer couldn’t actually log into itself to load AD. Very bad for the root of the forest. I wasn’t able to find anything helpful in my searches, so thought I’d let you know the fix:

Name it back to the old name and try again:
Reboot into Safe Mode.
netdom computername localhost /makeprimary:dc04.lwtest.corp
shutdown -r -t 0
Boot normally
netdom computername localhost /makeprimary:dc04.lwtest.corp
netdom computername dc01 /enum
netdom computername dc01 /verify
shutdown -r -t 0
After *that* reboot, make sure, with the verify command, that the old name took, and that you can log in, and just try the rename again.

I couldn’t get the “rename back” to take untill after the attempt in safe mode. Strange, but it’s working great now! Hopefully this will help someone.

Thanks to the k9mail wiki on debugging connection issues and the fact that I already had the Android SDK installed, I was able to solve the 2 related errors I was getting. I would either get an “HTTP 404 not found” or an “HTTP 501 Not Implemented” depending on the settings I chose. With no additional settings other than suggested in the Wiki, I’d get a “501 not implemented”. If I tried to set a mailbox path, or a WebDAV path, I’d get the HTTP 404 Not Found.

In the debugging log, I saw that the system was calling “http://mail.$domain.exchange.ms/”$webDAVpath/Inbox – if I set it to a full URL, the full URL was getting appended. When I attempted to hit those same paths in a full browser, I’d always get an HTTP 404. So, digging in my history in Firefox, I found the following (cleaned) path:

http://mail.$domain.exchange.ms/exchange/$emailaddress/

In this case $emailaddress was my Exchange mail address with the “@” stripped out. Appending “Inbox” to the end of this path resulted in a valid load of my OWA inbox.

Plugging then: /exchange/$emailaddress/ into the WebDAV box in K9Mail, and my email immediately loaded up.

Now I have Android syncing my calendars and contacts, and k9mail is handling my massive inbox!

On a Samba server joined to AD with winbind, this is easy to deal with because Samba’s winbind can treat the computer accounts just like user accounts, and assign them access to the unix filesystem with whatever backend has been configured. When a Samba server is joined with Likewise, however, the machine accounts are not visible, and the “username is invalid” message comes up.

Fortunately, Samba gives us a method to handle this, in form of the “username map” directive in /etc/samba/smb.conf.  There are two ways to use this, the first is with the username map file.
In smb.conf, to simply add:
[global ]
username map = /etc/samba/smbusers
then create a file named /etc/samba/smbusers and populate it with localuser=aduser pairs, like:
COMPUTER1$ = compacct
COMPUTER2$ = compacct
CITRIXFARM1$ = citrxact
and so on. Lastly, you’ll have to add the local accounts from the pairs above:
useradd -c "Account for AD Computers to use Samba" compacct -G users -u 998
useradd -c "Account for AD Citrix Servers to use Samba" citrxact -G users -u 999
Then, whenever one of the AD computers in the list attempts to access the Samba share, it’ll be mapped to the local account.

The problem with this is when you have a lot of servers, like a Citrix MetaFrame farm, or a Windows Server 2008 R2 Remote Desktop Services farm, that may be changing frequently, because managing that file could get hard. In this case there is the username map script directive, which is added to smb.conf as:
[global ]
username map script = /usr/lib/samba/auth/machine-acct-map.pl
Then download this script and save it in /usr/lib/samba/auth/ and make it executable (chmod +x /usr/lib/samba/auth/machine-acct-map.pl). Then run:
useradd -c "Account for AD Computers to use Samba" compacct -G users -u 998
Now, all computers which access the share will be remapped to the “compacct” user, and you won’t have to manage a file for every time the server farm changes.

Get the file here.

Of note is a response for T. Colin Dodd regaring his short and sweet post regarding Red Hat Flaws according to Secunia. In short, Mr. Dodd (please correct me if the address is wrong), yes, Red Hat should be proud of what they’ve accomplished, but…

Well, that’s 2 pages of text that’s not yet finished.

As any good Exchange administrator knows, Exchange stores its data (for a store)�in 2 files, the EDB file, and the STM file.� However, there’s not a really great explanation of the differences between the two files – the best I’ve found so far is at MessagingTalk.org, but they only explain that the STM is MIME formatted, and the EDB is MAPI content. Why, though, and how does it affect the end users? This is what we’ll explore.

What brought me into this topic was an end-user complaining that an email with a 7MB attachment was getting bounced, and the�NDR said that it was over the mail system limit.� Normally, nothing worth investigating: zip the attachment, and resend.� The problem was, the user had already zipped it down, and the messaging size limit for the Exchange Organization is 10MB.� We had the user forward the message to several admins, and it went through ok.� About 500KB of HTML-formatted email, and a 7.1MB attachment, for a total size of 7.6MB.� When I sent it out to my account back�here, it bounced back – over size limit, and the emails were being rejected by the Front End Exchange servers, so the emails weren’t even leaving the organization.

But Why?

Remember how we said earlier that the EDB file contains MAPI-formatted emails?� All messages submitted to Exchange via MAPI clients (Outlook) are stored directly as submitted in the EDB file for the user’s store.� All the message information is stored alongside that message in the store, and Exchange is able to serve the messages back to Outlook conversion-free.�

In the same way, all messages submitted to the store in MIME format (SMTP, POP3, IMAP4, or HTTP – OWA, WebDAV, OMA, or Exchange ActiveSync) are stored in the STM file in their original MIME format, and the message properties that Exchange needs to track are stored separately in the EDB file with pointers to the message location in the STM file.� This means that the STM file is straight plain text, so any data in it is 100% extractable if the store is dismounted (file unlocked).� I do have to say, grepping your STM file for your own email address is pretty interesting, if you’re into that kind of thing.

The thing about MIME formatted email is that any non-ASCII text�has to be Base64 encoded, which causes a 33% increase to the size of the attachment. (See http://www.faqs.org/rfcs/rfc1521.html�section 5.2 for the explanation.)� And aren’t most of your users writing “Rich Text” format emails with Word as their default editor?� Well, all that extra formatting most likely causes them to have to be base64 encoded when they’re converted to MIME.

Email sent between Exchange servers within the Organization are sent through the SMTP connector, but are kept in MAPI format by BDAT encoding the contents.� However, email sent outside of the organization (especially if the remote SMTP server doesn’t accept ESMTP) must be converted to MIME encoding before delivery.

So, looking at my particular user’s case:� The 7.1MB attachment grows to 9.5MB, and the 500KB of HTML formatted messaging replies adds�up to�10MB on the nose.� Add SMTP headers, and the whole thing gets stopped by the SMTP connector on the Front-End Exchange server for just breaking the limit.

In this case, we used it as an opportunity to review how the client was sharing data with customers, and find a way to help them not use email as a file share.

System Error 317 has occurred. The system cannot find message text for message number 0x*** in the message file for ***.

Further information about adding the “Allowed to Authenticate” right to the trusted users is available at Microsoft TechNet. If you have the opportunity to raise your forest and domain functional levels to take advantage of this, I highly recommend it. But I recommend also (even more strongly) documenting precisely what you set.

We’ve seen some issues over the past approximately 2 months, particularly with MS SQL 2000 clusters (1 Exchange 2003 cluster), where the cluster group fails on one node, and the other node (or nodes) fails to pick up the group, leaving the complete cluster group offline. In each of the cases (on both HP and Dell hardware) the first striking piece of evidence in the logs is that all nodes that fail to bring up the cluster report that the Cluster IP Address resource couldn’t be brought online, because of an IP address conflict on the network

Making this issue particularly fun is that most of the information we used to solve the problem, is a lack of information.  In particular, there is absolutely nothing interesting at all in any nodes’ cluster.log file.You see the disks negotiate from node to node, but nothing that makes the failover look any different than if you had right-clicked the group and chosen “Move Group” from Cluster Administrator.

What starts the problem off is Event ID 1228 from source “ClusNet”, which says that the “ClusNet driver couldn’t communicate with the ClusSvc for 60 seconds, the Cluster service is being terminated.” Most of the time, you might even miss that this event is there, because it causes so many Event Source Tcpip, ID 4199; Source ftdisk, ID 57; and Source ntfs event ID 50 events, that it’s easy to look over 1 little error. Especially when monitoring systems like Microsoft Operations Manager (MOM), or Idera SQLDiagnostics Manager (SQLDiag) or HP Systems Insight Manager (SIM) all report the cluster as having issues 30-60 seconds after the CluNet 1228 event is written (timing which corresponds exactly to the Tcpip 4199 events (IP address conflict) or the ftdisk 57 events (failed to flush transaction data). So, here’s what happens, based on conversations with Microsoft, training with Microsoft and HP, and a LOT of reading.

First, we have to discuss the layout of Microsoft Clustering.  There is the well-known “ClusSvc” – the “Cluster” service that you find in the “Services” MMC console. This is what’s known as a user-mode process (as opposed to kernel-mode), so it runs in the same memory and processor spaces as Exchange and SQL.  There are 2 other “services” – the Cluster Network driver (ClusNet) and the Cluster Disk driver (ClusDisk).  If you were ever an NT4 MCSE, you might remember the old Control panel “Drivers” extension.  These services would show up in there, if it still existed, along with a lot of other driver/services – they haven’t gone away, just the GUI for editing them.  Anyways, because ClusSvc runs in the same processor context as SQL or Exchange, and because it shares memory with SQL or Exchange, it can be starved of resources by SQL or Exchange, or any other user-mode process running on your server.  To keep track of whether this has happened or not, Microsoft wrote the ClusNet driver to not only keep track of the private and public networks, and the group IP address resources currently owned by the node, they wrote it to also check a heartbeat (default of 60 seconds) back with ClusSvc.  The best explanation I’ve found is here on Microsoft’s site.

So, the ClusNet event 1228 means that the ClusSvc failed to heartbeat within the 60 second timeout window, so ClusNet killed the Cluster.exe process (ClusSvc).  Generally the OS is set to auto-restart the Cluster service if it terminates, so the node will recover… but what about the cluster as a whole?  Some interesting things happen when cluster.exe is forcibly terminated.

1. Because it’s terminated, cluster.exe can’t tell ClusNet to drop registration of the group IP address resources owned by the node, so ClusNet keeps the IP addresses advertised (arp’d) on the network.
2. Cluster.exe also can’t tell ClusDisk to flush ntfs logs to disk in preparation for the disks being grabbed by another node.
3. Because the Cluster service isn’t running, the node can’t heartbeat to the other nodes in the cluster.
4. Therefore, the other node(s) reach a quorum, decide that the first node is unavailable, and begin the process of a failover.
5. First, disks are negotiated on the SCSI/FibreChannel bus, eventually forcibly taken from the “offline” node (causing the ftdisk 57 and ntfs 50 errors).
6. Simultaneously (unless you have IP address resources having a requirement on your disks), the preferred failover node (or next in the round-robin process) attempts to advertise that its MAC address is now the IP address of the cluster group.
7. And things get ugly.

Because the original node still hasn’t restarted its Cluster Service (services are normally restarted after 60 seconds / 1 minute, unless you changed your cluster), the ClusNet driver on the original node is still arping the group IP address.  This causes the tcpip 4199 errors on the node that’s attempting to bring up the group.  These errors cause the group to fail to come up on the 2nd node, so the cluster group moves to the next node in the round-robin (or preferred owners) list, untill all nodes have failed to bring up the group.  In my experience, 4 nodes all fail to bring up the group in under 60 seconds, so the cluster group stays offline failed, even after the cluster service restarts on the original node.  Also, the disks have been moved to every node in the cluster, so you can probably gather where the disk errors come from.

So how can you fix or avoid this problem?  I am taking the advice written between the lines of the previously mentioned Microsoft article. The default action of a failure of the heartbeat is to restart the cluster service, which causes the issues mentioned above.  However, there are 2 other options available:  Do Nothing (and log or not), and Bugcheck.

If you set the mentioned registry key to “3″, the heartbeat failure will cause a bugcheck of the server, dumping out a kernel memory dump (or minidump or full dump, depending on your settings, but the default is a kernel dump) of exactly what was going on when the heartbeat failed.  Then, because the failed node is now Blue Screened while it writes out the memory dump, it will no longer fight the disk arbitration OR advertise the cluster group IP addresses that it had owned.  The next node in the list will pick up the group, restart the resources, and your cluster is once-again highly available.

And you now have more troubleshooting information than you would if you just let the service restart.

Now, we had some discussions with clients about this setting, and they were concerned that bugchecking a server could cause data corruption, especially on a busy SQL server.  This is true. However, restarting the cluster service causes a forced unload of running cluster group processes anyways, so bugcheck or cluster service restart causes the exact same data corruption issues, but a bugcheck gives the technician looking at the problem more data, to know how to avoid the problem in the future.  Also, if no action is taken, you could have the cluster in a resource starvation situation, in which case the technician has to manually kill processes, if he can even get into the server to kill processes.  The resource starvation may even force someone to physically reset power to the server node.  And while that’s happening, the cluster may not fail over to the working nodes, causing an actual business outage.  And isn’t that why you’re building clusters in the first place?

But the problem came back 22 days later on one of the 2 stores that the users were moved to, so we knew something else must be up.� I�ll cut to the chase and explain that Microsoft now is very positive of what is happening, just not who is causing it or why it�s happening.

What�s frustrating about this is that all the tools that can be used to look deeper into this problem aren�t available to me as a technician outside of Microsoft.� All I�ve been able to do for my client is set up triggers to cause �Exchange store.exe dumps� which are essentially process freezes followed by private memory dumps to disk.� The good thing is that the end users don�t notice, nor does the Windows 2003 Cluster service.� Also, our Microsoft support team has been great at sharing information with us.

But the problem still remains, that there is nothing at all that I can do to fix this problem.� I can�t run the debug programs (I can run a debug against the process, but not to the same level of detail, due to a lack of published information) that Microsoft has available, despite a very deep understanding of how the ESE runs the EDB, STM, and LOG files (for an outside consultant who just reads voraciously).� This inability to better service my customers frustrates me to no end, whether Microsoft�s technicians are fantastic or not (there have been other times�).

So, while I wait for them to get back to me on yet another dump that has been generated, looking for a very elusive fSearch() operation against one of my client�s many Exchange 2003 stores, I sit on my hands in anticipation, wishing to be able to do more.

I read both of these blogs, and I�m frankly disgusted by the way both sides are treating the data. I understand that statistics are often more useful for what they hide, than what they show. In this case, the 2 competing ideas seem to be: �We fix more bugs, which means we�re working harder to protect you�, vs. �we fix fewer bugs because we have fewer bugs, so we�re working harder to protect you�. I think both of these arguments are invalid, so I hope both sides see this and pay attention.

1. Jeff Jones: Jeff does a very interesting quarterly (or so) patch report – what OS�s have had the most patches applied in �xx� time frame (past quarter, past year, etc.). I get a lot of out this report, and he does very good trending. Find them on his blog and read them.To that end, he does a very good job selling Microsoft as a security company. By purely counting �number of patches submitted�, Microsoft will automatically look better, simply because �Windows (XP and 2003 combined)� has fewer features than �Red Hat Enterprise Linux� or �SUSE Enterprise Linux� or �Ubuntu Desktop Edition�.Jeff makes a point that Microsoft has only released patches for 649 security vulnerabilities across all Microsoft products in 7 years, but�What Windows does have that the GNU/Linux variants don�t have: .NET Framework, which is a HUGE project, but when it�s updated, you get a single update, so it counts as �1? in Jeff�s analysis. Also, Microsoft doesn�t have conflicting software product lines – they have the Office team which has swallowed the �Works� team, but there are at least 3 �Office� suites in any GNU/Linux distro (OOo, koffice for KDE, and the suite including ABIWord for gnome).

   Then we can discuss kernels – when there is a driver update for a 3rd party product (Intel i810/845/945 motherboard, for example), it�s a module in the kernel, which requires an updated kernel package from the GNU/Linux distributors, but when there�s a driver update for a 3rd party application, Microsoft doesn�t even have to count it, since it�s �3rd party.� And on the subject of kernels, I don�t recall ever seeing an actual �kernel� update for Windows that wasn�t included in a service pack, or a box on a shelf.

2. Truth Happens writers: Selling �look how many bugs we fix� to a corporation is a pretty crappy way of doing business, in my opinion. That I can put an appointment in my calendar for 3pm the 2nd Tuesday of each month to review patches, test them that afternoon, and start rolling them out to QA the next morning, is a fantastic way to work. When Red Hat comes out with an update, it�s at a random time, and I have to review each one individually against what I may have installed on my systems.Now, this isn�t a dig against any GNU/Linux distribution out there – free (Ubuntu) or enterprise (Novell / Red Hat) – they are forced into this disclosure/fix model by the fact that these packages are not maintained solely by the companies that are pushing the fixes. In fact, in these cases, the patches have to be done on a �per-report� basis because of how most open-source software vulnerabilities are reported.This is a great time to ask: why is OOo included in a server distro? There *has* to be some GPL or package management reason behind it, but I�d be really interested to know.

So here we see 2 points of view: MS�s (Jeff Jones�) �we�re great because we don�t have a lot of patches, which means we�re more secure;� and RH�s (Truth Happens�) �we�re great because we�ve patched all of the bugs that have been found, no matter how small.� In truth, I think the real point should be that they are 2 completely different companies with huge differences in their offerings in the �Operating System� category. To have both representatives of both companies post what amount to �nyah nyah, we�re better than you are� blogs, keeps the entire discourse of security at a childish level that helps nobody.

So, to both Jeff and the writers of �Truth Happens�: please, out of respect for your readers, look deeper into the numbers and provide some insight, don�t just knock your competition.