First step is to determine which servers are responsible for mail delivery inbound for the domain you’re sending to. You do this by looking in DNS for the “MX” type records. These are provided in the format “priority servername.domain.” Priority is reverse-ordered. The easiest way to remember priority order is that it’s the order in which servers are attempted.

rob@rob-kubuntu3:~$ dig MX totalnetsolutions.net +short
10 docsmooth.isa-geek.net.
rob@rob-kubuntu3:~$ dig MX likewise.com +short
10 server1.inboundmx.com.
20 server2.inboundmx.com.

This tells you the servers, in order, that *all* mail will be sent to for the domain listed. So, anything to my likewise.com address will go to server1.inboundmx.com. The higher priorities are only used if the lower priorities fail to answer. If no server answers, the mail is held by the sender and retried, generally every 1 or 4 hours for up to 4 days, but this retry is configured on the *sending* server. That means, your own email admin (or you, if you’re the mail admin).

Next thing to check is: does the server work, and is it your sender, or their receiver? Check with telnet!
Stuff I type is in red:

rob@rob-kubuntu3:~$ telnet docsmooth.isa-geek.net 25
Trying 99.29.179.119...
Connected to docsmooth.isa-geek.net.
Escape character is '^]'.
220 totalnetsolutions.net Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at Tue, 31 May 2011 08:43:08 -0500
HELO
250 totalnetsolutions.net Hello [12.130.116.175]
MAIL FROM: me@me.com
250 2.1.0 me@me.com....Sender OK
RCPT TO:you@you.net
250 2.1.5 you@you.net
DATA
354 Start mail input; end with .
from:me@me.com
to:you@you.net
subject:test manually
test
test
.
250 2.6.0 Queued mail for delivery
quit
221 2.0.0 totalnetsolutions.net Service closing transmission channel
Connection closed by foreign host.

The last “.” is SUPER important – it tells the mail server when you’re done sending that email. You could use that channel to send other messages, rather than sending “QUIT” if you’d like. You might notice that I entered the “From” and “To” lines twice. The first entries are for the SMTP header (analogy would be the message envelope), and the second entries are for the MIME headings (analogy would be the return address header in a formal postal letter, if anyone sends those). The MIME headings are what most mail programs display, and actually don’t technically need to match the SMTP header (but if MIME and SMTP don’t match many anti-spam programs will throw out the message).

The MIME header is pretty complex, but not order-dependant, although I prefer to enter it in order, so that I can be sure I don’t miss anything.
If you want to add an attachment, just base64 encode it first with:
perl -e 'use MIME::Base64; qw(encode_base64); print encode_base64("@ARGV");' cat attachment-to-send.zip
Then you can just paste it into the email. In the MIME heading (right after the subject), just add (with the appropriate mime coding, probably application/octet-string:

------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: application/zip;
name="attachment-to-send.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="attachment-to-send.zip"
<paste your base64 encoded attachment/ >
Content-Type: multipart/mixed;
boundary="----=_NextPart_000_000D_01CC1C41.21F38080"
------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

Now that you know *how* to send an email message by hand, you can use the returned error codes to troubleshoot where the message may be disappearing. Remember, that this just gives you transport troubleshooting between yourself and the initial destination mail server. Many large (and even medium-sized) organizations will have a perimeter mail server which then forwards the message to one or more internal servers. If the mail is being dropped at that point, you’ll have to contact the reciever with the proof that their server is accepting your messages.

Thanks to the k9mail wiki on debugging connection issues and the fact that I already had the Android SDK installed, I was able to solve the 2 related errors I was getting. I would either get an “HTTP 404 not found” or an “HTTP 501 Not Implemented” depending on the settings I chose. With no additional settings other than suggested in the Wiki, I’d get a “501 not implemented”. If I tried to set a mailbox path, or a WebDAV path, I’d get the HTTP 404 Not Found.

In the debugging log, I saw that the system was calling “http://mail.$domain.exchange.ms/”$webDAVpath/Inbox – if I set it to a full URL, the full URL was getting appended. When I attempted to hit those same paths in a full browser, I’d always get an HTTP 404. So, digging in my history in Firefox, I found the following (cleaned) path:

http://mail.$domain.exchange.ms/exchange/$emailaddress/

In this case $emailaddress was my Exchange mail address with the “@” stripped out. Appending “Inbox” to the end of this path resulted in a valid load of my OWA inbox.

Plugging then: /exchange/$emailaddress/ into the WebDAV box in K9Mail, and my email immediately loaded up.

Now I have Android syncing my calendars and contacts, and k9mail is handling my massive inbox!

As any good Exchange administrator knows, Exchange stores its data (for a store)�in 2 files, the EDB file, and the STM file.� However, there’s not a really great explanation of the differences between the two files – the best I’ve found so far is at MessagingTalk.org, but they only explain that the STM is MIME formatted, and the EDB is MAPI content. Why, though, and how does it affect the end users? This is what we’ll explore.

What brought me into this topic was an end-user complaining that an email with a 7MB attachment was getting bounced, and the�NDR said that it was over the mail system limit.� Normally, nothing worth investigating: zip the attachment, and resend.� The problem was, the user had already zipped it down, and the messaging size limit for the Exchange Organization is 10MB.� We had the user forward the message to several admins, and it went through ok.� About 500KB of HTML-formatted email, and a 7.1MB attachment, for a total size of 7.6MB.� When I sent it out to my account back�here, it bounced back – over size limit, and the emails were being rejected by the Front End Exchange servers, so the emails weren’t even leaving the organization.

But Why?

Remember how we said earlier that the EDB file contains MAPI-formatted emails?� All messages submitted to Exchange via MAPI clients (Outlook) are stored directly as submitted in the EDB file for the user’s store.� All the message information is stored alongside that message in the store, and Exchange is able to serve the messages back to Outlook conversion-free.�

In the same way, all messages submitted to the store in MIME format (SMTP, POP3, IMAP4, or HTTP – OWA, WebDAV, OMA, or Exchange ActiveSync) are stored in the STM file in their original MIME format, and the message properties that Exchange needs to track are stored separately in the EDB file with pointers to the message location in the STM file.� This means that the STM file is straight plain text, so any data in it is 100% extractable if the store is dismounted (file unlocked).� I do have to say, grepping your STM file for your own email address is pretty interesting, if you’re into that kind of thing.

The thing about MIME formatted email is that any non-ASCII text�has to be Base64 encoded, which causes a 33% increase to the size of the attachment. (See http://www.faqs.org/rfcs/rfc1521.html�section 5.2 for the explanation.)� And aren’t most of your users writing “Rich Text” format emails with Word as their default editor?� Well, all that extra formatting most likely causes them to have to be base64 encoded when they’re converted to MIME.

Email sent between Exchange servers within the Organization are sent through the SMTP connector, but are kept in MAPI format by BDAT encoding the contents.� However, email sent outside of the organization (especially if the remote SMTP server doesn’t accept ESMTP) must be converted to MIME encoding before delivery.

So, looking at my particular user’s case:� The 7.1MB attachment grows to 9.5MB, and the 500KB of HTML formatted messaging replies adds�up to�10MB on the nose.� Add SMTP headers, and the whole thing gets stopped by the SMTP connector on the Front-End Exchange server for just breaking the limit.

In this case, we used it as an opportunity to review how the client was sharing data with customers, and find a way to help them not use email as a file share.

We’ve seen some issues over the past approximately 2 months, particularly with MS SQL 2000 clusters (1 Exchange 2003 cluster), where the cluster group fails on one node, and the other node (or nodes) fails to pick up the group, leaving the complete cluster group offline. In each of the cases (on both HP and Dell hardware) the first striking piece of evidence in the logs is that all nodes that fail to bring up the cluster report that the Cluster IP Address resource couldn’t be brought online, because of an IP address conflict on the network

Making this issue particularly fun is that most of the information we used to solve the problem, is a lack of information.  In particular, there is absolutely nothing interesting at all in any nodes’ cluster.log file.You see the disks negotiate from node to node, but nothing that makes the failover look any different than if you had right-clicked the group and chosen “Move Group” from Cluster Administrator.

What starts the problem off is Event ID 1228 from source “ClusNet”, which says that the “ClusNet driver couldn’t communicate with the ClusSvc for 60 seconds, the Cluster service is being terminated.” Most of the time, you might even miss that this event is there, because it causes so many Event Source Tcpip, ID 4199; Source ftdisk, ID 57; and Source ntfs event ID 50 events, that it’s easy to look over 1 little error. Especially when monitoring systems like Microsoft Operations Manager (MOM), or Idera SQLDiagnostics Manager (SQLDiag) or HP Systems Insight Manager (SIM) all report the cluster as having issues 30-60 seconds after the CluNet 1228 event is written (timing which corresponds exactly to the Tcpip 4199 events (IP address conflict) or the ftdisk 57 events (failed to flush transaction data). So, here’s what happens, based on conversations with Microsoft, training with Microsoft and HP, and a LOT of reading.

First, we have to discuss the layout of Microsoft Clustering.  There is the well-known “ClusSvc” – the “Cluster” service that you find in the “Services” MMC console. This is what’s known as a user-mode process (as opposed to kernel-mode), so it runs in the same memory and processor spaces as Exchange and SQL.  There are 2 other “services” – the Cluster Network driver (ClusNet) and the Cluster Disk driver (ClusDisk).  If you were ever an NT4 MCSE, you might remember the old Control panel “Drivers” extension.  These services would show up in there, if it still existed, along with a lot of other driver/services – they haven’t gone away, just the GUI for editing them.  Anyways, because ClusSvc runs in the same processor context as SQL or Exchange, and because it shares memory with SQL or Exchange, it can be starved of resources by SQL or Exchange, or any other user-mode process running on your server.  To keep track of whether this has happened or not, Microsoft wrote the ClusNet driver to not only keep track of the private and public networks, and the group IP address resources currently owned by the node, they wrote it to also check a heartbeat (default of 60 seconds) back with ClusSvc.  The best explanation I’ve found is here on Microsoft’s site.

So, the ClusNet event 1228 means that the ClusSvc failed to heartbeat within the 60 second timeout window, so ClusNet killed the Cluster.exe process (ClusSvc).  Generally the OS is set to auto-restart the Cluster service if it terminates, so the node will recover… but what about the cluster as a whole?  Some interesting things happen when cluster.exe is forcibly terminated.

1. Because it’s terminated, cluster.exe can’t tell ClusNet to drop registration of the group IP address resources owned by the node, so ClusNet keeps the IP addresses advertised (arp’d) on the network.
2. Cluster.exe also can’t tell ClusDisk to flush ntfs logs to disk in preparation for the disks being grabbed by another node.
3. Because the Cluster service isn’t running, the node can’t heartbeat to the other nodes in the cluster.
4. Therefore, the other node(s) reach a quorum, decide that the first node is unavailable, and begin the process of a failover.
5. First, disks are negotiated on the SCSI/FibreChannel bus, eventually forcibly taken from the “offline” node (causing the ftdisk 57 and ntfs 50 errors).
6. Simultaneously (unless you have IP address resources having a requirement on your disks), the preferred failover node (or next in the round-robin process) attempts to advertise that its MAC address is now the IP address of the cluster group.
7. And things get ugly.

Because the original node still hasn’t restarted its Cluster Service (services are normally restarted after 60 seconds / 1 minute, unless you changed your cluster), the ClusNet driver on the original node is still arping the group IP address.  This causes the tcpip 4199 errors on the node that’s attempting to bring up the group.  These errors cause the group to fail to come up on the 2nd node, so the cluster group moves to the next node in the round-robin (or preferred owners) list, untill all nodes have failed to bring up the group.  In my experience, 4 nodes all fail to bring up the group in under 60 seconds, so the cluster group stays offline failed, even after the cluster service restarts on the original node.  Also, the disks have been moved to every node in the cluster, so you can probably gather where the disk errors come from.

So how can you fix or avoid this problem?  I am taking the advice written between the lines of the previously mentioned Microsoft article. The default action of a failure of the heartbeat is to restart the cluster service, which causes the issues mentioned above.  However, there are 2 other options available:  Do Nothing (and log or not), and Bugcheck.

If you set the mentioned registry key to “3″, the heartbeat failure will cause a bugcheck of the server, dumping out a kernel memory dump (or minidump or full dump, depending on your settings, but the default is a kernel dump) of exactly what was going on when the heartbeat failed.  Then, because the failed node is now Blue Screened while it writes out the memory dump, it will no longer fight the disk arbitration OR advertise the cluster group IP addresses that it had owned.  The next node in the list will pick up the group, restart the resources, and your cluster is once-again highly available.

And you now have more troubleshooting information than you would if you just let the service restart.

Now, we had some discussions with clients about this setting, and they were concerned that bugchecking a server could cause data corruption, especially on a busy SQL server.  This is true. However, restarting the cluster service causes a forced unload of running cluster group processes anyways, so bugcheck or cluster service restart causes the exact same data corruption issues, but a bugcheck gives the technician looking at the problem more data, to know how to avoid the problem in the future.  Also, if no action is taken, you could have the cluster in a resource starvation situation, in which case the technician has to manually kill processes, if he can even get into the server to kill processes.  The resource starvation may even force someone to physically reset power to the server node.  And while that’s happening, the cluster may not fail over to the working nodes, causing an actual business outage.  And isn’t that why you’re building clusters in the first place?

But the problem came back 22 days later on one of the 2 stores that the users were moved to, so we knew something else must be up.� I�ll cut to the chase and explain that Microsoft now is very positive of what is happening, just not who is causing it or why it�s happening.

What�s frustrating about this is that all the tools that can be used to look deeper into this problem aren�t available to me as a technician outside of Microsoft.� All I�ve been able to do for my client is set up triggers to cause �Exchange store.exe dumps� which are essentially process freezes followed by private memory dumps to disk.� The good thing is that the end users don�t notice, nor does the Windows 2003 Cluster service.� Also, our Microsoft support team has been great at sharing information with us.

But the problem still remains, that there is nothing at all that I can do to fix this problem.� I can�t run the debug programs (I can run a debug against the process, but not to the same level of detail, due to a lack of published information) that Microsoft has available, despite a very deep understanding of how the ESE runs the EDB, STM, and LOG files (for an outside consultant who just reads voraciously).� This inability to better service my customers frustrates me to no end, whether Microsoft�s technicians are fantastic or not (there have been other times�).

So, while I wait for them to get back to me on yet another dump that has been generated, looking for a very elusive fSearch() operation against one of my client�s many Exchange 2003 stores, I sit on my hands in anticipation, wishing to be able to do more.

We fixed it by having the DBA run:
xp_startmail
xp_stopmail
Then re-running the jobs. It seems to have worked, but I�m curious how we got hit by a fixed bug. My only thought is that it must have something to do with SQL thinking we essentially truncated the mailbox.

On anonther note, I�m writing up full instructions for how to change the IP on a DC, particularly for small offices with only 1 or 2 DCs.� It�ll post by end of the weekend.

Here�s why: it�s a terrible consumer device. It�s great for geeks like me who understand software is buggy and difficult to write, but I couldn�t imagine any of my end users dealing with this thing. Examples:

The phone is ok, but the Bluetooth radio randomly shuts off on me. The worst is in the middle of a conversation, my headset will go silent. And it�s ONLY ever in the car. So there you are, driving down the street, and having to fumble for a soft-button to turn on the speakerphone. Many MANY people will mention something or other about software upgrades required, and believe me, I have them. I had them before Sprint support knew what I was talking about. I�m still 100% up to date, according to Starcom�s site. No, I�m not going to run a 3rd party update to a business tool, like I would my own personal phone.

The device requires a reset every 3 days. I have 2 pieces of add-on software, and I�ve actually REMOVED 8 installs from Sprint (from the hidden ROM, so they don�t come back during a hard-reset). I installed Google Maps, because it�s fantastically worthwhile, and �TCPMP� for media playing (Windows Media Player doesn�t play Cisco UNITY voicemails sent to email). Out of the box, it required a reboot every 1-2 days. After 3 days of no reboots, ExchangeActiveSync stops reliably retrieving email. The touchscreen becomes wildly out of sync (I can�t hit the scrollbar anymore, usually). Phone calls actually hang up in the middle of conversations. And the phone finally won�t respond to any buttons or screentaps, save for the �power� button.

It Phantom-Dials. I�ll read an email, put the phone back in its holder, and walk down the hall. 45 seconds later, one of the recipients of the email will call me back asking me what I called them for. Best I can figure, is that the joystick is getting depressed, scrolling up, getting depressed again (on a recipient of the email), and the call button is getting hit. In the holder designed for the device, on my hip. I�ve been wearing cell phones on my hip for work since I was 19 – don�t tell me I�m hitting it with my elbow. Having to lock your phone in your pocket is understandable. But in it�s designed holster?

So, what DO I like about the phone?

I don�t have to open my laptop at home to watch email every night, in case something breaks. Exchange ActiveSync DirectPush in conjunction with SMS for alerts is fantastic. I turn on the sounds for text messages, but not email, and if it�s an actual issue, I get the alert, hear it, and respond. But if it�s a co-worker sending out a �I finished doing this overnight work.� email, I don�t get woken up. Better response time for the business, more sleep for me.

Charges from USB on my laptop. On the road, this has been a life-saver.

Can install all sorts of neat software. Games, document readers, etc. Adobe Acrobat for PPC and Microsoft Reader, combined, give me hours of reading material. Especially manuals for things I have to do the next day.

Pocket IE: www.weather.com, www.opentable.com, www.google.com (which has a great mobile device reformatter). I took just my phone to New York City for a 3 day weekend, and was able to book every dinner, find directions, and verify that Ferries were open on the days I wanted to go places.

Google Maps. I use this to check the traffic of my drive on my way to the car in the parking lot, so I can decide which way to drive home. Add accurate directions that I can pull from my contact list� wow.

Now I just have to figure out RAPIP, so I can plug it into my Ubuntu Feisty laptop and sync THROUGH the GNU/Linux OS, rather than just through WinXP.