I recently had a co-worker trying to figure out a “lost email” problem we were possibly experiencing. He was blind to everything after hitting “send” because the intermediate servers weren’t sending delivery recipets, even though they were requested in the MIME header (Does any mail admin allow those to be sent out of the organization these days?) So, to help him out, I wrote up the following “how to test SMTP by hand” HOWTO.

First step is to determine which servers are responsible for mail delivery inbound for the domain you’re sending to. You do this by looking in DNS for the “MX” type records. These are provided in the format “priority servername.domain.” Priority is reverse-ordered. The easiest way to remember priority order is that it’s the order in which servers are attempted.

rob@rob-kubuntu3:~$ dig MX totalnetsolutions.net +short
10 docsmooth.isa-geek.net.
rob@rob-kubuntu3:~$ dig MX likewise.com +short
10 server1.inboundmx.com.
20 server2.inboundmx.com.

This tells you the servers, in order, that *all* mail will be sent to for the domain listed. So, anything to my likewise.com address will go to server1.inboundmx.com. The higher priorities are only used if the lower priorities fail to answer. If no server answers, the mail is held by the sender and retried, generally every 1 or 4 hours for up to 4 days, but this retry is configured on the *sending* server. That means, your own email admin (or you, if you’re the mail admin).

Next thing to check is: does the server work, and is it your sender, or their receiver? Check with telnet!
Stuff I type is in red:

rob@rob-kubuntu3:~$ telnet docsmooth.isa-geek.net 25
Trying 99.29.179.119...
Connected to docsmooth.isa-geek.net.
Escape character is '^]'.
220 totalnetsolutions.net Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at Tue, 31 May 2011 08:43:08 -0500
HELO
250 totalnetsolutions.net Hello [12.130.116.175]
MAIL FROM: me@me.com
250 2.1.0 me@me.com....Sender OK
RCPT TO:you@you.net
250 2.1.5 you@you.net
DATA
354 Start mail input; end with .
from:me@me.com
to:you@you.net
subject:test manually
test
test
.

250 2.6.0 Queued mail for delivery
quit
221 2.0.0 totalnetsolutions.net Service closing transmission channel
Connection closed by foreign host.

The last “.” is SUPER important – it tells the mail server when you’re done sending that email. You could use that channel to send other messages, rather than sending “QUIT” if you’d like. You might notice that I entered the “From” and “To” lines twice. The first entries are for the SMTP header (analogy would be the message envelope), and the second entries are for the MIME headings (analogy would be the return address header in a formal postal letter, if anyone sends those). The MIME headings are what most mail programs display, and actually don’t technically need to match the SMTP header (but if MIME and SMTP don’t match many anti-spam programs will throw out the message).

The MIME header is pretty complex, but not order-dependant, although I prefer to enter it in order, so that I can be sure I don’t miss anything.
If you want to add an attachment, just base64 encode it first with:
perl -e 'use MIME::Base64; qw(encode_base64); print encode_base64("@ARGV");' cat attachment-to-send.zip
Then you can just paste it into the email. In the MIME heading (right after the subject), just add (with the appropriate mime coding, probably application/octet-string:

------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: application/zip;
name="attachment-to-send.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="attachment-to-send.zip"
<paste your base64 encoded attachment/ >
Content-Type: multipart/mixed;
boundary="----=_NextPart_000_000D_01CC1C41.21F38080"
------=_NextPart_000_000D_01CC1C41.21F38080
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

Type your message here, ending with a “.” Standalone on a line.
.

Now that you know *how* to send an email message by hand, you can use the returned error codes to troubleshoot where the message may be disappearing. Remember, that this just gives you transport troubleshooting between yourself and the initial destination mail server. Many large (and even medium-sized) organizations will have a perimeter mail server which then forwards the message to one or more internal servers. If the mail is being dropped at that point, you’ll have to contact the reciever with the proof that their server is accepting your messages.

We ran into this bit of fun while setting up a NIS domain for testing in the lab today:
rob@rob-kubuntu3:~$ ypcat -d nisdom -h rhel5-64-2 passwd.byname
No such map passwd.byname. Reason: No such map in server's domain

It turns out this was a problem with the /var/yp/securenets file, but I’m still not sure what is wrong. The man page for ypserv shows:

A sample securenets file might look like this:

# allow connections from local host — necessary
host 127.0.0.1
# same as 255.255.255.255 127.0.0.1
#
# allow connections from any host
# on the 131.234.223.0 network
255.255.255.0 131.234.223.0

So we set up our securenets to look like this:

host 127.0.0.1
255.255.255.0 10.10.10.0

And tried to connect to the server:
rob@rob-kubuntu3:~$ ip addr show dev wlan0 |grep "inet "
inet 10.10.10.210/24 brd 10.10.10.255 scope global wlan0
rob@rob-kubuntu3:~$ ypcat -d nisdom -h rhel5-64-2 passwd.byname
No such map passwd.byname. Reason: No such map in server's domain
rob@rob-kubuntu3:~$ ping -c1 rhel5-64-2
PING rhel5-64-2 (10.10.10.213) 56(84) bytes of data.
64 bytes from rhel5-64-2 (10.10.10.213): icmp_req=1 ttl=64 time=0.823 ms

--- rhel5-64-2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.823/0.823/0.823/0.000 ms

Removing the /var/yp/securenets file allowed us access, so it wasn’t firewall or rpc or portmap issues, to the best I can determine. Adding “host 10.10.10.210” also worked and allowed the client access. So what’s wrong with the format / man page?

I upgraded the TNS lab this past week from Windows 2008 to Windows 2008 R2, including replacing the 4 Domain Controllers (rather than upgrading). It gave me a chance to review the procedure for moving a Certificate Server to a new system, which I hadn’t done since 2005. For those who haven’t tried, the procedure for moving a Certificate Server is reasonably well documented at the Microsoft Support site here: http://support.microsoft.com/kb/555012. The part of this that’s especially tricky, especially in our lab, is the renaming of the DC.

In our lab we have an empty forest root, as per the old (Windows 2000-era) Microsoft recommendations, to match several large customer environments. Because it’s a lab, and no clients connect to it, we only have a single DC. I snapshotted it as a backup, and went through the procedure to rename a domain controller, also well documented by Microsoft, this time at TechNet.

For review, the procedure we planned to run was:
netdom computername dc04 /add:dc01.lwtest.corp
netdom computername dc04 /makeprimary:dc01.lwtest.corp
shutdown -r -t 0
netdom computername dc01 /enum
netdom computername dc01 /verify
netdom computername dc01 /rem:dc04.lwtest.corp

I’m still not sure what caused it, but in this case, this command failed:
netdom computername dc04 /makeprimary:dc01.tns.lab
At this point, I couldn’t make the old name primary again (I would get an “Access Denied” error), so I rebooted to see which name had taken. And that’s where things went bad.

When the DC came up, we were getting this error: Netlogon EventID 5602. Source: NETLOGON, EventID: 5602, Data: “An internal error occurred while accessing the computer’s local or network security database.”

Because the DC rename hadn’t completed successfully, the computer couldn’t actually log into itself to load AD. Very bad for the root of the forest. I wasn’t able to find anything helpful in my searches, so thought I’d let you know the fix:

Name it back to the old name and try again:
Reboot into Safe Mode.
netdom computername localhost /makeprimary:dc04.lwtest.corp
shutdown -r -t 0

Boot normally
netdom computername localhost /makeprimary:dc04.lwtest.corp
netdom computername dc01 /enum
netdom computername dc01 /verify
shutdown -r -t 0

After *that* reboot, make sure, with the verify command, that the old name took, and that you can log in, and just try the rename again.

I couldn’t get the “rename back” to take untill after the attempt in safe mode. Strange, but it’s working great now! Hopefully this will help someone.

I had a Bourne Shell (sh) script I needed to capture the exit status of, but it was being run through “tee” to capture a log file, so “$?” always returned the exit status of “tee”, not the script. In a nutshell, it went something like this:
#!/bin/sh
DO_LOG=$1
LOGNAME="`hostname`.out"
if [ "$DO_LOG" -eq "1" ]; then
# Logging is turned on, so relaunch ourself with logging disabled, and tee the output to the logfile
sh $0 0 | tee $LOGNAME
exit $?
fi
#... Do lots of things in the script
exit $ERRORCODE

Now, the important thing here is that the script sets very specific error codes (we have 16 defined) based on different error states, so that a tool like HP Opsware can give us different reports based on the exit status. When run with “0″ for no logging, this works great, but it requires the controlling tool to capture logs, and not all do (especially cheap “for” loops in a shell script.)

But when run with logging enabled, all of the fancy error code handling (45 lines of subroutines’ worth) gets lost, because “$!” is equal to the status code of the “tee” command. Bash scripters out there will say “but what about $PIPESTATUS ?” If we could use bash, the code would be:
#!/bin/sh
DO_LOG=$1
LOGNAME="`hostname`.out"
if [ "$DO_LOG" -eq "1" ]; then
# Logging is turned on, so relaunch ourself with logging disabled, and tee the output to the logfile
sh $0 0 | tee $LOGNAME
exit ${PIPESTATUS[0]}
fi
#... Do lots of things in the script
exit $ERRORCODE

(Note the single line change in the conditional exit.)

But, I don’t have the luxury of bash (thanks AIX and FreeBSD and Solaris 8), so we needed to get fancy…
#!/bin/sh
DO_LOG=$1
LOGNAME="`hostname`.out"
if [ "$DO_LOG" -eq "1" ]; then
# Logging is turned on, so relaunch ourself with logging disabled, and tee the output to the logfile
cp /dev/null $LOGNAME
tail -f $LOGNAME &
TAILPID=$!
sh $0 0 >> $LOGNAME 2>&1
RETURNCODE=$?
kill TAILPID
exit $RETURNCODE
fi
#... Do lots of things in the script
exit $ERRORCODE

In this last example, we’re creating the empty logfile by copying /dev/null to the logname, then starting a backgrounded “tail” command on the empty file. Because we haven’t disconnected STDOUT in the backgrounding, we will still get the screen output we desire from “tail”. The script now only writes *its* output, with redirected STDOUT and STDERR, to the log file, which is already being tailed to the actual screen. At the end of the script, we capture the true exit code, clean up the tail ugliness, and exit with the desired status code.

This does have a serious downside that if the script encounters and error and exits, the “tail” is left running indefinitely on Linux and Solaris, since the kernel there will simply scavenge the process to be owned by init. So, if you take this method, be very careful to capture all errors you may possibly encounter. Or, just use a better scripting tool. 🙂

I have recently pushed the main ESX host for TNS to 70% overcommit on RAM, since upgrading to 4.1. Interestingly (expectedly), the performance now is the same as it was on 3.5 with 2 fewer VMs and only 50% overcommit. But, it’s still pretty poor in the “Lab” performance pool, even after changing that pool from “low” to “normal” shares. So we finally ordered new memory, doubling the server to 16gb. It goes in Sunday night, so we’ll see how things perform next week when Rob’s on site with customers.

« Previous Page — Next Page »