Unit-test your mail server with eximunit

Many years ago, I was one of the volunteer admins for starship.python.net and mail.python.org. I also ran my own personal email server for several years before giving in and switching to Gmail. In both capacities, I regularly tinkered with the configuration of Exim, the MTA (message transfer agent, aka email server) used on all of those machines. Every time I did so, I was a little bit nervous that my change might break the existing configuration. So I did a flurry of manual testing each time.

Now I'm trying to wean myself off Gmail and go back to running my own email server. I still like Exim, so figured I'd stick with what I know best. But there's still that little problem: how do I know my email server configuration is correct, and how do I keep it correct while changing it?

Of course the answer is obvious to a programmer: automated tests! It turns out that I'm not the only one who has faced this problem. Unlike me, David North actually did something about it and wrote eximunit. The idea is simple: it runs exim -bhc (fake SMTP conversation) in a subprocess and tests that the fake SMTP server behaves as expected. Bogus recipients are rejected, good recipients are accepted, relaying is denied, etc.

Here's an excerpt from the test script for my personal email server, which accepts email to domains gerg.ca and lists.gerg.ca, but nothing else:

class ExternalTests(eximunit.EximTestCase):
    def setUp(self):
        self.setDefaultFromIP("192.168.0.1")

    def test_no_relay(self):
        session = self.newSession()
        session.mailFrom("spammer@example.com")
        session.assertRcptToRejected("victim@example.net", "relay not permitted")

    def test_known_recipients(self):
        session = self.newSession()
        session.mailFrom("random@example.com")
        session.rcptTo("*CENSORED*@gerg.ca")
        session.rcptTo("*CENSORED*@gerg.ca")
        session.rcptTo("*CENSORED*@gerg.ca")

In that last test, I'm obviously censoring three different and valid email addresses -- don't want to make things too easy for the bad guys.

Anyways, the idea is fairly simple: you create a session (which is just a wrapper for exim -bhc in a child process) and call methods like mailFrom() or rcptTo(). Those versions assert that the command works, returning the expected 250 response code. For negative tests, you use methods like assertRcptToRejected(), which tests that the response code is a failure (550 in this case) and that the rejection message is what you expect.

Of course, this is not a full end-to-end test for an email server. It says so right there in the name: eximunit. If you want functional testing or integration testing, perhaps you want something like Swaks? (I have not tried it.)

Author: Greg Ward
Published on: Apr 24, 2013, 2:51:53 PM
Permalink - Source code