Testing Emails in Automated Test Scripts with a Fake SMTP server: MailCatcher

Safe, Fast, and Reliable Email Testing with Selenium WebDriver and MailCatcher

This article is included in my “How to in Selenium WebDriver” series.

On a recent software project, I heard of an embarrassing mistake: emails had been sent to real customers during internal system testing.

CUSTOMERS SHALL NEVER RECEIVE TESTING EMAILS!

Unfortunately, this could happen to software testing after certain production data was loaded for verifying production issues when lacking good prevention procedures.

To avoid this, I have seen many solutions have been used, such as:

• Disable emails sending

• Replace all account emails with a fixed test email address

• An improved version of the above by including the recipient testuser+realusername@gmail.com

the characters after ‘+’ got ignored but will be still shown to reveal the real recipient

However, all of the above were complex and inadequate. My favourite (and simplest) solution is to use a fake SMTP server, such as MailCatcher. I have been testing emails this way for over 10 years, never had any problems. There are some alternative commercial email testing servers/services on the market, such as MailTrap (recommended plan costs $49/month). Even taking the cost out of the equation, I still think the free MailCatcher is far better.

Why fake SMTP Server makes sense for Testing emails?

The purpose of email testing is to ensure that

• the emails are received (to the correct address)

• the email content is correct

The actual sending/receiving via external Internet in the context of software testing does not matter. That has been taken care of by the SMTP protocol, and we don’t need to test every time.

Furthermore, there are three extra benefits of using a fake SMTP Server:

• email delivery is not guaranteed
  Email is only a ‘best effort’ delivery. Even though it is highly reliable by design, the delivery is not guaranteed. There are no such concerns with fake SMTP servers: the “sent” emails are guaranteed to arrive.

• More reliable
  As we know, sending/receiving real emails takes some time. With functional testing, the non-deterministic delay makes automated test execution less stable. With a fake SMTP server, there is no such problem. The delivery is instant.

• Speed
  Execution time is an important attribute in functional testing. Any time-saving technique is welcomed.

Requirements

Let’s examine the requirements for a fake SMTP server from the perspective of email testing.

• Easy Configuration
  If it is complex to set up and configure, infrastructure engineers tend not to use it (as their focus is on production setup).

• Fast

• View the emails in UI
  Visual inspection is a mandatory feature. We (especially manual testers and business analysts) want to see what the email looks like in an email client (e.g. Outlook).

• APIs to assist test automation

• Clear all the emails

• Search emails
  Help to find a specific email for verification.

MailtCatcher

MailCatcher is a free SMTP server written in Ruby, it is built to make testing emails easier.