Testing Emails in Automated Test Scripts with a Fake SMTP server: MailCatcher

Safe, Fast, and Reliable Email Testing with Selenium WebDriver and MailCatcher

This article is included in my “How to in Selenium WebDriver” series.

On a recent software project, I heard of an embarrassing mistake: emails had been sent to real customers during internal system testing.

CUSTOMERS SHALL NEVER RECEIVE TESTING EMAILS!

Unfortunately, this could happen to software testing after certain production data was loaded for verifying production issues when lacking good prevention procedures.

To avoid this, I have seen many solutions have been used, such as:

• Disable emails sending

• Replace all account emails with a fixed test email address

• An improved version of the above by including the recipient testuser+realusername@gmail.com

the characters after ‘+’ got ignored but will be still shown to reveal the real recipient

However, all of the above were complex and inadequate. My favourite (and simplest) solution is to use a fake SMTP server, such as MailCatcher. I have been testing emails this way for over 10 years, never had any problems. There are some alternative commercial email testing servers/services on the market, such as MailTrap (recommended plan costs $49/month). Even taking the cost out of the equation, I still think the free MailCatcher is far better.

Why fake SMTP Server makes sense for Testing emails?

The purpose of email testing is to ensure that

• the emails are received (to the correct address)

• the email content is correct

The actual sending/receiving via external Internet in the context of software testing does not matter. That has been taken care of by the SMTP protocol, and we don’t need to test every time.

Furthermore, there are three extra benefits of using a fake SMTP Server:

• email delivery is not guaranteed
  Email is only a ‘best effort’ delivery. Even though it is highly reliable by design, the delivery is not guaranteed. There are no such concerns with fake SMTP servers: the “sent” emails are guaranteed to arrive.

• More reliable
  As we know, sending/receiving real emails takes some time. With functional testing, the non-deterministic delay makes automated test execution less stable. With a fake SMTP server, there is no such problem. The delivery is instant.

• Speed
  Execution time is an important attribute in functional testing. Any time-saving technique is welcomed.

Requirements

Let’s examine the requirements for a fake SMTP server from the perspective of email testing.

• Easy Configuration
  If it is complex to set up and configure, infrastructure engineers tend not to use it (as their focus is on production setup).

• Fast

• View the emails in UI
  Visual inspection is a mandatory feature. We (especially manual testers and business analysts) want to see what the email looks like in an email client (e.g. Outlook).

• APIs to assist test automation

• Clear all the emails

• Search emails
  Help to find a specific email for verification.

MailtCatcher

MailCatcher is a free SMTP server written in Ruby, it is built to make testing emails easier.

• Installation
  After Ruby is set up, installing MailCatcher is as simple as one command:
  gem install mailcatcher

• Start MailCatcher
  mailcatcher

• Use MailCatcher as an SMTP server in test/integration servers

{ :address => '127.0.0.1', :port => 1025 }

• MailCatcher UI
  Open http://127.0.0.1:1080/ in your browser to view MailCatcher UI. MailCatcher UI is like a web email client with features such as filtering, viewing an individual email’s raw source, downloading attachments, …, etc. On top of that, a feature for testing: clearing emails.

• API
  MailCatcher provides RESTful APIs, such as /messages/id.json

Demonstration: reset password

I will demonstrate the use of MailCatcher with a common test case: Reset Password. During the test execution, the automation script needs to extract the reset link in the email.

Reset Password Automation in raw Selenium WebDriver (Testing tool: TestWise)

Please note that there were other emails in the MailCatcher, the automation script finds the latest reset password email by email subject.

Test Script

describe "Reset Password" do
  include TestHelper
  include MailCatcherHelper  

  before(:all) do
    @driver = Selenium::WebDriver.for(browser_type, browser_options)
   @driver.navigate.to(site_url)
  end

  it "User reset password by email" do
    visit("/sign-in")
    driver.find_element(:link_text, "Forgot password?").click       
    try_for(2) { expect(page_text).to include("Reset Password") }
    reset_password_page = ResetPasswordPage.new(browser)
    reset_password_page.enter_email("james@client.com")
    reset_password_page.click_reset_button
    sleep 0.75
    reset_url = nil
    open_email("Password Reset for WhenWise") {
      reset_url = driver.find_element(:id, "reset-url")["href"]
      puts reset_url
    }   driver.get(reset_url)
   change_password_page = ChangePasswordPage.new(browser)
   change_password_page.enter_new_password("pass02")
   change_password_page.enter_password_confirmation("pass03")
   change_password_page.click_change_password
   expect(page_text).to include("Password confirmation doesn't match Password")
   change_password_page.enter_new_password("pass02")
   change_password_page.enter_password_confirmation("pass02")
   change_password_page.click_change_password
   expect(toast_text).to include("Password has been reset!")   sign_in("james@client.com", "pass02")
   try_for(2) { expect(toast_text).to include("signed in successfully") }
  endend

The test script (raw Selenium WebDriver in Ruby) follows the Maintainable Automated Test Design.

The open_email is a reusable function defined in the test_helper.rb .

def open_email(subject_start_query, mailcatcher_url = "http://mail.server:1080", &block)
  driver.get(mailcatcher_url)
  email_subject = site_env ? "[#{site_env}] #{subject_start_query}" : subject_start_query
  if site_env =~ /ci/
    email_query = email_subject + " " + site_env
  else
    email_query = email_subject
  end
  mailcatcher_search(email_query)  sleep 0.75
  try_for(4, 2) { mailcatcher_open_message_from_top }
  sleep 1
  driver.switch_to.frame(0)
  sleep 0.75
  begin
    yield
  ensure
    driver.switch_to.default_content
    sleep 0.1
  end
end

The above helper function will do the following steps by using Selenium WebDriver:

• Open MailCatcher UI in the browser

• Search the email by the subject

• Open the first matching email

• Perform the operation such as extracting a link or verifying a piece of text in the email. (using Ruby Block here)
  reset_url = driver.find_element(:id, "reset-url")["href"]

Please note this test visits the pages in two domains and MailCathcer used iframes. This might not work with some so-called “test automation frameworks” such as Cypress, see this article: Why Cypress Sucks for Real Test Automation? (Part 2: Limitations).

With Selneium WebDriver, there are no such (or any) limitations, and it is easy!

An alternative (fast and more reliable) way is to use MailCatcher API:

url = mailcatcher_server + "/messages"  # REST API
messages = JSON.parse( RestClient.get(url))
messages = messages.reverse # latest first
result = result.select { |x| x["subject"].include?(email_subject) }
# ...

Advanced usage: Parallel execution with multiple test servers

I use 10 test server instances in the Continuous Testing process for my WhenWise app: ci1-whenwise , ci2-whenwise … ci10-whenwise . For every build in our Continuous Testing Server (BuildWise), 10 build agents run Selenium automated tests (500+) against these 10 test servers, in parallel.

This setup introduces challenges for email testing as there is only one MailCatcher server that is shared by 10 application servers. To help distinguish an email from a specific test server, I prefixed the server no, e.g. ci8 , in the email subject.