Testing Emails in Automated Test Scripts with a Fake SMTP server: MailCatcher
Safe, Fast, and Reliable Email Testing with Selenium WebDriver and MailCatcher
This article is included in my “How to in Selenium WebDriver” series.
On a recent software project, I heard of an embarrassing mistake: emails had been sent to real customers during internal system testing.
CUSTOMERS SHALL NEVER RECEIVE TESTING EMAILS!
Unfortunately, this could happen to software testing after certain production data was loaded for verifying production issues when lacking good prevention procedures.
To avoid this, I have seen many solutions have been used, such as:
Disable emails sending
Replace all account emails with a fixed test email address
An improved version of the above by including the recipient
testuser+realusername@gmail.com
However, all of the above were complex and inadequate. My favourite (and simplest) solution is to use a fake SMTP server, such as MailCatcher. I have been testing emails this way for over 10 years, never had any problems. There are some alternative commercial email testing servers/services on the market, such as MailTrap (recommended plan costs $49/month). Even taking the cost out of the equation, I still think the free MailCatcher is far better.
Why fake SMTP Server makes sense for Testing emails?
The purpose of email testing is to ensure that
the emails are received (to the correct address)
the email content is correct
The actual sending/receiving via external Internet in the context of software testing does not matter. That has been taken care of by the SMTP protocol, and we don’t need to test every time.
Furthermore, there are three extra benefits of using a fake SMTP Server:
email delivery is not guaranteed
Email is only a ‘best effort’ delivery. Even though it is highly reliable by design, the delivery is not guaranteed. There are no such concerns with fake SMTP servers: the “sent” emails are guaranteed to arrive.More reliable
As we know, sending/receiving real emails takes some time. With functional testing, the non-deterministic delay makes automated test execution less stable. With a fake SMTP server, there is no such problem. The delivery is instant.Speed
Execution time is an important attribute in functional testing. Any time-saving technique is welcomed.
Requirements
Let’s examine the requirements for a fake SMTP server from the perspective of email testing.
Easy Configuration
If it is complex to set up and configure, infrastructure engineers tend not to use it (as their focus is on production setup).Fast
View the emails in UI
Visual inspection is a mandatory feature. We (especially manual testers and business analysts) want to see what the email looks like in an email client (e.g. Outlook).APIs to assist test automation
Clear all the emails
Search emails
Help to find a specific email for verification.
MailtCatcher
MailCatcher is a free SMTP server written in Ruby, it is built to make testing emails easier.
Installation
After Ruby is set up, installing MailCatcher is as simple as one command:gem install mailcatcher
Start MailCatcher
mailcatcher
Use MailCatcher as an SMTP server in test/integration servers
{ :address => '127.0.0.1', :port => 1025 }
MailCatcher UI
Open http://127.0.0.1:1080/ in your browser to view MailCatcher UI. MailCatcher UI is like a web email client with features such as filtering, viewing an individual email’s raw source, downloading attachments, …, etc. On top of that, a feature for testing: clearing emails.API
MailCatcher provides RESTful APIs, such as/messages/id.json
Demonstration: reset password
I will demonstrate the use of MailCatcher with a common test case: Reset Password. During the test execution, the automation script needs to extract the reset link in the email.
Reset Password Automation in raw Selenium WebDriver (Testing tool: TestWise)
Please note that there were other emails in the MailCatcher, the automation script finds the latest reset password email by email subject.
Test Script
describe "Reset Password" do
include TestHelper
include MailCatcherHelper
before(:all) do
@driver = Selenium::WebDriver.for(browser_type, browser_options)
@driver.navigate.to(site_url)
end
it "User reset password by email" do
visit("/sign-in")
driver.find_element(:link_text, "Forgot password?").click
try_for(2) { expect(page_text).to include("Reset Password") }
reset_password_page = ResetPasswordPage.new(browser)
reset_password_page.enter_email("james@client.com")
reset_password_page.click_reset_button
sleep 0.75
reset_url = nil
open_email("Password Reset for WhenWise") {
reset_url = driver.find_element(:id, "reset-url")["href"]
puts reset_url
} driver.get(reset_url)
change_password_page = ChangePasswordPage.new(browser)
change_password_page.enter_new_password("pass02")
change_password_page.enter_password_confirmation("pass03")
change_password_page.click_change_password
expect(page_text).to include("Password confirmation doesn't match Password")
change_password_page.enter_new_password("pass02")
change_password_page.enter_password_confirmation("pass02")
change_password_page.click_change_password
expect(toast_text).to include("Password has been reset!") sign_in("james@client.com", "pass02")
try_for(2) { expect(toast_text).to include("signed in successfully") }
endend
The test script (raw Selenium WebDriver in Ruby) follows the Maintainable Automated Test Design.
The open_email
is a reusable function defined in the test_helper.rb
.
def open_email(subject_start_query, mailcatcher_url = "http://mail.server:1080", &block)
driver.get(mailcatcher_url)
email_subject = site_env ? "[#{site_env}] #{subject_start_query}" : subject_start_query
if site_env =~ /ci/
email_query = email_subject + " " + site_env
else
email_query = email_subject
end
mailcatcher_search(email_query) sleep 0.75
try_for(4, 2) { mailcatcher_open_message_from_top }
sleep 1
driver.switch_to.frame(0)
sleep 0.75
begin
yield
ensure
driver.switch_to.default_content
sleep 0.1
end
end
The above helper function will do the following steps by using Selenium WebDriver:
Open MailCatcher UI in the browser
Search the email by the subject
Open the first matching email
Perform the operation such as extracting a link or verifying a piece of text in the email. (using Ruby Block here)
reset_url = driver.find_element(:id, "reset-url")["href"]
Please note this test visits the pages in two domains and MailCathcer used iframes. This might not work with some so-called “test automation frameworks” such as Cypress, see this article: Why Cypress Sucks for Real Test Automation? (Part 2: Limitations).
With Selneium WebDriver, there are no such (or any) limitations, and it is easy!
An alternative (fast and more reliable) way is to use MailCatcher API:
url = mailcatcher_server + "/messages" # REST API
messages = JSON.parse( RestClient.get(url))
messages = messages.reverse # latest first
result = result.select { |x| x["subject"].include?(email_subject) }
# ...
Advanced usage: Parallel execution with multiple test servers
I use 10 test server instances in the Continuous Testing process for my WhenWise app: ci1-whenwise
, ci2-whenwise
… ci10-whenwise
. For every build in our Continuous Testing Server (BuildWise), 10 build agents run Selenium automated tests (500+) against these 10 test servers, in parallel.
This setup introduces challenges for email testing as there is only one MailCatcher server that is shared by 10 application servers. To help distinguish an email from a specific test server, I prefixed the server no, e.g. ci8
, in the email subject.