How to Implement Real Automated Regression Testing?
Strategy and practical advice on successful automated regression testing to enable daily production releases.
In “Regression Testing Clarified”, I shared my thoughts and experience on the Whats and Whys of automated regression testing. In this article, I will show Hows.
Table of Contents
· Strategy: Run all E2E Tests daily
· Discipline: Fixing broken builds is the highest priority
· How to Start?
· 100% coverage is impossible and unnecessary
· Specific AdviceStrategy: Run all E2E Tests daily
Set a clear goal and work towards it.
Goal: run all automated E2E UI tests every day and get a Green build by the end of the day.
Of course, there are exceptions, such as a quite bad check-in, achieving a green build every day is not always possible. However, every team member needs to understand the team’s priority: GREEN BUILD, passing all automated (unit, API, and more importantly, E2E UI) tests.
Some readers might wonder:
“This sounds easy.”
No, it is not with a large test suite. For example, there are 28000+ test steps for WhenWise’s (one of my apps) automated regression suite. A green build means each of these 28,000 test steps must pass.
I am not trying to scare you off. Focusing on the goal every day will make this challenging task easier, “Slow And Steady Win The Race”.“This is too vague.”
I do have a list of specific advice in the next section. Here, I will highlight one important attribute of testing: feedback. Focus on the goal and everything will become natural if you adjust your approach based on feedback. During the journey (Test Automation and Continuous Testing), if you encounter challenges or difficulties, they are the feedback to you. Be open-minded and make changes accordingly to achieve the goal daily. I will show you some examples here:
(1) The execution of certain test scripts fails intermittently.
This means that your test script is not reliable. Check out this article: Working Automated Test ≠ Good Reliable Test.
Or your tests can not handle the waits for dynamic operations (such as AJAX) properly. Check out this article: Test AJAX Properly and Efficiently with Selenium WebDriver, and Avoid ‘Automated Waiting’
(2) Can’t open the app in two tabs (like this).
You might have been using Cypress or alike, and this issue is one of its known limitations (among many others). Switch to the free and better Selenium WebDriver.
(3) Our Business Analysts want to review/run automated tests, but they found it challenging.
Are your test scripts written in Ruby? If not, switch to Ruby which has much better syntax. Are you running test scripts in a programmer’s tool? You may instead want to try TestWise, a dedicated testing IDE.
(4) Our team spends a lot of time modifying test scripts to keep up with the app changes.
The design of your test scripts is not good (often over-engineered). Check out Maintainable Automated Test Design.
Or your testing tool is not efficient. For example, for one test step failure, do you need to rerun from the very beginning every time when debugging it? That would be very unproductive. You should find a tool that does this: “Attach test execution to the existing browser”.
(5) Running the whole test suite takes too long (over 1 hour).
Set up a Continuous Testing Server and run tests in parallel on multiple build agents. Check out my book “Practical Continuous Testing” and the article, Case Study: Rescue Unreliable 20 hours of Automated Regression Testing in Jenkins ⇒ 6-Minute Highly-Reliable in BuildWise CT Server.
…
By focusing on the goal (daily green build), you will be better positioned to identify issues and find solutions.
Discipline: Fixing broken builds is the highest priority
Kent Beck, Father of Agile, wrote down a story about Toyota Production System (in the first Agile book, Extreme Programming Explained: Embrace Change) describing this practice. It has been proven the most beneficial and efficient.
My advice to managers: when there is a broken build, treat everything else, except urgent production issues, as secondary. There should be a code freeze and no check-ins unless they are related to resolving the broken build.
Important Advice: Framework first, choose a productive tool afterwards
Stay with the free, open-source framework WebDriver (Selenium for web apps, Appium with Desktop and Mobile apps) which is a W3C standard. It is reliable, fast, feature-complete, well documented and supported, .., etc. In short, it ticks all the boxes. It is worth mentioning again: Facebook achieved “Release twice daily” with Selenium WebDriver.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Don’t be fooled by the so-called new automation framework or tool. Don’t even bother checking fancy tool demos; focus on the framework first. A simple fact is that: if the test framework is limited or syntax is complex, your team simply won’t be able to maintain the test suite.
It is okay if not be able to automate certain business features:
If your team cannot automate certain business features, that’s okay as long as it is limited to a small scope. Maybe it is just hard, or the team lacks the knowledge (seek external help if you can find a good test automation coach). In addition, you can still supplement it with a certain degree of manual regression testing.
If a team do have ambitious goal to automate most of business features, should consider seeking help for professional test automation coaches. A good coach can guide you quickly with his/her experience (all web sites work pretty much the same). This is also a good reason that you should choose a standard-compliant Selenium WebDriver.
As long as you meet the goal (daily green build), the regression suite will provide value.
Active Learning:
The chances are that none of your engineers has seen real test automation and Continuous Testing. Please read this article: “Test Automation and Continuous Testing Competition Week” — A Practical and Fun Way to Create a Successful Test Automation and Continuous Testing Formula in Your Company.
Don’t let mediocre (judged by skills, not the job title) software engineers or fake test automation engineers ruin it.
As I have mentioned earlier, most so-called senior engineers have negative or zero knowledge about test automation and CT. They are usually quite nice people but went to wrong path blindly, and unfortunately, more often than not, they defend their wrong choices later. They are not your enemies.
First of all, the team need to be clear about your goal, and be specific. It is OK let them try their options, but put measurements in. If they fail earlier, it will be better as there won’t be much embarrassment.
The most cost-effective way is to engage real test automation and CT coach. Real test automation and CT coaches are rare. Please don’t just take the candidate’s words for it, verify them as in the Test Automation and Continuous Testing Competition Week with higher expectations (they are claimed to be coaches). The fake ones can be identified easily. Though those people are good at talking, their hands will shake, and then they will give up when they are asked to do hands-on testing.
How to Start?
Start writing the first test today! Don’t waste time planning it. I was shocked at one agile transformation team lead’s talk: “We have been working on a roadmap to implement automated regression testing for 9 months”.
In test automation, the test maintenance effort is far greater than test creation (for more information, read this article: Is Your Test Automation on Track? Maintenance is the key). If a team has no confidence to start on Day 1, in my opinion, the battle has already been lost.
Which tests to start with today? The generic and simple ones, such as “User Login”, “Sign up”, “Reset Password”, “Access Privilege checks”, and “Change profile/password”. These scenarios require little business knowledge.
The purpose is NOT the test count, it is the value (i.e. safety net) we are looking for.
After that, move to the core cases quickly. There are two approaches:
User Stories for the current sprint
Why not automate the test cases for the user stories in the current sprint? They need to be tested anyway, and the team members shall be familiar with them. (some might refer to this as Shift-Left testing or In-Sprint Test Automation)Existing regression testing suite
Every software company has identified a set of regression tests (sometimes known as user acceptance tests) for its running-in-production apps. Pick the ones with the most value. For example, “an applicant can submit a claim” for an insurance app is a good candidate.
New tests shall be automatically added to the regression test suite, which will run daily (still remembering the goal?). Please be reminded, to achieve at least one green build (passing all E2E tests) in a Continuous Testing server every day.
100% coverage is impossible and unnecessary
A common mistake in a software company’s automated regression testing plan: aiming for 100% coverage (of features). Be realistic, the target is not achievable, nor necessary. It is okay to not have full coverage. In fact, 20% of coverage is quite good already. Every valid regression test has its value. Focus on the stability and validity of the tests, rather than a plan of hype.
With an increasing number of regression tests, the challenge to a green build grows dramatically. Anyone with an engineering mindset would understand that. As a matter of fact, besides my client projects, I have never seen one single software team that can maintain over 50 user-story level E2E (UI) tests in my city (that run relatively reliably daily, which is Level 2 of AgileWay Continuous Testing Grading).
According to the famous 80/20 rule, if you can automate 20% of the regression suite (and more importantly, keep them running daily), you will get 80% of the value. What about the rest that you are not able to automate reliably? Manual testing. This would still be a significantly better situation than full manual regression testing, isn’t it?
Some might say: “I want more, at least 95%, which will enable me to push to production daily, like Facebook.” This is admirable. However, are you ready to invest the effort?
I have been doing daily deployment (if changes were made on that day) for all of my apps: TestWise, ClinicWise, BuildWise, SiteWise, and WhenWise, for over nine years.
I spend ~70% of all software development efforts (including management, design, coding, BA work, deployment, testing, maintenance, meetings, defects, …) on Test Automation and Continuous Testing. Every working day, I spend 70+% time using TestWise (test automation IDE) and BuildWise (CT server). This makes me at least 10X more productive (I created and maintained all my apps in my spare time). Will you do that? Unless you are a CEO/CIO who really cares about quality, productivity, and huge cost-saving, think twice before trying to promote this at work. Surely, this is the right thing to do according to the spirit of Agile, but will your colleagues and managers be comfortable? Remember, software development is a human activity.
Therefore, my advice is: to be comfortable with 20%, and increase the safety net as you go.
Specific Advice
Following the above strategies, given some time and under good leadership, some of your motivated engineers will come up with a solution. As an executive, you need to do objective assessments and time-box prototypes (no longer than 5 days).
If you want to fast-track the process, copy an existing successful formula. As a matter of fact, the practices of E2E test automation and Continuous Testing are almost universally applicable to all software projects.
My specific advice:
Use raw Selenium WebDriver as the automation framework for web apps
Use RSpec as the syntax framework, and therefore, Ruby as the scripting language (see why?)
I was a veteran Java programmer. Learning Ruby is one of the best decisions I have made in my career. The bonus: Ruby is quite easy to get started.Use TestWise or Visual Studio Code as the test scripting IDE/editors, though some other tools are okay too. Two rules: High Productivity and the same tool for all team members (including BAs, managers, architects, manual testers, and customers).
No big planning. Start small, slow, and steady. Day 1: 2 tests; Day 2: 3 tests, …, etc. Important note: keeping existing tests up to date (i.e. maintenance) requires much more effort than test creation.
Set up a Continuous Testing server (like BuildWise), not Jenkins and Bamboo alike CI servers, and run all E2E UI tests on Day 1.
Must run ALL E2E UI tests in the CT server at least once per day.
On broken builds (which will happen a lot), suspending other work and fixing the broken build is the №1 priority.
Get all team members (business analysts, programmers, manual testers, and even customers) to utilize test automation daily. For example, a business analyst may run a test case in TestWise to navigate to a specific page or use the generated test data from a recent build on BuildWise.
No internal defect tracking at all. Defect tracking is an illusion, with real test automation, it is (at least internally) unnecessary (see why).
Once the regression suite reaches a certain size (covering > 95% of core features), push the build to production if all tests have passed in the CT Server.
Show great respect to Test Automation Engineers (real ones, of course), and pay them well as they do a much harder job than software engineers.
Once the total number of UI tests reaches 25+, set up a parallel testing tab, i.e. distribute tests to run on multiple build agents in parallel. Read my book “Practical Continuous Testing” for more on this topic.
Have a small celebration after a green build in the CT server and production release, until it becomes the new normal (i.e. multiple times a day)
I will end this article with a quote from the classic “Implementing Lean Software Development: From Concept to Cash” book: “That is a very difficult discipline to put in place, but once it is working, no one would ever think of going back to the old way of doing things.”