Regression Testing Clarified
Regression testing needs to be automated and run in a Continuous Testing server multiple times a day.
This article is one of the “IT Terminology Clarified” series.
Between 2010–2013, I spoke at a few international software testing conferences (and decided no more, see reasons here). Quite commonly, there were pre-attendance surveys on the attendants’ interested topics. From my memory, the top three (the order might vary) were the same.
Test Automation (End-to-End)
Regression Testing
Web App Testing
However, topics on regression testing were rare at software testing conferences. Several attendants told me this at different events (after attending mine).
In this article, I will share my thoughts and experience in regression testing. This article is a bit long, with the following sections:
Status of Regression Testing in Software Projects (quite sad)
Regression Testing must be automated, however, …
How often do you do regression testing?
(correct answer: multiple times a day)How long should full regression testing take?
(correct answer: about 30 minutes or less)
For readers who are interested in the implementation, check out this article: How to Implement Real Automated Regression Testing?
1. What is Regression Testing?
Regression Testing in software means “re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change” (Wikipedia). Every software team member understands the objective of regression testing, as we often see regression errors. However, not many software engineers (including testers) understand how to do regression testing properly.
2. Status of Regression Testing in Software Projects
Most software projects conduct regression testing manually. As a result,
Long feedback loop
Commonly, manual regression testing takes a few weeks or months in some software projects.Expensive
Many human resources and a long time, which means high costs in software development.
Also, there are costs associated with the server/resources tied to a long regression testing cycle.
Limited Coverage
Due to the high cost and time constraints (of manual regression), regression testing coverage is small, often getting smaller. In the end, regression testing becomes limited to the business features the team thought it has been impacted by the new changes. Of course, it is wrong. The nature of software development is that a change might impact unknown parts of software, that’s why we need regression testing in the first place.Management overhead
Besides major releases, there are minor updates and production fixes. All these require regression testing, some do it on a much small scale.
The risk with production fixes may be small, but risks are risks. I prefer no risks if possible by a proper and fast regression testing process. (I always do full regression testing because I do it efficiently with automation.)Low Morale for testers
Obviously, nobody likes to do repetitive tasks, after one or two times.
With so many shortcomings, why do most (my estimation is >95%) software projects still do manual regression testing?
3. Regression Testing must be automated, however, …
There are many reasons such as lacking test automation skills and experience in managing test executions in a Continuous Testing (note, not CI) server. Often, “IGNORANCE IS STRENGTH”, engineers were faking test automation and CT. As a result, we see this often in so-called ‘agile’ projects:
a handful of test automation engineers work on something no one knows or cares about.
A large group of manual testers conduct manual functional testing slowly. Most of their efforts are spent on regression testing.
Of course, there are no excuses for such costly and inefficient regression testing in agile projects. But most managers/executives chose to be ignorant.
4. Manual Regression Testing is Evil in Agile
Manual Regression Testing is bad not only because of its high cost and lengthy time. The quality of testing suffers too.
The best adjective I can relate to manual regression testing is “CRUEL”, the best noun would be “SWEATSHOP”. The best adjective for manger and tech leads, who let manual regression testing happen, is “INCOMPETENT”. One purpose of using software is automation, how can we let poor manual testers do these repetitive tasks again and again?!
If a project’s regression testing is mostly done manually, it surely is NOT Agile.
“Using JIRA does not mean Agile; End-Users do not care story points or velocity charts. They don’t mind not-perfect new features, but they will get annoyed if existing features were broken by updates. So, as software professionals, we need to do regression testing properly” — Zhimin Zhan
5. How often do you do regression testing?
Many software professionals often provide a vague answer to this question. Generally speaking, it depends on how often you push releases to production.
(The below are two typical production release cycles in real agile projects. I exclude fake agile projects in which a production update occurs every 1+ months. My advice is to those projects is to drop everything and review their process seriously. Stop estimating story points is a good start)
1. Every Iteration (sprint in Scrum), typically 2 weeks
2. Every day, typically twice a day
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — from the presentation of Damien Sereni (engineering director at Facebook) at Selenium 2013 Conference
BTW, I have practised daily releases for over 12 years. When I got a green build in my BuildWise CT server, I pushed the build to production immediately.
6. How long does it take?
You might have heard of “10-minute build” in some agile talks. The origin of ‘10-minute build’ is the classic (and the first) Agile book by Kent Beck, the Father of Agile.
But Kent did not say specifically what a build means. There are several understandings:
Compile and Package, such as .war file for a Java app
Compile, Unit Testing and Package
Compile, Unit Testing, Code Coverage, Integration/API testing and Package
Compile, Unit Testing, Integration Testing, Package, Deploy and E2E (UI) testing
My take is № 4. If a green build (pass all tests) in a CT server, I can push the build to production. For most software companies, a 10-minute build (of №4) seems impossible, but it can be achieved.
How could Facebook achieve (not always though, I strongly recommend watching the presentation) that? The picture of its parallel testing lab below might answer the question.
Facebook takes automated regression testing seriously, as it enabled “release twice a day”, which is “the heart of its culture” (see the quote earlier). By the way, Facebook uses the free Selenium WebDriver. So, don’t be fooled by the salesmen of expensive commercial tool vendors such as Micro Focus, Tricentis, SmartBear, …, etc. The best is free.
Some engineers might be overwhelmed by Facebook’s efforts and determination (from the top executive) in Test Automation and Continuous Testing. Tech leads, before you come up with excuses, there is possible that small and medium sized IT companies can achieve good regression testing time (around 30–59 minutes) with a very tight budget (~$200 to $400 per month).
Before I show you how to achieve that (in a separate article in the coming days), I recommend the executives/managers have a realistic assessment of the capability and openness of your engineers. Surely, daily automated regression testing (desired by most CIOs) will have a huge impact on software development. Wired magazine wrote this in the article “The Software Revolution Behind LinkedIn’s Gushing Profits”: “completely overhauled how LinkedIn develops and ships new updates to its website and apps”. It is worth repeating: “completely overhauled”. If an IT executive thinks: “Buying an expensive tool or engaging some expensive consultants will get CT implemented”, they are dreaming.
From my experience, after initial hesitation or objection, most engineers would thank you for making this happen. Once implemented, life would be better for everyone.
“This (Continuous Integration, in the context, the authors means ) is a difficult discipline, partly because developing a rapid build and test capability requires time, skill, and ongoing attention, and partly because stopping to fix problems can be distracting. But a great epiphany occurs once a software development team finds ways to do rapid builds and continuous integration and everyone faithfully stops to fix problems as soon as they are detected. We have been told many times that this is a very difficult discipline to put in place, but once it is working, no one would ever think of going back to the old way of doing things.”
- Implementing Lean Software Development: From Concept to Cash, by Mary Poppendieck and Tom Poppendieck
Why don’t software teams do automated regression testing?
The short answer is that the tech leads usually don’t understand test automation, or worse, they think they do. I was in that position (~2005) before; it took me about 2-years of learning and practising test automation (at work during the day and at home during the night). Since then, I have been doing hands-on test automation and Continuous Testing every working day (for my clients and my own apps).
Let me share a story.
Many years ago, I joined a large .NET project as a contract software engineer. Technically, the project was a mess. To give you a feel, the configuration engineer (whose main role is to build the software package) once yelled with excitement: “It compiled!”.
The time I joined was after the first internal release, and many quality (functional) issues were reported. The star programmer Rob, who was a senior technical consultant at a well-known international software consulting firm, proposed automation, especially for regression. A few weeks later, an internal demonstration meeting was arranged.
The newly joined project manager, who worked with me on the previous project, suggested that I show my approach. I accepted, which I later thought was a mistake. For this reason, I remember this session quite well.
In the meeting, Rob demonstrated his approach to regression testing:
Invoke a test to send input to the application (a workflow product), and save the output to a file, e.g. case1–1209.txt
On the next build candidate, repeat the above to save to a different output file.
Run a diff tool to compare these two files; if they are the same, no regression issues.
I was quite shocked, this naive ‘regression testing’ demo was well received by other team members. The tech lead even said, “cool”.
Then I came up and showed my regression testing, which I used for my own work for a while. I picked up a case like Rob’s. Different from Rob’s regression testing approach, mine is simpler:
Verify every step inside the test scripts
If all steps pass, the regression run of the test case passes! And I showed a report of the execution history (daily for ~3 weeks) of my test suite (~50 test cases).
There was a moment of silence after my demo. Clearly, everyone realized this was real regression testing, as it did real verification. Comparing the output approach was no good:
if there is timestamp info, the output will never be the same
adding an extra
printstatement will cause ‘regression failure’we have no idea (from examining the output) which step failed.
there were no assertions, i.e., no testing.
They probably were embarrassed (for their reactions to Rob’s demo) after seeing my demo. The PM was wise to notice the awkwardness. He stood up and quickly wrapped up the meeting.
Of course, nothing changed after the meeting. I still did my own regression testing; No one talked about automated regression testing anymore. However, it seemed there was a gap between other engineers and me. That’s why I said it was a mistake to show my approach.
It is not those engineers' fault, though. It is the failure of IT education and training. We have programmers studying for 3 or 4 years for an IT bachelor's degree but barely touch on functional test automation and continuous testing. But in a typical software team nowadays, the number of testers often doubles the programmers' count. And strangely enough, very few software companies reach for external help on training and coaching test automation and CT.