'Fix One Software Bug, Create Three More' Syndrome — Part 3: Solution: Automated End-to-End (UI) Regression Testing
A logical cure that many software professionals hesitate to adopt—yet it can make a real difference.
In this article series:
Part 1: Symptoms and Consequences
Part 2: Why is it happening? *
Part 3: Solution: Automated End-to-End (UI) Regression Testing
Part 4: Why is this syndrome still unresolved at many software teams?
It is not difficult to recognize that the remedy for the "Fix One Software Bug, Create Three More" syndrome is regression testing—specifically, well-implemented automated end-to-end (E2E) regression testing. Yet, only a small fraction of software professionals truly understand regression testing, let alone trust or embrace it.
Robert’s Story Continued
In Part 1, I shared a story about a senior contractor developer Robert, who has 'Fix One Software Bug, Create Three More' Syndrome.
Robert realized this, too. He once proposed a team meeting to share his newly discovered approach to regression testing:
Invoke a test to send input to the application (a workflow product), and save the output to a file, e.g. case1–1209.txt
On the next build candidate, repeat the above to save to a different output file.
Run a diff tool to compare these two files; if they are the same, no regression issues.
I was quite shocked, this naive ‘regression testing’ demo was well received by other team members. The tech lead even said, “cool”.
Then I came up and showed my regression testing (PM requested me to), which I used for my own work for a while. I picked up a case like Rob’s. Different from Rob’s regression testing approach, mine is simpler:
Verify every step inside the test scripts
If all steps pass, the regression run of the test case passes! And I showed a report of the execution history (daily for ~3 weeks) of my test suite (~50 test cases).
There was a moment of silence after my demo. Clearly, everyone realized this was real regression testing, as it did real verification.
The most important type of Regression Testing is End-to-End (UI), not Unit Testing
A common misconception among senior software developers is that unit testing alone suffices for regression testing—but this is not the case. While unit tests can help catch some regression errors, they are far less effective in addressing the types of defects associated with this syndrome.
Why? Because the “three new regression bugs” introduced by a bug-fix check-in are usually discovered from the end user’s perspective, through the UI during real usage, or end-to-end testing.
Although unit tests run very quickly—often fast enough to execute the entire suite within an IDE (as I did on a Java project I led: ~3,000 unit tests in under 2 minutes)—and are typically integrated into even the most basic CI pipelines, this-syndrome-defects still often slip through undetected.
For more on effective unit testing, see Michael Feathers’ well-known article “A Set of Unit Testing Rules.”
There Are Many Other Great Benefits of Automated E2E (UI) Testing
There are many benefits that many software professionals are unaware of, pity! Check out my other articles.
E2E Test Automation Regression Testing is Far More Preferrable than a Formal Code Review PR Process
Real End-to-End UI Automated Regression Testing: The Best Weapon for Addressing Technical Debt
Automation Assist, Part 3: Test Data Generation (coming soon)
Benefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
Why Don’t I Use Defect Tracking? No Need, I do real Continuous Testing.
Why I don’t use Jira and Confluence at all for my software development?
If you're unsure about the significant benefits of real end-to-end test automation, check out this excellent presentation by Facebook.
If you think real automated end-to-end (UI) regression testing is something only the world’s top IT firms can do, check out my showcase.
Related reading:
