Flaky Tests in E2E Test Automation Clarified — Part B: Test Suite Size Matters

Why we commonly see “excited to announce a new test automation attempt with a new framework/tool, a few weeks/months later, wind up quietly”?

In this series:

• Part A: Flaky Tests are Common but People tend to Neglect them

• Part B: Test Suite Size Matters *

• Part C: Causes 1 - 3

• Part D: Causes 4 - 7

• Part E: Solutions 1-3

• Part F: Solutions 4 -7*

• Part G: FAQ

Test Count, an important factor for Test Automation Success

For a small suite (< 20 end-to-end tests, Level 1 of AgileWay Continuous Testing Grading), getting a green run (ALL PASS) is quite easy. Usually, the starting tests are simple ones too, such as “user login” and “user sign up”.

With an increasing number of test scripts, the maintenance effort grows exponentially.

Image credit: https://www.marketfit.co/six-stages-of-exponential-growth/

Reminder: ALL PASS means all test scripts pass, i.e., every single test step passes. The flakiness will show up, more and more, even a test script self was of high-quality, as there are inter-dependencies between tests. Non-professional (>95+%) test automation engineers get stuck here.

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015), author of “How we test software at Microsoft”

“Automated testing through the GUI is intuitive, seductive, and almost always wrong!” - Robert C. Martin, co-author of the Agile Manifesto, on his blog (in 2009)

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, software legend, in a podcast (2018)