Flaky Tests in E2E Test Automation Clarified - Part C: Causes 1 - 3

Some causes of Flaky Tests in E2E Test Automation

In this series:

• Part A: Flaky Tests are Common but People tend to Neglect them

• Part B: Test Suite Size Matters *

• Part C: Causes 1 - 3

• Part D: Causes 4 - 6

• Part E: Causes 7 - 10 (upcoming)

• Part F: The Right mindset about Flakiness in E2E test automation (upcoming)

• Part G: Solutions 1 - 3 (upcoming)

• Part H: Solutions 4 -6* (upcoming)

• Part I: Solutions 7 -10* (upcoming)

• Part J: FAQ (upcoming)

Recap from Part 1: Flaky tests are common, even at Google.

“1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests.” — (Google Testing Blog, 2016)

That’s why I have been saying, “Real test automation engineers are extremely rare”. 95+% of test automation engineers are unable to handle flaky tests when the test suite reaches a certain size. So-called ‘flaky-handling’ test automation frameworks, such as Cypress, make things actually worse. The reason: inexperienced test automation engineers naively believe ‘flakiness’ would be handed by the automation framework, of course, this is very wrong and often leads them to be fakers. 

Think about it, Google engineers created Chrome browser and contributed the most reliable automation framework: Selenium WebDriver (W3C Standard). If handling flakiness is simple as Cypress's marketing team says, do you think Google would publish a 16% flakiness rate in its testing blog? For start, there are many causes contributing to flaky end-to-end (UI) tests, how could one or two so-called framework features solve? Don’t forget, real end-to-end test automation is aiming nearly 0% flakiness rate. 

In this article, I will clarify the causes of flaky tests in web test automation.

CAUSE 1: AJAX

Let me start with the one that made me aware of flaky tests: AJAX. 

An example web page that uses AJAX

Different from standard HTTP requests,