Why do E2E Test Automation Attempts Repeatedly Fail Quickly at Software Companies? Part 1: The Hard Truth
Given the facts presented, it is inexcusable.
I start with a quote from a world authority in Software Development and Agile:
“Automated testing through the GUI is intuitive, seductive, and almost always wrong!” — Robert C. Martin, co-author of the Agile Manifesto, on his blog (in 2009)
And, one by the author of “How we test software at Microsoft” book.
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)
Also, check out a great story told by Michael Feathers (renowned agile expert and the author of ‘Working Effectively with Legacy Code’) in 2009.
One veteran manual tester once told me, “This is the fourth time this company (large, 500+ IT staff) attempted to implement E2E Test Automation. Every time, people started with excitement; Then very quickly, no one was not interested in it any more, until the next time”.
He learned web test automation by getting me to mentor him “one question per day”, and achieved far higher than anything he has seen over the last decade (the failed four attempts cost the company tens of millions of dollars 💰). Remember, he was a manual tester. How?
Effective problem-solving begins with understanding the problem. Unfortunately, many self-proclaimed ‘senior automated testers’ lack a solid grasp of the fundamentals of end-to-end (E2E) test automation — a shortcoming largely rooted in the failures of IT education and training in software testing. Have you learned E2E test automation through a university course or on-the-job training? My guess is NOT.
Nevertheless, a qualified engineer will analyze the issue objectively and proactively seek a solution.
The Fundamentals of E2E Test Automation
Just briefly.
E2E means an end-user can verify test execution.
Some will say, “Zhimin, of course it is. Why are you saying this (so obvious)?”. Really, have you heard of “Cypress/Playwright API Testing” or “headless testing”? Remind you, do a regular end user using the app in headless mode or API? By the way, Cypress claimed to be an E2E Testing tool.E2E Automated Test Scripts are known as brittle.
One simple code/test script/infrastructure change may cause an unknown number of test failures.
I’ve often heard principal software engineers or architects use this as an excuse for their failed E2E test automation efforts. It is low, acted as if they weren’t aware of it from the start. E2E test automation is a capability — if you lack it, acknowledge it. That’s the first step toward learning and improving.The AUT (application under test) changes constantly and frequently.
Doing “Agile”, right?A successful run of E2E Test Automation means each of every test step must pass.
As an example, my WhenWise E2E test suite (605 raw Selenium + RSpec tests as of today) is over 30,000 test steps.Many management and senior tech leads thought E2E test automation was easy.
It's a very wrong mindset. Even top software companies, such as Microsoft and Google, think it is very hard.It is very easy to give up on E2E Test Automation
“It is OK to go back to manual testing”
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)
Most software professionals will accept the above facts, right?
Now, let’s just focus on web test automation, the most common form of E2E testing.
Web technologies barely changed for over two decades in terms of E2E Testing.
Do you accept the above assessment? Many will nod their heads. I started web test automation in 2005 with an impressive success rate, if excluding non-technical factors, 100% (in doubt? Check out this showcase). Since 2011, I have been using the exact winning formula. This leads to an obvious realization, but many neglect or are unwilling to admit:
If one test automation engineer has ever done one successful web test automation, he/she should replicate the success almost every time. To make it easier to understand, E2E Test Automation is like bike-riding or car-driving.
This means most so-called “Test Automation Engineers/Experts” are fake.
“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, software legend, in a podcast (2018)
With the above understanding, I will explain the reasons of the title question in Part 2.
Related reading
