Web Test Automation Clarified. Part C: Automation
The term “automation” implies: Speed, Objectivity and Cost-effectiveness.
This Web Test Automation Clarified series:
The old slogan for enterprise software: ‘office automation’.
In Web Test Automation, “automation” means writing scripts to drive the application in a browser (e.g., Chrome) for testing purposes.
The term “automation” implies:
Speed: It should be significantly faster than manual testing.
Objectivity: No human bias — failures are clear-cut, not “maybe OK.”
Cost-effectiveness: Like factory-made products, automated tests should be much cheaper than manual testing.
However, in many software companies — even those that proudly call themselves agile — end-to-end (E2E) and regression testing are still done manually. Why wouldn’t they choose the much cheaper and more efficient automation?
At the same time, test automation is often seen as “easy” by some. When a new E2E automation initiative begins, one engineer might propose a new cool test automation tool or framework, deliver an impressive demonstration, and declare, “This tool/framework is the solution.” Yet, it often quietly fails within a few months. This cycle repeats itself time and again.
Why? Creating a new automated E2E test is easy. As my father always says, it’s only 10% of the effort — the real challenge lies in maintenance. This is quite obvious, isn’t it? I have seen a number of posts saying “E2E test automation is flaky, therefore shouldn’t do it” or “E2E test automation via UI is not possible”. Some expressed in a way not knowing this beforehand. Like many things, this is a matter of capability.
There is also a more subtle attribute of Automation: Visible.
I’m a big fan of watching automation in action, especially in factory settings, like the “How It’s Made” TV show. Automation should be visible to all team members so that any issues can be quickly identified and addressed.
However, in web test automation, ‘headless testing’ has somehow become dominant, which I never quite understood. When I watched my father run web tests, they were always visible. Headless testing doesn’t significantly speed up tests, as most of the execution time is spent on the application itself.
My father explained to me the history of headless testing, which started with PhantomJS. At that time, many so-called test automation engineers claimed doing headless web test automation, which turned out to be a big lie. Because PhantomJS “eats memory like crazy”, according to its creator. By the way, the creator of PhantomJS is a great guy, just did not expect his open-source product to be used by fakers for lies. Anyway, PhantomJS was deprecated in 2017.
During my internship, a senior test automation engineer asked me to re-implement my Selenium suite (which had 29 tests) using Playwright. I did it quickly (after learning Playwright), only to later discover that this senior engineer was struggling with a suite of just 12 tests. When I asked him to show me his test execution, some of them didn’t even run correctly. I bet that if he had made his test execution visible and run it more frequently, he could have caught those issues much sooner.
Related reading:
My father’s new book: "End-to-end Test Automation Anti-Patterns"
