A Long List of Failed Test Automation Hypes
They were so plainly wrong. How could I predict their failures early? Easy!
A repost of my past article on Medium
Since I have worked in software development and test automation for over two decades, I have seen many failed hypes in test automation, such as:
Record-n-Playback Test Automation Tools
Such as HP QTP (now Micro Focus UFT) and IBM Rational Functional Tester. These so-called industry-leading test automation tools, usually quite expensive, are less and less relevant now. Microsoft even dumped its Coded UI Test to recommend Selenium (for Web) and Appium (for desktop and mobile)
Cross-Browser Testing using SauceLabs or BrowserStack
Nowadays, people care little about CBT (focus on Chrome only). If you are interested, check out my article, Cross Browser Testing Clarified.Headless Testing with PhantomJS
An embarrassment for many, I correctly predicted its failure (written in my book). For more, check out Headless Browser Testing Clarified.Protractor (JS)
the testing framework based on Selenium from the super hot Angular.js team, deprecated in 2021. Check out Protractor, another Automation Framework I rightly avoided, is being DeprecatedTest Cafe, Puppeteer and Cypress
Those three have been or are being replaced by Playwright. Check out Why Cypress Sucks for Real Test Automation?Codeless Test Automation
such as Katalon, check out my article “Codeless Test Automation is Irrational” and “Katalon Codeless Test Automation vs Selenium WebDriver by Example”
The above are the failed automation hypes. If we extend to include test syntax:
FitNesse
many people don’t hear it now, back in ~2008, it was a quite big hype. There are several books on it, like this one.Gherkin Styled BDD (Behaven Driven Development)
while there are still quite many software projects still trying Gherkin frameworks (such as SpecFlow and Cucumber), its halo has faded. Check out “Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?”
I predicted the failures of all the above (and others), with proven records: my books (since 2009), my tools (since 2007), and my articles. Since 2011, I have been implementing successful end-to-end (via UI) test automation exactly the same way: raw Selenium WebDriver + RSpec. Prior to Selenium WebDriver (released in 2011), I used Watir + RSpec (for executing tests in IE).
Table of Contents:
∘ 1. Record-n-Playback Test Automation Tools
∘ 2. Cross-Browser Testing Tools
∘ 3. Headless Testing with PhantomJS
∘ 4. Protractor (JS)
∘ 5. Test Cafe, Puppeteer and Cypress
∘ Making a wrong framework decision in Web Test Automation is NOT excusable.
1. Record-n-Playback Test Automation Tools
Reason for failure: Hard to maintain Test Scripts.
In the early 2000s, Automated Testing was almost equal to “Record-n-Playback” (here, Automated Testing means end-to-end UI test automation. Unit testing was done by programmers, JUnit only appeared in 2002). The industry-leading test automation was Quick Test Pro (HP acquired it from Mercury Interactive in 2006). It was very expensive (~US$20,000 a license), and it was only good for demonstration, at least for every case I witnessed.
The recorder-generated proprietary test scripts are terrible: difficult to read, and almost impossible to maintain when the test suite reaches a certain size.
I used it once at a government project, which purchased a license but no one was using it. After playing it for a while, I realized it was just a fancy & expensive tool. I used it to fill out my timesheet.
I also reviewed IBM’s RFT once in 2013 at a state government. By the way, the government purchased this tool purely based on the recommendation of the Gartner Report. Of course, it was a big mistake. IBM’s RFT failed in every aspect compared to the free Watir and Selenium WebDriver.
2. Cross-Browser Testing Tools
Reason for failure: No signs of daily execution one Chrome, little need for worrying about others.
In 2013, a test lead told me the company (500+ IT staff) spent quite some money on SauceLab’s services for Cross-browser Testing. I said: “I bet it provides little value”. He checked and verified I was correct.
He asked: “How did you know?”
I replied, “There are no signs of any Automated End-to-End Testing (in CI/CD) server. In other words, test automation is not working well in one dominant browser: Chrome, why worry about IE and Firefox?”
Also, there was clearly a trend that Chrome (or Chromium) would extend its dominance in the browser market. IE was dying, and Edge Classic sucked and is now based on Chromium.
By the way, Selenium WebDriver is best for Cross-Browser Testing, the reason is simple: All major browser vendors only support WebDriver, a W3C standard.
3. Headless Testing with PhantomJS
Reason for failure: Not a real browser, how could handle ever-growing complex web apps. Furthermore, more often than not, we want to see test execution.
Between 2010 and 2015, headless browser testing with PhantomJS was widely hyped among test architects and engineers. I couldn't understand how people failed to recognize its obvious flaws.
If a test script fails on a simulated browser (not setting the web page), how do you debug it?
PhantomJS was deprecated in 2017, and suddenly, headless browser testing disappeared (until a few years later, Google Chrome implemented headless mode in v58). For more, check out
4. Protractor (JS)
Reason for failure: Making intuitive Selenium syntax complex, injecting Programmer’s thoughts into test automation.
Between 2013 and 2018, AngularJS dominated as the go-to web development framework. On paper, Protractor appeared to be the automation framework programmers had long been waiting for.
Based on the proven W3C standard-based Selenium WebDriver, with so-called “improvements”
By the talents behind Angular.js (I believe many contributors worked at Google)
However, it failed (deprecated in 2021). It made good raw Selenium syntax bad based on wrong knowledge of ‘improving’ it. When I saw a sample Protractor test for the first time, I felt like seeing a clean room that was badly decorated.
Generally speaking, a senior software development engineer needs significant upgrades in knowledge to reach a real test automation engineer. Sadly, that’s not the general reception of most people.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)“We see programmers managed to code user stories most of time. Have you meet one test automation engineer who can develop/maintain the automated end-to-end (via UI) tests for all user stories as regression testing (run daily)? I bet you haven’t” . — Zhimin Zhan
Let me put it simply. If a senior software engineer’s salary is $100,000. It will be very lucky for the CIO to be able to lure one real test automation engineer at a $1,000,000 salary. Hard to believe? check out Wired (the most respectable IT magazine) article: “The Software Revolution Behind LinkedIn’s Gushing Profits”.
Protractor is a framework not needed, raw Selenium WebDriver is much better.
For more, check out the article, Protractor, another Automation Framework I rightly avoided, is being Deprecated.
5. Test Cafe, Puppeteer, and Cypress
Reason for failure: a web test automation framework must support all web technologies defined by W3C. But these don’t.
I know Cypress’s UI (as a tool) might appear appealing and good for demonstration. That’s all.
At one project, I was shocked to find out that Cypress was chosen for the test automation framework. The application uses frames quite intensively. For God’s sake, Cypress did not support frames (not with the workaround then), clearly stated on its website, along with several other serious limitations. The tech lead (a JS developer) chose Cypress for two reasons: JavaScript and looking good in the demo video. This was insane. This whole decision was so fxxked-up, no one was doing real work in test automation, and no one did care, back to full manual testing.
When the testers (and two developers) saw my implementation of several key tests (and running in a CT server) on the first day, they found that raw Selenium (with RSpec) is much better than Cypress, in every aspect.
Making a wrong framework decision in Web Test Automation is NOT excusable.
Web Test Automation means Objective and Quick Results. Why? Testing means objective; Core Web technologies have not changed over two decades.
“If a test automation engineer has ever done one successful test automation, the knowledge, skills and practices can be directly applied to future ones. It is like “Bike Riding”, once you really master it, you prove it instantly. “ — Zhimin Zhan
It shall be quite obvious after being pointed out (see the above), isn’t it?
In a logical world, it is impossible to make a wrong decision in selecting the correct web automation framework or tool, given a very static and easy-to-understand target (i.e. web), yet it happens most of the time. A real test automation engineer can develop one automated test for the main business scenario on the spot. I have accomplished this every single time since 2011, using raw Selenium WebDriver with RSpec.
Some might argue, “We prefer JavaScript or Java”. That’s OK, but remember, this is testing (verifying), nothing to do with the coding language. The goal is to develop and maintain a comprehensive suite (~200) of automated end-to-end (via UI) tests, and run them reliably on a daily basis. The goal is not to make your team members feel comfortable with a scripting language.
Most people would agree that test maintenance (for automated end-to-end tests) is hard. This is true, the effort is almost exponential growth if the test scripts are not well-designed and the tool is not highly efficient.
“Test Creation in end-to-end test automation is a minor effort, compared to ongoing maintenance” — Zhimin Zhan
Why could I accurately predict the failures of test automation frameworks/tools? I evaluate from a maintenance and daily regression testing perspective. If solely from the test creation perspective, numerous people have done that with QTP and alike tools over three decades ago, and this approach has been proved wrong!
In terms of specific how, check out my articles:
Related reading: