Is Your Test Automation on Track? Maintenance is the key
Automated test scripts require ongoing care and maintenance. Unmaintainable or hard-to-maintain automated test scripts are in vain.

The above picture is real, and it was not taken in a third-world country (or the last century). This is the ND&W Railway in Ohio, USA! See this video for action, đ
trainfanatics.com named it âThe Worst Track Weâve Seen!â:
âPoorly maintained tracks are very dangerous as they are known to cause derailment!â
âThis railroad has seen over 50 years of neglect!â
I have seen many test automation failures (in fact, most of them), and the â1 cause: âThe team failed to maintain the automated test scriptsâ. Some engineers admitted it while others came up with some excuses.
As a matter of fact, creating automated tests is only a minor part (~20%) of the test automation effort, the other 80% consists of
Refining/stabilizing the test script
Check out this article: Working Automated Test â Good Reliable TestMaintaining the test script with change
Application changes, test data changes, server infrastructure changes, âŚ, etcAddressing the massive test execution feedback from the Continuous Testing server
Running the whole regression (UI tests) regularly (multiple times a day) in a CT server is the best way to keep the test scripts healthy and remain valid. But it is a lot of work, very valuable (to the project) but is often under-appreciated.
Consider the following facts:
software changes all the time, especially for projects that adopt agile methodologies
test scripts are changing too, which might affect other tests
test data might become obsolete
out-of-date automated test scripts are useless.
However, many engineers often ignore the above hard facts. They like to talk about tool/framework features for creating tests, but do not take test maintenance seriously. An example behaviour is to favour Record-n-Playback test automation tools which have been proven a complete failure. A former industry-leading record-n-playback tool, HP QTP => Micro Focus UFT. Now Micro Focus is the selenium-level sponsor of Selenium WebDriver.
The correct approach is to use raw Selenium WebDriver with a powerful scripting language, such as Ruby which is recommended by the classic âAgile Testingâ book. See this article âWhich Selenium WebDriver Language Binding is the Best?â
Other factors impact test maintenance.
Use a Gherkin syntax framework
This is simply silly to add a lot of unnecessary maintenance work for virtually no gains. I have never seen a Gherkin test automation attempt with more than 20 automated tests (that run regularly). The team would have thought that âexecutable scripts in Englishâ was a good idea. However, when the test suite reached a certain size (20 is Level 1 of AgileWay Continuous Testing Grading), the team realized that they could not maintain it. This story repeats in every Gherkin attempt I know of.
For more on this, please read the article: Why Gherkin (Cucumber, SpecFlow,âŚ) Always Failed with UI Test Automation?Developer-dominated test automation framework
Most developers do not know test automation and CT at all, but many of them would like to think they do. For example, in recent years, JavaScript test automation frameworks, such as Cypress and Puppeteer, have become popular. However, each of the Cypress testers I met was fake. There were some laughable stories (feel sorry for the companies) about them, such as this one: âclaimed writing hundreds of SharePoint tests, but at that time Cypress official document clear stated: Cypress does not support frames (SharePoint used frames a lot); Of course, they could not develop real tests, usually only like meetings and talked about test automationâ.
There are plenty of Cypress limitations, for more, please read this article âWhy Cypress Sucks for Real Test Automation? (Part 2: Limitations)â.
Besides technical limitations, a big problem with these JS automation frameworks is that manual testers and business analysts would find the test scripts hard to read. Different from unit/integration testing, the audience of End-to-End UI tests is the whole team. Quite often, a developer wrote a JavaScript UI test and ran it once, then forgot about it. Other team members didnât want to touch it as its syntax was bad (for E2E tests). The programmers usually donât consider maintaining UI tests as a part of their job. Before long, the test script is obsolete.
For more on why you shall avoid JavaScript in test automation, please read this article: Why JavaScript Is Not a Suitable Language for Real Web Test Automation?Test Automation Engineers donât know how to design maintainable test scripts
This knowledge, unfortunately, would not be obtained from education at universities. Engineers with real successful test automation experiences are extremely rare.
â95% of the time, 95% of test engineers will write bad GUI automation just because itâs a very difficult thing to do correctlyâ.
- this interview from Microsoft Test Guru Alan Page (2015)âTesting is harder than developing. If you want to have good testing you need to put your best people in testing.â
- Gerald Weinberg, software legend, in a podcast (2018)
The only practical and the most cost-effective way to receive dedicated training (like mine) is from a renowned expert in this field. Alternatively, read a good and practical book (like mine: Practical Web Test Automation) and do a lot of exercises. I did both.
4. Created âown test automation frameworkâ
This is silly too, I saw âexperience of creating an automation frameworkâ a lot as a selection criterion in many job ads. Come on, be realistic. Even Microsoft dumped its âCoded UI Testâ and recommended Selenium WebDriver. The chance of your engineers creating a working automation framework is extremely slim (like winning the lotto). On the opposite, these âfake test automation engineersâ often ruin the best automation framework: Selenium WebDriver.
Every so-called âin-house test automation frameworkâ (usually a badly implemented tier on top of Selenium WebDiver) I have seen was poor, that is, not reliable, slow, buggy, hard to maintain, âŚ, etc. The teams usually have separate defect tracking for their âtest automation frameworkâ. What a joke! An example of âShot Yourself in the Footâ.
Tip: Just use raw Selenium WebDriver with Ruby, ticking all the boxes: Easy to learn, Reliable, Flexible, Free (as in freedom and free beer), Feature Complete, Future Proof (browser vendor support), Easy to get help, Easy to maintain, Test Scripts in Plain Text, No Vendor Locking, Easy to version control, Easy to integrate with any CI/CT server, well supported (free), âŚ, etc.
Check out this article: AgileWay Test Automation Formula
I have rescued a handful of failed test automation attempts. When the testers started to write raw Selenium WebDriver tests after my one-day training, they often commented: âIt is much easier, quicker, and betterâ. Yes, raw Selenium WebDriver is the easiest to learn, much much easier than Cypress, Ranorex, Playwright, âŚ, etc. My daughter started writing raw Selenium tests at the age of 12; Your child could too, with proper guidance. Donât believe it? check out this article âStep by Step showing how to learn to write raw Selenium WebDriver test scripts in minutesâ, with a video.
5. Management only cares about the test count
Maintaining tests (the whole suite) is much harder than creating a new test, yet often this hard work goes unnoticed. Inexperienced managers care about the velocity and test count, to them, the contribution of the best engineer who maintains the test suite is close to 0. How sad! No wonder, under that environment, engineers do not want to touch existing tests. As a result, automated tests became obsolete.
Over the last 12 years, I have seen many failed test automation attempts (often a handful in large companies). The best test pass rate (with >20 automated UI tests) within 6 months in a CI server, from my memory, is 48%. (The ideal answer is 100% every working day, in reality, at least 100% every 2 days to cater late code check-ins). Yet, these software projects claimed âAgileâ!
In the classic âPragmatic Programmersâ book, there is one section on âBroken windows theoryâ, which is surely applicable to test automation. Often, before starting a new work, the management told me âthere would be some hands-on workâ. My reply: âI only do the hands-on workâ. I simply cannot imagine, as a test automation engineer/coach, not working on automated test scripts every day. The reason: If the test scripts were not maintained today, it would be harder tomorrow. A few days later, nobody cares about it anymore.
When the automated tests are not well maintained, just like the ND&W Railway tracks, software teams simply cannot go fast.
Related readings:
Series: Why Do Most UI Test Automation Fail? (Technical)
- Part 1: Wrong choice of automation framework
- Part 2: Wrong choice of test syntax framework