Is Your Test Automation on Track? Maintenance is the key

Automated test scripts require ongoing care and maintenance. Unmaintainable or hard-to-maintain automated test scripts are in vain.

The image is taken from this youtube video

The above picture is real, and it was not taken in a third-world country (or the last century). This is the ND&W Railway in Ohio, USA! See this video for action, 😅

trainfanatics.com named it “The Worst Track We’ve Seen!”:

“Poorly maintained tracks are very dangerous as they are known to cause derailment!”
“This railroad has seen over 50 years of neglect!”

I have seen many test automation failures (in fact, most of them), and the №1 cause: “The team failed to maintain the automated test scripts”. Some engineers admitted it while others came up with some excuses.

As a matter of fact, creating automated tests is only a minor part (~20%) of the test automation effort, the other 80% consists of

• Refining/stabilizing the test script
  Check out this article: Working Automated Test ≠ Good Reliable Test

• Maintaining the test script with change
  Application changes, test data changes, server infrastructure changes, …, etc

• Addressing the massive test execution feedback from the Continuous Testing server
  Running the whole regression (UI tests) regularly (multiple times a day) in a CT server is the best way to keep the test scripts healthy and remain valid. But it is a lot of work, very valuable (to the project) but is often under-appreciated.

Consider the following facts:

• software changes all the time, especially for projects that adopt agile methodologies

• test scripts are changing too, which might affect other tests

• test data might become obsolete

• out-of-date automated test scripts are useless.

However, many engineers often ignore the above hard facts. They like to talk about tool/framework features for creating tests, but do not take test maintenance seriously. An example behaviour is to favour Record-n-Playback test automation tools which have been proven a complete failure. A former industry-leading record-n-playback tool, HP QTP => Micro Focus UFT. Now Micro Focus is the selenium-level sponsor of Selenium WebDriver.

The correct approach is to use raw Selenium WebDriver with a powerful scripting language, such as Ruby which is recommended by the classic ‘Agile Testing’ book. See this article “Which Selenium WebDriver Language Binding is the Best?”

Other factors impact test maintenance.

1. Use a Gherkin syntax framework
   This is simply silly to add a lot of unnecessary maintenance work for virtually no gains. I have never seen a Gherkin test automation attempt with more than 20 automated tests (that run regularly). The team would have thought that “executable scripts in English” was a good idea. However, when the test suite reached a certain size (20 is Level 1 of AgileWay Continuous Testing Grading), the team realized that they could not maintain it. This story repeats in every Gherkin attempt I know of.
   
   For more on this, please read the article: Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
   

2. Developer-dominated test automation framework
   Most developers do not know test automation and CT at all, but many of them would like to think they do. For example, in recent years, JavaScript test automation frameworks, such as Cypress and Puppeteer, have become popular. However, each of the Cypress testers I met was fake. There were some laughable stories (feel sorry for the companies) about them, such as this one: “claimed writing hundreds of SharePoint tests, but at that time Cypress official document clear stated: Cypress does not support frames (SharePoint used frames a lot); Of course, they could not develop real tests, usually only like meetings and talked about test automation”.
   
   There are plenty of Cypress limitations, for more, please read this article “Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)”.
   
   Besides technical limitations, a big problem with these JS automation frameworks is that manual testers and business analysts would find the test scripts hard to read. Different from unit/integration testing, the audience of End-to-End UI tests is the whole team. Quite often, a developer wrote a JavaScript UI test and ran it once, then forgot about it. Other team members didn’t want to touch it as its syntax was bad (for E2E tests). The programmers usually don’t consider maintaining UI tests as a part of their job. Before long, the test script is obsolete.
   
   For more on why you shall avoid JavaScript in test automation, please read this article: Why JavaScript Is Not a Suitable Language for Real Web Test Automation?

3. Test Automation Engineers don’t know how to design maintainable test scripts
   This knowledge, unfortunately, would not be obtained from education at universities. Engineers with real successful test automation experiences are extremely rare.

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, software legend, in a podcast (2018)

The only practical and the most cost-effective way to receive dedicated training (like mine) is from a renowned expert in this field. Alternatively, read a good and practical book (like mine: Practical Web Test Automation) and do a lot of exercises. I did both.

4. Created ‘own test automation framework’
This is silly too, I saw “experience of creating an automation framework” a lot as a selection criterion in many job ads. Come on, be realistic. Even Microsoft dumped its ‘Coded UI Test’ and recommended Selenium WebDriver. The chance of your engineers creating a working automation framework is extremely slim (like winning the lotto). On the opposite, these ‘fake test automation engineers” often ruin the best automation framework: Selenium WebDriver.

Every so-called ‘in-house test automation framework’ (usually a badly implemented tier on top of Selenium WebDiver) I have seen was poor, that is, not reliable, slow, buggy, hard to maintain, …, etc. The teams usually have separate defect tracking for their ‘test automation framework’. What a joke! An example of ‘Shot Yourself in the Foot”.

Tip: Just use raw Selenium WebDriver with Ruby, ticking all the boxes: Easy to learn, Reliable, Flexible, Free (as in freedom and free beer), Feature Complete, Future Proof (browser vendor support), Easy to get help, Easy to maintain, Test Scripts in Plain Text, No Vendor Locking, Easy to version control, Easy to integrate with any CI/CT server, well supported (free), …, etc.
Check out this article: AgileWay Test Automation Formula

I have rescued a handful of failed test automation attempts. When the testers started to write raw Selenium WebDriver tests after my one-day training, they often commented: “It is much easier, quicker, and better”. Yes, raw Selenium WebDriver is the easiest to learn, much much easier than Cypress, Ranorex, Playwright, …, etc. My daughter started writing raw Selenium tests at the age of 12; Your child could too, with proper guidance. Don’t believe it? check out this article “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”, with a video.

5. Management only cares about the test count
Maintaining tests (the whole suite) is much harder than creating a new test, yet often this hard work goes unnoticed. Inexperienced managers care about the velocity and test count, to them, the contribution of the best engineer who maintains the test suite is close to 0. How sad! No wonder, under that environment, engineers do not want to touch existing tests. As a result, automated tests became obsolete.

Over the last 12 years, I have seen many failed test automation attempts (often a handful in large companies). The best test pass rate (with >20 automated UI tests) within 6 months in a CI server, from my memory, is 48%. (The ideal answer is 100% every working day, in reality, at least 100% every 2 days to cater late code check-ins). Yet, these software projects claimed ‘Agile’!

In the classic ‘Pragmatic Programmers” book, there is one section on “Broken windows theory”, which is surely applicable to test automation. Often, before starting a new work, the management told me “there would be some hands-on work”. My reply: “I only do the hands-on work”. I simply cannot imagine, as a test automation engineer/coach, not working on automated test scripts every day. The reason: If the test scripts were not maintained today, it would be harder tomorrow. A few days later, nobody cares about it anymore.

When the automated tests are not well maintained, just like the ND&W Railway tracks, software teams simply cannot go fast.

---

Related readings:

• Maintainable Automated Test Design

• False ‘Selenium WebDriver Cons”​

• Series: Why Do Most UI Test Automation Fail? (Technical)
  - Part 1: Wrong choice of automation framework
  - Part 2: Wrong choice of test syntax framework