Why Do Most UI Test Automation Fail? (Part 1: Wrong Automation Framework)

The management chose the wrong framework. They should select Selenium WebDriver for web apps, and Appium for Mobile and Desktop apps

Nov 14, 2022

FACT: not many software engineers (in Test) received proper education or training in test automation, which is unfortunate. Considering this: 30–40% of staff in a software team are usually testers. Programmers and business analysts spend a lot of time performing functional testing as well. However, to my knowledge, there are few dedicated Functional Testing courses offered at universities. Therefore, it is not surprising that many wrong decisions on test automation are made due to a lack of knowledge.

In this “Why Do Most UI Test Automation Fail?” series, I will share the common mistakes that contribute to test automation failures in software projects.

Wrong choice of automation framework
Wrong choice of test syntax framework
Wrong choice of test scripting language
Wrong choice of test automation tool (coming soon)
Don’t know how to design easy-to-maintain test scripts (coming soon)
Lack of efficiency in creating, debugging, and maintaining automated tests (coming soon)
Run automated UI tests in CI servers such as Jenkins (coming soon)

The above are just technical mistakes. There are other ones that are made due to human reasons (I plan to write some articles on it as well). Anyway, here is the first mistake of the series.

Wrong choice of automation framework

This is the most frustrating area in my 14-year consulting career in test automation. Managers or architects often make wrong decisions when selecting the automation framework, while the good choice is obvious.

There is no more convincing than a software giant depreciating its own testing framework/tool and recommending an alternative, right? Here is one: Microsoft deprecated Coded UI Test and recommend Selenium WebDriver in 2018.

Let’s not forget, besides Windows and Office Suite, Microsoft is a browser (e.g. IE and Edge) and tool (e.g. Visual Studio) vendor. If Microsoft stated that “Our (test automation) software was not good and will abolish it, we recommend X”, and X is free and a standard, it should be a no-brainer, right? However, the reality is far from common sense.

Here are three forms of how Software projects choose the wrong automation framework:

1. Record-n-playback tools are bad

Record-n-playback tools have long existed in software testing, with bad reputations. At every software testing conference I attended, the general response to those record/playback tools was ‘NO’. However, new record/playback test automation frameworks/tools come and go in slightly different forms, such as RFT, QTP/UFT, Ranorex, Sahi, and TestComplete. To me, the recording-based approach is fundamentally wrong as recorded test scripts are hard to maintain. And we all know the biggest effort of test automation is in test maintenance,

“Record/Playback scripts are notoriously costly from a maintenance perspective.” — ‘Agile Testing’ book on page 315

From my knowledge, every project that has used one of those record-n-playback tools failed test automation badly.

Some might argue that these testing tools do look impressive. Wrong! We shall always select the automation framework first, not the tool(s). It is the framework that decides the test script and its testing capability while a good tool enhances the team’s productivity. If the framework underneath a tool is not stable or lacks features, it doesn’t matter how impressive the tool may seem to be, it simply will not do the job.

Related reading: Web Automation Framework Trend ⇒ Selenium WebDriver, which lists many commercial testing tools vendors who are actually the sponsors of Selenium.

Q & A:

a) Compared to using recorders, hand-crafting automated test scripts is much slower, isn’t it?

Not really. If you take the refinement into consideration, hand-crafting automated tests are much more efficient. Seeing is believing, please read the video/animation in the following two articles to see how efficiently a teenager (my daughter) can develop raw Selenium tests in TestWise.

Also, “Why I Created a Selenium Recorder But Rarely Use it myself?”

2. ‘self-created framework’ on top of Selenium WebDriver

Have you seen “building test automation frameworks from scratch” in Job Ads for Test Automation Engineers? I saw a lot in Australia. There are many “Automation Framework design” tutorials on YouTube. The fact is that engineers (such as Simon Stewart, the creator of Selenium WebDriver) who can create a high-quality automation framework from scratch are extremely rare. These engineers are most likely unavailable on the job market.

I have worked as a consultant on a few projects with the so-called ‘own test automation framework’. How bad can it be? I saw that two projects had to separate defect tracks for their own test automation framework! Those ‘frameworks’ added a badly implemented layer of abstraction on top of the Selenium WebDriver. This unnecessary abstraction layer was based on the misunderstandings of test automation which were often inspired by some record-n-playback tools.

Not surprisingly, test automation with ‘own framework’ failed not only at the technical aspect but also with subtle human factors. The embarrassment will prevent the management (who made the wrong call) from adopting the correct test automation framework: raw Selenium WebDriver.

In my opinion, raw Selenium WebDriver is the easiest-to-learn automation framework because its syntax follows an intuitive pattern: Locate a control then Drive it. For example:

driver.find_element(:link_text, “Login”).click driver.find_element(:id, “username”).send_keys(“agileway”) driver.find_element(:name,”password”).clear driver.find_element(:xpath,”//input[@value=‘Sign In’]”).click puts puts driver.find_element(:tag, “body”).text

The equivalent of the above in Java:

driver.findElement(By.linkText(“Login”)).click(); driver.findElement(By.id(“username”)).send_keys(“agileway”); driver.findElement(By.name(“password”)).clear(); driver.findElement(By.xpath(“//input[@value=‘Sign In’]”)).click(); System.out.println(driver.findElement(By.tagName(“body”)).getText());

3. JavaScript automation frameworks selected by the developers in the team

JavaScript automation frameworks, such as Cypress and Puppeteer, gained popularity in recent years. These frameworks are not good. Please read my other articles:

Besides technical limitations (see the above), the developer’s mindset is not fit for test automation.

“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)

Test automation is objective and should be practically used daily for the team’s development. You can easily assess a software engineer's testing capability. One simple way is to measure against AgileWay Continuous Testing Grading. From my experience, every Software Engineer in Test I have met was either Level 0 (mostly) or Level 1. The average level of a real agile team is Level 3, which, as Alan Page of Microsoft pointed out, is “very difficult”, even for Microsoft.
(Therefore, the quickest and the most cost-effective way is to engage a real Test Automation coach to lift the team’s capability)

Next time, if a programmer suggests another JavaScript automation framework, please be aware. The reason he/she made this suggestion might be that he/she was comfortable with JavaScript and did a few hello-world-level tests.

Q & A:

b) Raw Selenium automated tests are hard to maintain, that’s why I created “a framework on top of it”.

Did ‘your framework’ actually work? Try to assess it against the simple and objective AgileWay Continuous Testing Grading. The score of most software projects is either Level 0 or 1. Comparably, I achieved Level 4 for my own web apps: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”, by using raw Selenium WebDriver.

Using the well-known Page-Object pattern (a Selenium Best Practice) can greatly enhance the design and maintainability of the test scripts. See my article: “Maintainable Automated Test Design”.

c) Our testers found the raw selenium tests are hard to master, so we created ‘a framework’ on top of it.

I disagree. A combination of good test design, a good syntax framework in a good scripting language, and a good testing tool will dramatically bring down the threshold of learning test automation. Most of the manual testers who attended my one-day training started to develop real tests at work the next day. My daughter started writing raw Selenium tests when she was 12 years old, and so could your child. (see some of my other articles on her work).

Again, seeing is believing, check out these test scripts (on Github). I am sure that you will agree that manual testers shall have little problems understanding (or even modifying) the tests.

A Story

After attending my one-day Selenium WebDriver in-house training at a government department, a tester told me this story.

The project had one test automation attempt that failed within a week. The reason was the expensive testing tool (chosen by the test architect) did not support testing frames that were used in the web app. As we know, failing to drive one step means being unable to test many scenarios. They contacted the support but got no help but frustration.

Noises

Between 2011–2017, Selenium WebDriver unquestionably dominated the test automation for web apps, and there were no real competitors (as it should be because WebDriver is a W3C standard). While most projects failed (due to lack of knowledge, experience, and no access to real test automation coaching), they should be at least on the right track. The solution is simple: get help from a real test automation coach.

So many projects are not willing to take on Selenium WebDriver, a common excuse is “Selenium has a steep learning curve’. This is totally wrong! Please read my two other articles:

There is no shame in admitting “I don’t know test automation”. Again, lacking this knowledge is the failure of education and training in Computer Science.

However, since 2018, I have started to hear many false statements about Selenium, such as:

“Selenium WebDriver test execution is not reliable”
“Selenium WebDriver unable to test AJAX or Single Web-Page app”
“Selenium WebDriver is slow”

The above statements are so wrong! Please read my other article: “Irrational and False Excuses for Web Test Automation Failures”.

I have been working in test automation and continuous testing since 2005. The successful test automation projects shared two (and the only two) test automation frameworks: Watir (now became a syntax tier on top of Selenium) and Selenium WebDriver. Here is my recent post on test automation of my own app: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”.

Lack of test automation knowledge and (a single) successful experience is OK and normal, people can learn (if under proper mentoring) and will learn very quickly. But making false statements is unprofessional (to say it at least), and it will lead to a wrong path. For example, JavaScript-based test frameworks such as Cypress and TestCase were hyped by JS programmers who claimed it ‘better than Selenium WebDriver”. From my knowledge, every project that has been using a JS-based framework failed test automation and often with big embarrassment.

Advice:

The choice is super easy: go for the WebDriver that is W3C compliant, and you don’t need to worry about:

feature completeness
browser support (all major browser vendors only support WebDriver)
execution stability
performance
documentation and technical support (plenty of resources online)
cost (free)
vendor-locking
access to training
seeking independent help from test automaton mentors
integration with CT servers
…

Again, here are Microsoft’s recommendations:

Selenium WebDriver for testing web apps
Appium + WinAppDriver for testing (Windows) Desktop apps
Appium + iOS Driver/Android Driver for testing mobile apps

AgileWay’s Test Automation & Continuous Testing Blog

Discussion about this post