Irrational and False Excuses for Web Test Automation Failures
Real reason: Incompetence. Just admit ‘lack of the skills’, change the mindset, and seek professional help if necessary.
FACT: Web Test Automation attempts failed in most software projects.
(Disagree? do an instant and objective assessment)
Over the years, I have heard many excuses, such as:
“It worked well on my last project”
“Tests failed because the application changed”
“The test scripts are fine, only test data is no longer valid”
“Our testers do not have the capability”
“The automation framework selected by the previous test architect was bad”
“Automated end-2-end tests are known as fragile”
…
To some people, these excuses might sound reasonable. However, a real test automation engineer will conclude that those excuses are plainly wrong, very wrong. ‘Irrational’ might sound like a strong word, but you may agree after reading my explanation, as these excuses are so obviously wrong and indefensible.
There is really only one reason: Incompetence.
In this article, I will list 14 common false excuses. But first, I will list some facts that might help you better understand the reasonings.
FACT: Automated UI Testing
Brittle, prone to application changes (compared to unit/API tests)
Long execution time (compared to unit/API tests)
Testing for all web applications, technically speaking, is the same. If one tester can do test automation successfully at one job, he shall be able to replicate the success on other jobs, with virtually no changes.
FACT: Selenium WebDriver
the only automation framework that is W3C compliant
free (in both freedom and cost)
the only automation framework supported by all major browser vendors
popular, open-source and well maintained
used and recommended by top IT companies such as Facebook, Microsoft and Google.
Now, let me expose those fake excuses or lies.
1. “It worked well on my last project”
This is a “Dog ate my homework” level lie used by naughty children. My question is “How about this project?” Show me something today, then tomorrow, and onwards. A real test automation engineer can implement a few key tests on day 1. The reason is simple: web test automation is pretty much the same for all projects.
Given the fact that web projects are 100% based on W3C standards now (previously not the case for Flash and Silverlight, which both died), it is silly that a fake professional pretends he/she “can do test automation” and IT management let them pretend for more than a week.
I take test automation consulting by days. Companies can engage my service for 1 day, and I delivered value on that day. For my recent consultation projects, if possible, I would ask a manual tester (with domain knowledge) to work with me right after greetings. If other people (engineers, managers and testers) are interested, I suggested connecting the computer to a projector: I develop real automated tests (in raw Selenium with RSpec) live against a web application which is new to me.
People attended the session were usualy impressed by my efficiency. What impressed them most is that the manual tester (and some audience) actually understood the test scripts (in Ruby) and had confidence to do modifications.
2. “Test execution failed because the application changed”
What?! Is “application changes” a surprise to you? Come on, this is normal. How could a software/test engineer even say that? If the application never changes, there is no need for regression testing.
3. “The test scripts are fine, only test data is no longer valid”
Wrong, test data is a part of test scripts.
4. “Our testers do not have the capability”
This is often used by managers and tech leads (such as Test architects, Chief/Principal software engineers). Yes, it was a fact. However, have you done anything to improve the situation?
Moreover, do the managers/tech leads know what are skills/capabilities required? I doubt it.
5. “Test Automation is expensive, a XXX tool/runtime license costs $$$$”
Why don’t use the 100% free and future-proof Selenium WebDriver? After all, it is the only one that is W3C compliant. Facebook and Microsoft recommended it.
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ”
- Katie Coons, software engineer at Facebook, “Continuous Integration at Facebook” presentation at F8 2015.
6. “Still working on some issues of our own framework, …”
Crazy. First of all, very very few engineers in this world can create a new working framework. This so-called “own framework” is normally just an unnecessary layer on top of Selenium. From my experience, each of these ‘new frameworks’ failed for a silly and simple reason: this badly-designed unnecessary layer was buggy.
Why not just use raw Selenium WebDriver like Facebook, Google and Microsoft (they recommend WebDriver, not another fancy name as a ‘new framework’)? It is fair to say that the engineers in your company are not as good as those at these IT giants.
For more, please read my other article: “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”.
7. “Selenium WebDriver has a steep learning curve”
Totally wrong! Selenium WebDriver is the easiest to learn test automation framework. Most people I mentored could work on real tests after one-day training. My 12-year-old daughter learned to write the raw Selenium WebDriver tests, so could you and your teenage child under proper mentoring.
Common feedback from the above article: “Yes, it does look easy; But it is Ruby, we want to use Selenium with Java/JavaScript”. Oh well, Ruby is one of five Selenium binding languages. Therefore, my statement of “Selenium is the easiest to learn” is correct. Being fixated on Java/JavaScript is a human problem, so using this excuse for Selenium is wrong. If tech leads are so sure of using Java/JS, make it happen. Otherwise, they are incompetent.
Note: I am probably more objective than most of the test engineers, as my Selenium Recipes Book series covers all of Selenium’s five official languages.
I could develop quite good Selenium tests in Java/JS/C#/Python, but I don’t have confidence in mentoring others to achieve the productivity required for successful test automation.
Above all, scripting automated tests in Ruby is efficient, easy, creative, and fun!
8. “The reason we have to use Java/JS/C# is that we have the skills to maintain”.
Untrue. Most people will agree that, compared to ongoing maintenance, creating automated tests is a lot easier. Now check the reality. After several months, the team might have been struggling with test creation by using the ‘preferred’ language. There is no point in talking about reaching the finishing line if already failed at the starting line.
The sensible way is to use Ruby, a recommended language by the Agile Testing book. Maybe with the help of a test automation coach, the team can create a handful of key tests every day.
The real reason is that a tech lead only knows one language and fears anything unfamiliar. I have met quite a number of these kinds of tech-leads. Believe it or not, I think some of them actually wanted the test automation to fail. They prefer fake test automation, as no changes will be required. If test automation succeeds, they won’t be comfortable adjusting to a new regime.
Once at a large finance company, a manager invited me do a proof of concept before my contract ended in a week. The two tech leads knew my work (in Ruby), but insisted on writing test scripts in JavaScript. By the way, the company failed badly with Java (with a Gherkin syntax, see my article: “Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?”). The failure was so big that I heard people joked about their embarrassment at another company.
I did a few tests in JavaScript. They were so surprised on how quickly I accomplished the task, well exceeding their expectations. I commented: “No, the efficiency was no good, with this productivity, it will be struggling with a suite of 50 tests” (Level 2 of AgileWay Continuous Testing Grading)
I created the tests in Ruby first and stabilized them, then converted them to JavaScript. In other words, most of the work was done in Ruby. Anyway, these two tech-leads still wanted tests in JavaScript. So, I declined the contract renewal.
A few months later, I saw a Job Ad from this company for a Senior Test Automation Engineer. One Criteria was interesting: “Test Automation experience in C#, Java, JavaScript or Python”, the only approved working (at this company) and the best test scripting language: Ruby was excluded.
9. “Selenium WebDriver execution is not stable”, “Selenium could not test AJAX”, …
Totally untrue. Selenium WebDriver is the most stable web automation framework. Remember, web test automation depends on browsers, and all major browser vendors only support the WebDriver standard. You may use ‘unstable’ to describe any other framework such as UTP and Cypress, but not Selenium WebDriver.
Testing AJAX with raw Selenium Webdriver, if done properly, is far easier than the others. See this article: “Test AJAX Properly and Efficiently with Selenium WebDriver”.
10. “The automation framework selected by the previous test architect was bad”
A typical office politics. Though it might be true, a test architect is only allowed to use this excuse for one day. Web test automation skills and practices are fully transferable. A real test automation engineer, like me and my teenage daughter, can implement a few key tests on the first day.
11. “Automated end-2-end tests are known as fragile”
Yes, that’s why you are hired to implement it. It is still professional for engineers to reject test automation because they are unable to accomplish the task. However, if someone did take the task (and getting paid) and failed it, using this excuse is quite low.
UI Test Automation can be done, see my article: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”. From my experience, after having done one successful web test automation properly, I can always replicate the success in other projects.
12. “Our testers have difficulty using Java/JavaScript, the language used in test scripts.”
Why not change to Ruby? Ruby is the best scripting language recommended by the classic ‘Agile Testing’ book. The real reason: Test Lead does not know Ruby, and lacks a learning altitude, which is more concerning.
13. “The automated UI test scripts are hard to read, difficult to maintain by nature”
Not necessarily. Writing tests in RSpec (Ruby) will greatly enhance the readability of the test scripts. I could maintain 617 and 500 selenium tests for my two apps ClinicWise and WhenWise, respectively. Of course, efforts are required but maintenance is surely possible. Otherwise, why is your company trying to do it anyway?
For more, check out my article: “Maintainable Automated Test Design” or book “Practical Web Test Automation”.
14. “The DevOps team is no good, unable to run our tests reliably in Jenkins”
I will write another article on silly and false excuses for CI/CD/CT failures. Here I would like to focus on test automation. My question is: “How reliable are your tests?” If your team members run the same test script at different times, are you confident of getting 10 successful test executions?
In my opinion, if the average test pass rate is <95%, don’t bother running your test suite in the CI/CD server. Stabilize your tests first!
Read this article: “Working Automated Test ≠ Good Reliable Test”.
Below are recent test execution stats for WhenWise, one of my own web apps. With ~500 Selenium tests, 99.5% pass rate was in the last month and 97.9% over the last 3 years.
