Script-less Automated End-to-End Testing is Silly, Doomed to Fail!

If the test scripts are not plain-text in a scripting language, avoid it.

Over the years, I have witnessed many attempts to test automation using Scriptless, Codeless, or No-Code approaches.

• Recorder-Generated
  Commonly found in old-generation test automation tools, such as Rational Functional Tester.

• So-called Codeless Test Automation
  Typically, allow users to drag and Drop based on Object-Identification to generate test scripts.

• AI Testing
  This is the hype of the moment, such as ChatGPT. Generating automated test scripts based on user requests.

Source: LinkedIn

Of course, all these attempts failed. The reason, it violates the fundamentals of Test Automation. I will illustrate with web test automation, the most popular type of testing. The same principles applied to Desktop and Mobile test automation.

Table of contents:
∘ 1. No matter what, it is the test scripts that drive the automation.
∘ 2. All software contains bugs, including the test automation software, of course
∘ 3. Test Creation is only ~10% effort in Test Automation; Generated test scripts make the rest task much harder.
∘ 4. Lack of flexibility in the ‘Codeless’ approach.
∘ 5. Observation: The execution speed of “Codeless Tests” is often much slower.
∘ 6. “Codeless Testing”: Not a good career choice.

1. No matter what, it is the test scripts that drive the automation.

Even for so-called “No-code” test automation solutions, there are test scripts behind, in a proprietary form which users don’t know. Accept this fact first. I will explain why this concept is important shortly.

Now think why these companies do that. Firstly, you shall know that the free, open, proven Selenium WebDriver, test scripts are in plain text format (in a scripting language). These code-based test scripts certainly worked and worked very well. Accept this fact as well.

Facebook Testing Pyramid, from this great presentation “Continuous Integration at Facebook”

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

The evil tool vendors lock the customer in by hiding the raw test scripts for higher financial gain. Of course, they can’t tell the public that. Instead, they use a sound better term, “Codeless test automation” or “No-Code test automation”.

I could tell you the perspectives of tool vendors from my own experience. I created TestWise tool, a nex-gen testing tool supports only free and open-source automation frameworks (Watir and Selenium) in 2009. I received several commercial investment/partner interests, one common request was to “make test script proprietary”. I rejected all of them.

2. All software contains bugs, including the test automation software, of course

Let me share an interesting story first. In 2013, I was tasked to review IBM’s Rational Functional Tester (RFT). This government department bought this test automation tool purely based on one Gartner report.

I was totally shocked at how bad it was. For example, it did not work on dual screens. Unbelievable? This defect is still listed on IBM’s support site.

In fact, IBM RFT is a combination of the Eclipse IDE and a record-n-playback module. Frustrated, I used $11000 RFT as a Java IDE and implemented automated tests in Selenium Java.

So, it is common that test automation tools, like any software, have bugs. But few, like IBM, did in this case, not resolving the fundamental bugs for years.

There is complex logic behind it. Most test automation attempts failed anyway, by selling a non-working and vendor-locking test automation tool at a very high price, the CIO (customer) could get it away (“another failure”). Sadly, the end-user satisfaction is not important here. Therefore, test automation tool vendors just need to spend the most effort on marketing, making record-n-playback appealing to management in demonstrations.

How did I know this? Since my test tool TestWise made its name in 2019, I received some commerical interests (partnership). A common proposal was to “increase TestWise price (about 10X)”, so that they are sell easier with various marketing practices”. By the way, I rejected them all.

As a comparison, if you are using free and open Selenium, you will receive top-level support, for free.

3. Test Creation is only ~10% effort in Test Automation; Generated test scripts make the rest task much harder.