Codeless Test Automation is Irrational
Avoid this doomed-to-fail hype.
This article is one of the “IT Terminology Clarified” series.
Update (2022–08–27): new article “Katalon Codeless Test Automation vs Selenium WebDriver by Example” is available.
Since 2018, I have seen a few so-called “Test Automation Predictions” listed “Codeless Test Automation” as the trend forward. Oh well, it hasn’t happened. Frankly, I don’t think it will ever happen, because “Codeless Test Automation” is irrational. The people who promote this do not understand test automation at all.
Table of Contents:
· Clarify ‘Codeless Test Automation’
· “Codeless Test Automation” is vague
· Why do some people hype “Codeless Test Automation”?
· Why “Codeless Test Automation” won’t work?
∘ Test Maintenance
∘ Test Execution with Quick Feedback
∘ Codeless does not address what it takes to succeed in automated testing
· Coding provides the flexibility that Test Automation needs
· Similar hypes failed before
· “Codeless Test Automation” is not FUN!
· Verify ‘Codeless Test Automation’ toolClarify ‘Codeless Test Automation’
“Test Automation” means driving the application with automated test scripts (as opposed to humans in manual testing). The special keyword is “Codeless”, in other words, scripts are not programming code in ‘codeless test automation’, or at least ‘test engineers’ do not need to understand code.
Let me clarify further, automated test execution is based on test scripts. whether we see the actual script or not, regardless of how it was created, there are scripts (somewhere) controlling the execution.
According to ‘Codeless Test Automation’ fans, “Script ≠ Code”. This is wrong and has been proven wrong repeatedly. I have rescued many failed test automation using old-fashioned ‘Codeless Automation Tools” such as QTP, IBM Rational Functional Tester, Ranorex, and Tosca. The testers created the scripts using recorders or performing drag-n-drops in the tool, and most of these testers did not have programming skills. As a result, these automation attempts all failed badly. How did I rescue them? By rewriting the test scripts in raw Selenium WebDriver in Ruby. Yes, Code!
Selenium WebDriver has dominated web test automation over the last decade. Some favour other new automation frameworks, such as Protractor, TestCafe, Cypress, Playwright, …, etc. These test scripts are also code (JavaScript). This at least means, the trend of “Codeless ➝ Code” has been correct so far.
Some people believe that the new Codeless with AI will be a game-changer. I have been working in test automation for over 16 years, but haven’t yet seen even one working AI-Codeless test automation.
I just did a google search of ‘Codeless Test Automation’, the 4 Ads were all from commercial tool vendors. This got to be the highest number of Google Ads (showing on the top) from my memory. Be aware.
“Codeless Test Automation” is vague
People who talk about “Codeless Test Automation” all have different understandings. In other words, the term is vague. For example, codeless test automation can be:
a tool that records user operations
a drag-n-drop tool to build tests
a tool to analyse the target web page(s) and generate automation using AI
Either way, the users don’t need to worry about the text-based script, i.e. code.
The process of Functional Test Automation is simple: just execute the scripts step by step. For example, a simple user login test:
(Prepare test data)
Visit Login Page
Enter username
Enter password
Click "Login"
Verify the “Login successful” messageI am struggling to see how AI or machine learning can assist here. For such a simple test case, wouldn’t be easier to just write Selenium scripts as below? (raw Selenium WebDriver using Maintainable Automated Test Design)
visit("https://whenwise.agileway.net/sign-in")
login_page = LoginPage.new(driver)
login_page.enter_email("drving@biz.com")
login_page.enter_password("CoolApp01")
login_page.click_sign_in
expect(page_text).to include("Login successful")For more on AI Test Automation, stay tuned for an upcoming article.
Why do some people hype “Codeless Test Automation”?
The assessment of whether a test automation attempt is successful is objective, as in my “Definition of End-to-End Test Automation Success”, many fake test automation engineers don’t like it. Some of them still debate over specific details, but many of them feel frustrated with repeated failures. Deep inside, they know they lack the skills in
Scripting
Maintenance
However, they wish for a silver bullet. After all, switching to another fancy new tool will cover their failures.
Why won’t “Codeless Test Automation” work?
Test creation is only a very small fraction of Test Automation, effort-wise. If a tester still mostly thinks that using a Codeless Test Automation will assist test creation, they will need to learn more about test automation. Ongoing maintenance is where the most test automation effort is.
Test Maintenance
A question: does an AI-Codeless tool make test maintenance easier, compared to Code? Do you believe that AI will detect frequently changed code (I did not use web pages here, as some changes might add or remove web pages), and magically update the internal scripts correctly? I doubt that.
I could do it with raw Selenium WebDriver with RSpec. Below is a recent regression run of my WhenWise test suite.
After getting a green build (passing all 548 Selenium tests), I pushed the new build to production. I have been doing this for all my other apps such as ClinicWise, TestWise, SiteWise and WhenWise since 2012. In other words, I am quite comfortable in maintaining several large Selenium/Appium test suites in my spare time.
Test Execution with Quick Feedback
Assuming you have created a large number of tests, how do you run them? For example, my WhenWise test suite has 548 Selenium tests, and the total execution time is nearly 4 hours if run on a single machine. Because all my test scripts are in Ruby scripts, I could run them easily on any CI/CT server. My choice is the internationally award-winning BuildWise CT server. Its auto-retry feature detected 12 false alarms in the above build.
If using a vendor-proprietary Codeless tool, you will most likely end up with one option: run tests in that tool. Let’s say that the AI-Codeless tool generates a lot of tests as you wish. But have you considered the following?
How long does it take to run all tests?
10-minute build is an XP practiceIf there are false alarms (test failures due to infrastructure or slowness of the server), what do you do? rerun it?
Codeless does not address what it takes to succeed in automated testing
The purpose of Test Automation is not about driving a few user login level tests, it is to enable the team to deliver high-quality software to production every day. Check out my article: “Definition of End-to-End Test Automation Success”. I have worked in software development for over 25 years. I estimate that <0.1% of IT professionals have ever witnessed real test automation.
Let us step out of software development for a while. In an engineering company, to fully automate a process without human intervention, the current manual or semi-automated process must be mature (and work well) for a long period. Commonly, software companies fail test automation multiple times, i.e, there are all failing experiences. They lack the vision of what it takes to win test automation (and usually don’t seek external help). History tells us that betting on a tool after a fancy demo is not wise.
Coding provides the flexibility that Test Automation needs
Before I get to the technical reasons, I will share some indisputable facts first. In 2007, HP acquired Quick Test Pro (QTP). At that time, QTP had been the world’s leading test automation tool for years. With HP’s marketing power plus surrounding products such as Quality Center, most analysts assumed HP would be extending QTP’s dominant position. However, it went the opposite. HP sold its software business (net loss $44M alone in Australia) out to Micro Focus in 2017.
Selenium WebDriver became the most popular test automation tool since its release in 2011. One reason, in my opinion, is that Selenium (and Watir before) embraced powerful scripting, i.e. coding in an open language. Watir uses Ruby scripting language; Selenium has 5 programming language bindings: Ruby, Java, Python, JavaScript and C#.
Before Watir and Selenium, junior automated testers could create automated test scripts using a recorder in a commercial tool with no programming skills required. Technically speaking, it was ‘Codeless Test Automation’. The automation scripts might or might not work reliably, but were certainly not easy to maintain. To address the frequent application changes, the automated test scripts must be flexible (such as Maintainable Automated Test Design, or Optimize Selenium WebDriver Automated Test Scripts: Maintainability).
Anyone who has worked in software development for a long time would know that only coding provides the ultimate flexibility. (otherwise, IT companies will be very happy to replace expensive programmers with AI-coding-robots) And that is what software testing needs. Therefore, this silly pursuit of codeless makes no sense, only benefits tool vendors.
Similar hypes failed before
In the early 1990s, there was a hype: producing code without programmers. I remember the concept was started by a famous professor at Tsinghua University (the top university in China). The idea was that software-generated code based on requirements in a specific format. At that time, I just started my first year at Uni majoring in software engineering. Someone saw that news and advised me to switch to another major because he thought programmers wouldn’t be needed in the future. I found the idea of generating complex code absurd. It turned out that I was right. 30 years on, programmers have always been in strong demand.
President Obama Calls On Every American to Learn Code (Video, in 2013):
“Learning these skills isn’t just important for your future, it’s important for our country’s future”“Everybody in this country should learn how to program a computer… because it teaches you how to think.” — Steve Jobs
Between 2005–2013, I worked on three projects that used a workflow engine (using BPML) with the hope of minimizing programming effort. The selling points were:
Business Analysts can generate working software (by drag-n-drop)
The testing effort is greatly reduced
All three workflow products did not work well at all. For the last project, I joined as the tech lead. After feeling frustrated with the commercial workflow product, I coded a new workflow library in Java and replaced the original one. This new library was well received by the team because it was simple and flexible.
Some might argue: “previous codeless for developing software failed because it was hard, test automation is easier”. Wrong, it is the opposite, “testing is harder than developing”. Try to answer the question below based on your own experience:
how many software projects have you worked on that implemented comprehensive automated regression testing that runs daily?
0, right? Yet, some projects managed to deliver the software.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)
“Codeless Test Automation” is not FUN!
In 2010, I switched my career from a Lead programmer to an automated tester. As you can imagine, I got less pay and respect at work, compared to being in a lead programmer role. The tradeoff is more challenging and fun with test automation (I code and test my apps at home).
In software, an abstraction layer might mean extra uncertainty. I once read this from a book: the Microsoft Excel team had its own C compiler. If a team uses a commercial Codeless Test Automation tool, this is a big and unknown abstraction that will limit your testing-ability and creativity. Yuck!
However, with raw Selenium WebDriver in Ruby scripting language, automated testers can be very creative, as well as super-efficient. A fun job is only sustainable by being creative!
Verifying ‘Codeless Test Automation’ tool is easy
It is easy and cheap to verify whether these so-called ‘Codeless Test Automation’ tools are fake. Please remember, functional test automation has very clear objectives. Don’t let talkers confuse you with other factors.
You can ask the salesperson: “You are a testing company, I assume you will not be against a quick test, right?”. Then, invite them to attend “Test Automation and Continuous Testing Competition Week”, in which you need to prepare some real tests (~25) for your main app.
So far, I haven’t experienced a so-called AI-powered Codeless test automation tool. But I can tell you, I have once completed the challenge (25 real End-do-End automated tests for a comprehensive web app unknown to me, while application changes) within 2 days by using the AgileWay Test Automation Formula (with all 100% free technologies, such as raw Selenium WebDriver). Business analysts usually find my automated test scripts very readable.
Update (2022–03–12), saw a good LinkedIn post by Jeffrey Schwab.
“Please stop pushing for (or buying into) no-code solutions. No code means no real version control, diffs, pull requests, or reviews. No code means vendor lock-in. No code means hard to search, hard to control, and hard to integrate. No code means somebody else has you by the unmentionables. No code means you’re going to waste your time developing non-transferrable skillsets.
Code is not your enemy. A pile of crappy code is admittedly no better than a pile of crappy anything else, but the fact that people sometimes drown does not mean you should avoid water; nor should you go on an anti-swimming campaign, even if you personally don’t swim. Do not fear the code. Code is your friend; or more accurately, it’s the tool that should be your go-to solution for most technical challenges. Whatever grief you fear from code, no-code/low-code solutions are almost universally worse.”