Katalon Codeless Test Automation vs Selenium WebDriver by Example
Put the claims of Katalon, the so-called leading Codeless Test Automation tool, under the microscope. My verdict: Avoid Katalon, Avoid Codeless Test Automation!
Is Codeless Test Automation another hype? This article will compare Katalon Studio, listed by LambdaTest as №1 in the Top 10 Codeless Testing Tools For 2021, with the well-proven script-based Selenium WebDriver.
Below are Katalon's claims.
From its demo, "Low-Code Test Design".
By the way, the above is an illustration, not an actual Katalon. Katalon Studio looks like this:
It looks like rebranded Eclipse IDE, a busy and confusing interface for testers. The fact that Katalon dare not show its product in its own marketing demo says it all.
IDE is a coding tool used by developers, Eclipse is known to be complex. Test Automation Vendors developed their tools based on the Eclipse platform for simple reasons: exploitative and cost saving. The Eclipse platform is free and mature for coding (in particular Java). I have reviewed IBM’s Rational Functional Tester (RFT) before (it was recommended by Gartner Report 2013 as the best test automation tool). RFT automation was ill-designed and tester-unfriendly, now RFT faded out of the test automation market (at least I haven’t heard anyone mentioning it for 5+ years).
In this article, I will verify Katalon’s claims.
95% Time Reduction in Test Creation
50% Cut in Regression Test Cycles
90% Decrease in Test Maintenance Efforts
1. Test Creation
Claim 1: "quick to create automated tests"
In Katalon's demo video, there was actually only one incomplete test step (6 seconds, 19' to 25') shown: to set text on the input_Username (via drag and drop) text box. However, the text wasn’t even entered on the demo.
Why? Because it would take several seconds to do it. It would appear less snappy (for this illustrated demo). After that, people might wonder what would be next. Then there would be another round of mouse moving to "+Add" → "Click" → Drag the button over from the object repository on the left. In fact, experienced automated testers know that this is a quite slow way (opposite to Katalon’s claim) to create automation tests. (For readers who are in doubt, in a moment, I will show creating the same test script in raw Selenium WebDriver, which is much more efficient).
Can you see the problem here? Even Katalon's demonstrator didn’t have the patience to complete a single test step when a demo aimed to show Katalon’s best capabilities.
Let's assume that creating one test step (enter short text 'Really' in a text box) in Katalon takes 10 seconds, which is the best-possible timing with IFs:
the user controls the mouse well and clicks the right buttons correctly
the web page is simple, with only a few controls. Otherwise, the user must scroll a lot to find the correct control identifier.
Don't forget the above timing is based on Katalon's animation, not real actions (I did spend some time searching for them, but was unable to find a simple and real demo). It is safe that the real timing will be at least 2X longer, which is 20 seconds.
Claim 2: "95% Time Reduction"
Based on Katalon's claim of "95% time reduction", other frameworks/tools shall take over 5 minutes for this test step (20s in Katalon). Let's verify that.
I will write the same step in raw Selenium WebDriver, using TestWise. Both of these have been there for over a decade.
The above is a quite standard IDE feature: Snippet. With a good extension, you can achieve the same speed with Visual Studio Code too, in terms of entering test scripts.
The time: 16 seconds, not 400 seconds (as Katalon’s claim of 95% reduction).
Verdict: Katalon's claim is false. Raw Selenium WebDriver is faster to develop in TestWise.
As shown in the above animated GIF, there were no skips or special editing to the process, just the normal steps on a test case creation. I have been doing software testing this way (Inspecting a page element in Chrome, then typing in raw Selenium step) since 2011 for many successful projects.
As you can see from one simple test step (Katalon didn’t even show a complete step in their demo), Katalon is definitely not a big time saver as it claimed (in fact, it is slower than scripting raw Selenium in TestWise).
Astute readers might have already figured out that my way is generic and universally applicable, while Katalon is dependent on its Object Repository, which is problematic and often useless for complex web pages. For more on that, please read my other article Why is Object Identification GUI Utility in Test Automation Unnecessary?
In terms of other factors, Codeless Automation tools are no advantages either.
2. Ongoing Test Execution (regression testing)
Automated End-to-End Testing is great for regression testing.
Claim 3: "50% Cut in Regression Test Cycles"
In my opinion, Katalon's claim on this is against manual regression testing, which is meaningless in this context (not apple vs orange, shall apple vs apple, I will show my proof shortly).
If this claim is to compare in the context of real automated regression testing, then this '50% cut' claim is a joke. In real Agile/DevOps projects, automated regression testing happens multiple times of day, we need absolute numbers, such as 30-min automated regression testing of 200 automated end-to-end tests.
Why? I spent ~70% of the whole software development effort in continuous testing (running a full automated E2E test suite as regression testing), which enabled me or my team to develop software with at least 10X development productivity and daily production releases. In other words, automated regression testing is essential to real agile/DevOps.
The above is a typical DevOps diagram, everyone in a software team is involved in Continuous Testing.
I will provide specific figures of regression testing for my apps:
ClinicWise, 611 Selenium WebDriver tests, 57 minutes (2013-)
TestWise: 307 Appium + WinAppDriver tests, 45 minutes (2019-)
TestWisely: 80 Selenium WebDriver tests, 8 minutes (2022.2 -)
WhenWise: 551 Selenium WebDriver Tests, 36 minutes (2018-)
As a comparison, a government test team (in which I worked as a consultant before) had about ten manual testers, and one regression run will take the team 1 month to execute about 3000 test cases. I developed an automated suite (this is API testing) that only took about 10 minutes. If I used that as a reference, my saving (with Ruby tests) of regression testing (manual) would be "99.99+% cut".
Verdict: Katalon's claim is invalid or wrong. Raw WebDriver is much better.
3. Test Maintenance
Automated test scripts require ongoing care and maintenance. The maintenance effort will be far bigger than the test creation itself as web apps change often.
The Fact: one minor code/infrastructure change might break an unknown number of automated end-to-end tests. — Zhimin Zhan
Claim 4: "90% Decrease in Test Maintenance Efforts"
Katalon's so-called reduction in Test Maintenance Effort refers to its "Object Repository", an old and failed approach used by last-generation commercial testing tools such as IBM's UFT and HP's QTP. This claim is based on the logic:
the tool re-generates the object repository when a web page changes
the "automated tester" does 're-dragging’ to update the tests
The above only works for simple and prepared scenarios, e.g. the sales demo. First of all, as the web page is now very dynamic, the object repository is often wrong. Secondly, can you imagine dragging and dropping (or scrolling up and down) a large number of elements if there is a long web page with 200 elements? The whole thing was made for demo purposes, it is a joke.
I have shown that I have been managing several large regression suites (in my spare time) previously. For example, the WhenWise test suite has ~28000 test steps (in my test scripts, one step is a user operation):
+------------+---------+---------+---------+--------+
| TEST | LINES | SUITES | CASES | LOC |
| | 26193 | 336 | 561 | 20542 |
+------------+---------+---------+---------+--------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 9883 | 181 | 1625 | 7466 |
+------------+---------+---------+---------+--------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 823 | 5 | 61 | 637 |
+------------+---------+---------+---------+--------+
| TOTAL | 36899 | | | 28645 |
+------------+---------+---------+---------+--------+Because I develop my automated tests based on Maintainable Automated Test Design, I can maintain several test suites (for different apps) these years, all in my spare time.
“Only well-designed tests in an Object Oriented Scripting Language, such as Ruby, provides the ultimate flexibility to ease maintenance efforts. “— Zhimin Zhan
Verdict: Katalon's approach is non-viable; raw Selenium WebDriver in a good design (Maintainable Automated Test Design) is much better.
4. Test Scripts
Inexperienced (or fake) testers might think Katalon's "codeless test scripts" look better.
First of all, the above is an illustration. The real test script in Katalon Studio looks like the one below.
Compared to illustrations created by graphic artists. The real scripts are far less appealing.
The above is a login test in Katalon. Have a look at an equivalent one in Selenium WebDriver.
# Story: Login to AgileTravel
driver.get("https://whenwise.agileway.net/sign-in")
login_page = LoginPage.new(driver)
login_page.enter_email("drving@biz.com")
login_page.enter_password("CoolApp01")
login_page.click_sign_in
expect(page_text).to include("Login successful")Many manual testers found that the selenium version was very easy to read. Besides the natural syntax, the above script is very easy to maintain. Check out this article Maintainable Automated Test Design.
“For every Automated Test Case, there are always codes behind it. It is just a matter of whether you can see it or not. “- Zhimin Zhan
To achieve better maintainability, it is far better for automated testers to work on test scripts with fewer or no abstractions. Every layer of abstraction, such as Gherkin, will introduce extra maintenance and proneness to errors.
Lack of Essential Features
Unlike manual Excel scripts, automated Test Scripts can be scripted using some programming logic. One reason that free Selenium Driver defeated HP's QTP and its similar commercial tools is that selenium comes with a programming language binding. (By the way, that is also where the fun comes from)
Codeless test automation disconnects the automated testers from programmable scripts. For this reason, it is doomed to fail. Here I will show a few common testing steps I used.
Parsing or Extracting Text using Regular Expression
<! — TestWise Version: 6.6.12 -->
...
<! — WhenWise Version: 3.0.6 -->
# the text in web pageapp_vers = driver.page_source.scan(/<!-- (\w+) Version: (.*?) -->/)
puts app_vers.inspect # [["TestWise", "6.6.12"], ["WhenWise",
"3.0.6"]]
expect(app_vers.size).to eq(2)
expect(app_vers.first).to eq(["TestWise", "6.6.12"])For more examples, check out Use Regular Expressions in Test Automation.
Some tool vendors might say: We have implemented a widget that supports regular expression. Yes, you might have. But that will add another layer of syntax (created by one progrmmer, and often buggy).
If and Looping logic
table_cells.each do |tc|
tc.click if tc.text.include?("special")
endDomain-Specific Language (DSL)
login_page.click_sign_in
# or
login_page.click_login # change to term that the team usesCalling API, SQL Query or external services
require 'mysql2' # gem install mysql2
client = Mysql2::Client.new(:host => "10.0.0.1",
:username => "tester",
:password => "wise",
:database => "buildwise_production")
results = client.query("SELECT duration FROM builds WHERE project_id=3 ORDER BY duration DESC")
longest_build_time = nil
results.each do |row|
longest_build_time = row["duration"]
break
end
# found the longest build timeLack of Freedom
To my knowledge, all codeless test automation tools are proprietary, developed and marketed by a commercial tool vendor.
1. Vendor Locking
I don't like vendor locking. Check out my article: Crazy Web Test Automation: "Freedom Is Slavery."
2. Unable to switch to another tool
If your team finds a more productive tool, you will not be able to switch over with proprietary test syntax, With raw Selenium tests, you have total freedom 😊.
3. It might be difficult to run another Continuous Testing server
I haven't tried or researched executing Katalon tests in CI/CT servers such as Jenkins and BuildWise, but I won’t hold my breath for it. With raw WebDriver tests, there is no limitation at all 😊.
4. Buggy
All software has bugs. From my experience, those expensive test automation tools have a much higher percentage of bugs. For example, in 2013, I reviewed IBM’s RFT. Among many bugs, the craziest one is “failing on dual-screen monitors”, this issue is still listed on IBM’s website, unresolved.
While Selenium WebDriver is well supported, including browser vendors such as Google. Yes, Selenium WebDriver has bugs, but the team usually fix them quickly. Check out this article for an issue I had with Chrome v103 recently, Resolve ChromeDriver Issues in Web Test Automation. By the way, Selenium Users get this top-level support absolutely free.
5. Cost
With raw WebDriver tests, the cost: $0 😊.
Lack of Flexibility
I have worked with software that has an abstraction layer like Katalon. Frankly, I could develop a tool like Katalon quickly myself (effort-wise, much less than TestWise), but I won’t because that would be evil. In fact, I received at least two commercial requests to ask me to turn TestWise into something like that, I rejected them.
A common problem for multiple abstraction layers is that they lack flexibility. Over time, users will find more and more limitations and get frustrated. I will list a few common issues:
1. Dynamic Test Data
new_email = Faker::Internet.email
sign_up_page.enter_email(new_email)2. Random value input
new_member_page.select_gender(["Female", "Male].sample)For more, check out Practical DDT: Randomized data scenarios in one automated test script + Continuous Testing
3. Calling external libraries
A good scripting language, such as Ruby, has a large number of good libraries that will assist your testing. I showed Faker earlier for generating test data.
Here is another one,
date_17_years_ago = 17.years.ago.strftime('%Y-%m-%d') # 2022-07-12
driver.find_element(:id, "dob").send_keys(date_17_years_ago)In summary, avoid Codeless Test Automation if you can.
Related reading: