Why Test Automation Tool with Runtime License is Evil?
It is unwise to choose automation software that locks customers in for maximum financial gain. The best WebDriver (in every way) is 100% free.
The Runtime in the context of software execution means the software required to run a program (developed by you). To my knowledge, the term starts from Java Runtime Environment.
JDK (Java Development Kit) is for developing a Java program; JRE (Java Runtime Environment) is for running a Java program. The runtime software for coding is usually free.
In test automation, engineers use test automation tools to develop/refactor/run automated test scripts. The runtime usually means a sub-part of the tools that only allow users to run automated tests.
“free software” is a matter of liberty, not price. — GNU
It is perfectly fine for a software vendor to charge customers for a commercial tool license, However, in my opinion, it is not appropriate for a license for the Runtime.
Tools are for enhancing productivity, i.e., optional and customers may select alternatives.
Runtime is mandatory to run, i.e, no choice.
Sadly, this is quite common in test automation:
Ranorex Runtime License: €690
Micro Focus UTP Runtime Engine: $2,300/year
SmartBear TestExecute: $629/year
Please note, the above is the price of the Runtime, the tool’s price is higher.
And it is wrong. I understand many people will say “in a marketing economy, prices are justifiable”. But this does not mean “right”.
Why Runtime Licenses are Evil?
The purpose of Runtime licenses is to maximize financial gains for these tool vendors. Yes, this action itself does not necessarily mean evil in our time.
“The love of money is the root of all kinds of evil. Some have wandered away from the faith and have impaled themselves with a lot of pain because they made money their goal.” — 1 Timothy 6:10, Bible
However, in the context of test automation, it is.
1. The meaning of Evil in the industry
“Evil”, when used in the context of the IT industry, its meaning is not that strong. For example, “Don’t be evil” is a phrase used in Google’s corporate code of conduct.
Evil in IT generally means locking customers into using an inferior product at a high price.
2. The alternative and the best (in every way) is 100% free
In a healthy marketing economy, there is nothing wrong with a company charging a high price for its product given the product is better than other competitors. However, if the alternative is much cheaper and excels in every way, it is wrong.
This alternative in web test automation is Selenium WebDriver. WebDriver is far better than all other frameworks/tools, and it is at an unbeatable price: $0.
Some typical benefits:
Support
Check out WebDriver’s support ticketing system, I raised a bug with a release before, and it was quickly fixed.Easy to learn
Selenium is a lot easier to learn. Check out this article: Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutesSpeed
While functional test execution largely depends on the application itself, the raw Selenium WebDriver is quite fast, and better than Cypress. Check out “Cypress vs Selenium vs Playwright vs Puppeteer speed comparison”.
Please note, compared to others, Selenium WebDriver is the only feature-complete framework.Proven
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
For more, please read my other article: False ‘Selenium WebDriver Cons.
Let me repeat, WebDriver is far better than all other test automation frameworks in every way! That’s why Google and Facebook used it, even Microsoft dumped its own Coded UI Test and recommend WebDriver.
For testing web apps, it is simply insane (or fishy) to choose anything else.
3. High price might be easier to sell, with dodge practices
I have been working in this field for over 15 years, and have seen some inappropriate marketing practices. Around the Year 2010, a typical test automation tool license was around US$10,000. High prices mean high commissions …
My own testing tool TestWise was made public in 2009, a number of companies and individuals approached me for marketing TestWise. A common request was “TestWise’s price was too low, ~$500. And you should not offer the free community edition”. Some told me directly: “If TestWise is priced higher, ~$5000, it would be easier for them to convince management to purchase”. I rejected all of them. The reason: this is not just about money.
I believe:
High-Price Software Testing Tools Fail Test Automation!
Why do high-priced tools always fail?
First of all, I have never seen one single successful test automation using those expensive (>US$500 per seat) testing tools. My definition of ‘Success’ is the objective. (check out Definition of End-to-End Test Automation Success and Agileway CT Grading).
1. Executing test automation tests for all team members
Automated End-to-End Testing executes automated tests to perform functional tests. The audiences of automated tests are:
Test Automation Engineers
Developers
Business Analysts
Manual Testers
All the above spend a significant percentage of their work time verifying business features via UI. In other words, the whole team can take advantage of test automation. As a matter of fact, in a real DevOps project, the whole team must get involved in UI test automation.
This means the cost of software testing tools/runtime matters. For example, the total cost for Ranorex licenses for a 10-staff team would be over US$50,000 (€4,790 each). Comparatively, a raw (and far better) Selenium solution costs $0.
2. Continuous Testing
To effectively execute a large number of automated E2E (UI) tests, we need to run them in a proper parallel testing lab. This means every build agent needs a test execution runtime. Now, can you see the inconvenience now?
Furthermore, some DevOps engineers like to tear down and build up fresh build agent machines, runtime licenses will add more hassles.
In summary, be wise with your choice. The best stuff in test automation is often free (freedom) and free/low price. If you just want a simple answer, check out AgileWay Test Automation Formula. If you want to come up with a formula, I recommend holding an event such as Test Automation and Continuous Testing Competition Week.