Finally, A TestWise (Functional Testing IDE) Competitor

JetBrains Aqua, for people frustrated with Micro Focus UFT, Ranorex, Tosca, Cypress and Playwright.

Dec 19, 2022

Today in my LinkedIn feed, I saw a video introducing JetBrains Aqua. This host seemed pretty excited about this test automation IDE. What he did not know, a similar testing IDE, TestWise, existed 15 years ago.

Next-Gen Functional Testing Tool
Agile Alliance workshop on Next-Generation Functional Testing (Oct 2007), Envision “We are lacking IDE that facilitate things like:”
* refactoring test elements
* command completion
* incremental syntax validation
* keyboard navigation
* debugging…

I like JetBrains coding IDE (Intellij IDEA, RubyMine, PyCharm), especially Intellij IDEA, I could remember 99+% its keyboard shortcut. At one TDD presentation, I implemented one user story (live) on request. Some audience was stunned, and one said, “You did all of that without using the mouse”.

I Welcome JetBrains Aqua

I am quite happy to find out about JetBrains Aqua. Why? Isn’t it a TestWise competitor? Yes and No (see below for more).

I started developing TestWise in 2006 solo in my spare time. While TestWise got some attention (listed in world-renowned test experts’ presentation and a finalist in an international award), it is not a commercial success. There are many reasons, such as:

I don’t do any marketing or promotion
A good testing IDE is not enough for successful test automation. Without a proper Continuous Testing server, sooner or later you will hit the limit.
I am busy developing my other apps, using TestWise and BuildWise daily
Most (>99%) of test automation attempts failed using various bad and expensive commercial test automation tools, such as IBM’s RFT, HP QTP, Microsoft Coded UI tests, Ranorex, Tosca and TestComplete.
(some are already deprecated)

TestWise has been used in a very small group of elite automated testers. Frankly, I don’t mind it that way. I have been using TestWise every single working day since 2007.

After my test automation consulting, commonly, a good implementation of automated test scripts was replaced by a new test manager to introduce a commercial tool. A few mentees told me this in a sad tone. The reasons were:

“TestWise does not support record-n-playback”
Recorded test scripts are hard to maintain.
“TestWise does have object-identification GUI utility” to scan a web page to generate the “Object Map”
see why this is bad?
“We prefer codeless test automation”
check out this article, “Codeless Test Automation is Irrational”

Any of the above shows the test lead did not understand test automation at all.

Another frustrating reason is that “TestWise is not from a big software company”. Despite all test scripts created by TestWise being plain-text, raw Selenium/Appium (i.e. as open and free as it can be). If someone has prejudice, what can I do?

Over the years, I received commercial interests (to invest) in TestWise, they wanted TestWise to be more like other commercial testing tools, such as proprietary syntax, less open, adding record-n-playback, no free edition, raising license price to near others (~$5000), … etc. I rejected all of them. I believe TestWise is what a functional testing IDE should be.

The arrival of JetBrains Aqua (still in preview) boosted my confidence in my approach. A large and reputable software vendor is about to release a testing automation tool as a script-based (vs mouse operation) IDE, supporting the framework (Selenium) and practices (Page Object) TestWise has offered for over 11 years.

JetBrains Aqua’s General Pros

Powerful code editing
JetBrains Aqua will have the usual powerful coding features, including auto-complete, code-refactoring, snippets, …, etc.
A big name among software developers
Many software companies already use JetBrains tools, such as Intellij IDEA and TeamCity. It will be quite easy for companies to adopt it.
Supports five languages: Java, Kotlin, Python, JavaScript and TypeScript.
All these languages, as you can see, with strong software developers’ tendance. In my opinion, JetBrains is trying to sell Aqua to developers who might be trying test automation (usually not seriously).

JetBrains Aqua’s General Cons

It is Huge
Aqua’s macOS version download size is 779MB.
TestWise macOS version, only 8.9MB.
It will be slower
All JetBrains IDEs take a while to launch because it is written in Java (starting virtual machine).
TestWise launches in seconds, as it is written in C++, period.
It does not support Ruby, yet.
JetBrains certainly could add Ruby support (they already got RubyMine), but for commercial reasons (there will be considerable development and maintenance effort to justify), they probably want to just focus on popular ones in the test automation market.
TestWise supports scripting automated tests in Ruby best.

My mixed feeling about JetBrain Aqua’s approach to Web Test Automation.

1. Support only Selenium WebDriver ✅

At far as I can see, JetBrains Aqua only support Selenium for web UI testing. This is correct.

2. Support Page Object ✅

“Page object models” is listed as one of the best practices in Selenium’s documentation. In many commercial tools, they use “Object Map”, appearing easy but a very wrong way.

JetBrains Aqua, a new testing tool in 2022/2023, got it right, just simple Page Object Models.

3. A proper IDE for scripting/debugging automated test scripts ✅

👍Test-Script-focused. After all, it is the test script that matters. The so-called conventional test automation tool tries to hide test scripts, promoting just using our GUI interface. An extreme case of that is “Codeless Test Automation”.

4. Shall be cheap! ✅

Intellij IDEA’s IDEs are priced between US $249 and US$ 599 per year. I would assume JetBrains Aqua will be the high-end of that range. If so, compared to other commercial tools, it is a bargain.

Test Automation tools fit in one of the most expensive software categories:

Smart Test Complete Pro: US$ 5,075
Ranorex Enterprise: US$ 6,950
Tricentis Tosca: Pricing On Request
(from ANZTB 2010 conference’s sponsored speech, “only $25,000”, shall be cheaper now, thanks to Selenium WebDriver)
Micro Focus UFT One: US$ 3,200 /year

Beyond that, you also need to pay for runtime licenses (to use in test execution in CI/CD) as well. Check out my article, Why Test Automation Tool with Runtime License is Evil?

By the way, a TestWise licence has always been set under US$500 (current price: $30/month), due to my belief below:

“Every member in a software team can get benefits from running Automated End-to-End Tests. “ — Zhimin Zhan

Almost all investors contacted thought my idealist belief silly (of course, they used different terms), back 10+ years ago, the average testing tool price was over US$10,000 plus support. Anyway, I insisted. If JetBrains Aqua is priced in its normal range, which kind of proves my vision (15 years ago) wasn’t totally silly.

5. Embed Web Inspector inside the tool ❌

JetBrains Aqua embeds a web browser control inside the tool. It is, IMO, for marketing.

It is really not necessary, as you have seen my and my daughter’s numerous articles, it is very easy to inspect the normal Chrome browser.

Using an embedded web browser might appear handy, but it will encourage bad locator statements. Even in the JetBrains Aqua demo, the example locator cssSelector: "span[class='QCzoEc z1asCe MZyiRb'] svg is not good.

6. Shall support and recommend Ruby language ❌

Different from coding, the programming language is set, e.g. Java for Android, JavaScript for React app. The situation is different for web test automation, one scripting language is good for all websites. (I have been writing Ruby for all websites for over 15 years).

In my opinion, within the web test automation context, a real test automation engineer must have his/her preferences.

7. Target developers, not business analysts and manual testers ❌

This is not surprising, customers buy JetBrains tools for coding. However, automated end-to-end (via UI) tests are for all team members, including business analysts and manual testers. See, for those non-tech people, JetBrains tools are too complicated.

I have worked with many business analysts and manual testers, after showing them IntelliJ IDEA/NetBeans/Eclipse/Visual Studio Code and TestWise (for running automated end-to-end tests), every business analyst and manual testers chose TestWise.

8. Not free ❌

While JetBrains Aqua is yet to be released, with the information I see, it is fair to assume there won’t be a free version.

TestWise offers free mode.

Overall, I welcome JetBrains Aqua, another functional testing IDE to help lead automated testers to do proper test automation. The reputation of Test Automation Engineers has been severely damaged over the last decade by fake automated testers using bad frameworks, tools, and practices, such as Recorder-based tools, PhantomJS headless testing, Protractor and Cypress.

Test Automation Engineers shall focus more on the test scripts, which matter most. Don’t fall into the hype of codeless testing or a tool does all. Use W3C standard compliant Selenium WebDriver, with the power of a good scripting language like Ruby. Above all, understand the most effort in test automation is maintenance. To enhance the maintainability of test scripts, treat them as code, which requires constant refactoring.

The Agile Way

Discussion about this post