My Innovative Solution to Test Automation: TestWise, the Next-Gen Functional Testing IDE
Help QA Engineers to be highly productive!
While I was writing the “My Innovative Solution to …” series, such as “TestWise’s Keep the Browser Open after Executing an Individual Test” and “TestWise’s Attach test execution to the existing browser”. I realized that the TestWise tool itself is an innovation, as there was (still is) no dedicated Selenium/Watir testing tool that qualifies “Next-Gen Functional Testing IDE”.
Table of Contents:
· A Story
· TestWise Acknowledgements
· What is special or innovative about TestWise?
· TestWise Development Milestones
∘ iTest2 v1
∘ TestWise v2, v3, v4
∘ TestWise 5
∘ TestWise 6
· Commercialization History
· MarketingA Story: Don’t let others destroy your dream
At CITCON 2009 conference (informal format, which I quite like), I was alone preparing to show something. One person came by and saw the test execution on my laptop, he asked: “What is that?”
I replied: “Automated Testing with Watir”.
He said: “OK, I know it. (Watir was the mainstream in web test automation then), What tool did you use?”
I answered: “a testing IDE I developed. I named it …”
He looked at me, and said in a mocking tone: “You, created a testing tool?! IDE?!, hey, hey …”. He shook his head and walked away.
It might sound crazy for a programmer to create a functional testing IDE from scratch in his/her spare time. Yes, I did that. The acknowledgements, praises, and solid proof of commercial customers proved that TestWise, my testing IDE, has been successful. I have been using TestWise since 2008 every working day.
Frankly, when I started this journey of developing a functional testing tool, I didn’t think much about the effort, or the challenges, just wanted to use something more productive to develop/run/debug automated test scripts.
“Lesson Learned: Don’t let others destroy your dreams.” — Zhimin Zhan
TestWise Acknowledgements
TestWise reached the world stage.
In the presentation “Trends in Agile Testing” (2009) by Lisa Crispin, co-author of the Agile Testing book, iTest2 (TestWise’s former name) was listed as the first testing tool in the “Testing Tool Spectrum” slide (#24). Lisa used the same slide in her other talk, “Overview of Agile Testing”.
Here is a blog post by Mike Longin, an Agile 2009 speaker: “iTest2 (http://itest2.com/) was also shown off and really goes a long way in my mind to show off the capabilities of an IDE for functional testing tools. As the project lead for SWAT, one of my biggest pushes has been to take the tools I get in Visual Studios and bring them to the functional testing world. At least for Watir, iTest2 is there.”iTest2 was demonstrated (not by me) at Agile 2009 AA-FFT in Chicago, and listed in Ward Cunningham (co-author of Agile Manifesto)’s Functional Testing Tools list.
In October 2013, I demonstrated functional test refactorings (my presentation title: “Refactoring Automated Functional Tests”) at the STARWEST conference (‘one of the longest-running and most respected conferences on software testing and quality assurance’ in the world) with highly positive feedback.
a finalist in the 2010 Ruby Award, judged by the software legend Matz.
What is special or innovative about TestWise?
TestWise, to my knowledge, is still the sole testing tool developed from scratch to support only free and open test frameworks.
Support only WebDriver (Selenium and Appium) tests
(Watir as well, later Watir becomes a syntax layer on top of Selenium)
You might have heard of commercial testing tools such as HP QTP, IBM Rational Functional Tester, Ranorex, Tosca and TestCompelete. They all have their own test proprietary script syntaxes, why? Vendor lock-in! (that’s why these tools are very expensive)
Selenium WebDriver is 11 years old, besides TestWise, do you know any other tools that support Selenium only? I doubt that. A tool supports only an open-standard means competition. This kind of project proposal (for a comprehensive tool) will be shot down immediately in a typical software company.TestWise is simple and intuitive to use because it is purposely designed for testers.
You might have seen some testing tools built on the Eclipse platform, such as IBM’s Rational Functional Tester. It made development sense, saving a lot of coding effort on file management, UI, editing, version control, …, etc. I understood that because I had to code every feature of the above myself. But there is a cost of using Eclipse or other platforms, the resulting tool will be complex, that manual testers or business analysts will find it hard to use.No record-n-playback
Although record-n-playback has a bad reputation (generating unmaintainable test scripts), many commercial testing tools only highlight their recording feature. Yuck! Why do they do that? record-n-playback looks “cool” in sales pitches.
(by the way, I developed a separate TestWise recorder as a browser extension, but I rarely use it)Freely available
Not only those commercial testing tools are expensive (back 10 years ago, typically ~US$10,000 a license), there are even evil runtime licenses.
I want anyone, regardless of income or region, can use a fully functional TestWise (with just a minor limitation compared to licenced) to learn test automation and advance their career. One time I received a thank-you email from one tester in Ethiopia, which made me proud.
Next-Gen Functional Testing Tool
Agile Alliance workshop on Next-Generation Functional Testing (Oct 2007), Envision “We are lacking IDE that facilitate things like:”
* refactoring test elements
* command completion
* incremental syntax validation
* keyboard navigation
* debugging…
Unique Features
Prior to the Agile Alliance workshop in 2007, I had already implemented most of the features of Next-Generation Functional Testing. The special mention is Functional Testing Refactoring, I came up with the concept of six refactorings, and implemented them in TestWise.
TestWise Development Milestones
2006–04–11: the proof of concept starts
2007–09–25: v0.1 first version that Zhimin made available for a colleague.
2009–01–05: v1 first public release, supports Watir/RWebSpec v1
2011–08–18: v2 supports Selenium WebDriver, Watir 2.0 v2
2012–07–11: v3 supports
watir-webdriverand an important feature: “run selected test steps against current browser” v32014–08–18: v4 supports RSpec 3.0 v4, our main version until 2019.
2016–01–13: v5 a try of moving execution process out of IDE, to support Ruby 2.0 syntax.v5
2018–07–01: v6 development starts, an ambitious goal: a total rewrite in C++.
2020–03–24: v6.0 is publicly available
iTest2 v1
TestWise v2, v3, v4
Renamed to TestWise, support Selenium WebDriver as well as Watir.
With the “Attach test execution to current browser” feature, TestWise 3 (and later version 4) helped me to be productive in developing Selenium WebDriver for a few years until 2017. Selenium WebDriver 3 dropped the support for Ruby v1.8, on which TestWise 4 depended.
This limitation was set by wxRuby (a port of wxWidgets), a GUI library TestWise used. wxRuby was deprecated in 2011. It still works well but is limited to Ruby 1.8.
When Selenium 3 was released (October 2016), I had to download its ruby binding and perform quite a large-scale change to make it still work with Ruby 1.8 for every minor release. It was quite some effort, every time. I did this for me and my clients (including free community edition users) to enable us to continue to use this “Attach the browser” feature.
TestWise 5
I realized that it was just a matter of time before TestWise would end if unable to find an alternative to the deprecated wxRuby. I worked on two new prototypes in two languages:
Python
So that I could use thewxPythonlibrary. While Python is quite similar to Ruby, I did not have a lot of experience with it.Java, using a totally new Java UI library
I am quite a good Java programmer (started with v1.0.2), but don’t like Java GUI programming.
Anyway, I made some progress (demonstratable) on the above but realized it would be too much effort for me to handle them as a part-time project.
TestWise 6
One day I read an article that stated that “C++ 11 is the new C++”. A crazy thought came to me, “Why not go directly to the root? Develop TestWise in C++”. I coded in C++ briefly in my early career. To be frank, I feared using pointers, memory management, …, etc. But I love test automation, love developing my own apps, love daily production releases, …, etc. Without a high-productivity testing tool such as TestWise, all the above will be in vain.
On 2018–07–01, I started a total rewrite of TestWise in C++. Long story short, 18 months later, TestWise v6 was released. The new TestWise supports “Attach the browser” (of course, the usage is the same), and it is much faster (launch < 3 seconds) and better looking.
Commercialization History
Before 2009, I had not thought about the commercialization of TestWise. People (most are colleagues) just downloaded and used it. Then, a few people wanted to purchase it. Free was no good for many organizations, why? they wanted the security of product support and maintenance (this was 13 years ago, but is still valid for some).
Many would think “people wanting to buy your product is a good”, mostly yes, but I have hesitations. Because TestWise is my side project, commercialisation means there will be a lot of work ahead for me:
Website and purchasing
Documentation
I feared writing then, a lot. I still much prefer code and scripts automated tests than writing docs.Video Tutorial
often need to redo them many times.Support Ticking System
Marketing …
Anyway, I gradually set the above up, which took me quite a while (I developed SiteWise CMS, a content management system; SupportWise, a support system).
I kept my original vision: a 100% free usable version for everyone.
TestWise community edition (up to v4)
with some functional limitations, totally useable.TestWise professional edition, priced at US$500
I still felt not right to have a crippled community edition. Thanks to TestWise 6’s instant launch speed, I dropped the functional limit and introduced TestWise free execution mode: 15 test executions per launch, to continue, just simply relaunch it, which takes about 3 seconds.
The unlimited TestWise professional edition is priced at only US$30 per month.
Marketing: none
I am no good at marketing and don’t like doing it. To give you an example of how bad I was in marketing. At a government department, a test lead hinted that “If TestWise is priced 10X higher, the department will purchase a number of licenses”. I declined, stating that the TestWise price is static.
A bit background info about Testing Tool high-pricing: the department purchased IBM RFT for US$11000 a license based on a Gardner’s recommendation. IBM RFT failed badly, and Selenium WebDriver with the help of TestWise proved working well. Logially, it is stupid to ask to pay $5000 for $500 tool (TestWise’s price then).
But have you consider in the CIO’s position. He wasted so much money on RFT, then replaced that huge cost with $500 tool. It was not acceptable politically.
Therefore, there is never been marketing for TestWise. In recent years, there have been fewer commercial customers, despite the fact that TestWise is getting better. The reason: with growing demand in test automation, a set of so-called new tools (still bad, such as Cypress) heavily defame Selenium WebDriver to promote theirs. A few of my former clients, switched from Selenium to Protractor/Cypress/Playwright, and their test automation attempts totally failed (check out this article, Story: A Former Mentee Failed Test Automation with Protractor). The reality is not changed over the last two decades: 99+% test automation attempts failed (check out the Definition of End-to-End Test Automation Success).
People might ask, “Why don’t you hire someone to do marketing for you?”. There was no shortage of former colleagues who wanted to get involved, but they didn’t do well. The main reasons were:
Selling a practical tool such as TestWise is not the same as others. This requires hands-on rather than just talking well.
They were doing part-time, as you know, easy to give up.
They didn’t really (nor had the desire to) understand test automation well.
Anyway, if you are good at marketing, like real test automation and willing to make passive income, you may consider joining our new generous Marketing Partner Program (not officially announced yet) to earn some ongoing passive income, as well as help others.
Further reading: