A few good calls I have made on Test Automation and CT journey since 2005
Lessons I learned from 15 years of hands-on experience in Test Automation and Continuous Testing
The title might sound like a brag, but I am old enough to know that it is pointless to brag in a blog article as not many people would read it (for the younger generation, except academic studies, not many people have the patience to read anything longer than 140 characters). Rather, I want to share my lessons with people who are passionate about software testing, and hope that they might get something out of my past experience.
Let’s face it, test automation and CT are two areas that people can easily make wrong choices in. 95+% of test automation attempts failed though many claimed ‘success’. To check whether test automation is really successful is incredibly easy: Is your team confident to deploy the new build (with changes) today? , or assess your project against AgileWay Continuous Testing Grading.
One reason that I have done OK with test automation is that I have made some right calls along the way.
I perform test automation efficiently, develop/maintain thousands of Selenium tests to enable 3 of my web apps release to production on a daily basis. On top of that, I developed TestWise IDE and award-winning BuildWise, and authored 12 books.
Some would say it is easier for you to say “I predicted it” now. But I do have the proofs such as my software and books. For example, I started to design TestWise IDE to support Watir in 2006. and TestWise v2 to support Selenium WebDriver in 2011. These can be counted as solid proof, I believe.
1. Watir vs HTMLUnit vs QTP ( and other record-n-playback tools)
Status: Verified. Watir is still alive. There are still buyers every once in a while for my “Watir Recipes” (1st edition in 2012).
Reasoning: Ruby is a good scripting language, perfect for test automation. I was willing to learn it, and I am grateful that I did.
Proof: The test script examples in my two books “Practical Web Test Automation” 1st edition and “Watir Recipes” were in Watir.
2. Selenium v1 (JS) vs Selenium WebDriver
Status: Verified
Base: I tried Selenium v1 (JavaScript), but I did not like it (JavaScript and its limitation).
Proof: I have been using Watir, until Selenium2, aka. Selenium WebDriver, came out in 2009. I updated “Practical Web Test Automation” 2nd edition to use Selenium WebDriver as examples, and released 5 Selenium Recipes books. But I did not write any books or articles on Selenium v1.
3. PhantomJS vs Nothing, then Chrome v58
Status: Verified
Reasoning: Between 2012–2016, despite that many test architects and engineers talked about “headless testing with PhantomJS”, I found it unusable: slow and test execution was unreliable.
Proof: PhantomJS was deprecated in 2017, a year after I wrote down my doubts about PhantomJS in the Selenium WebDriver Recipes book.
4. HTML5 vs Flex & Silverlight
Status: Verified
Many years ago, there were hypes of Adobe Flex offering very dynamic websites. An example was that Microsoft released its competitor: Silverlight. I reviewed both technologies and decided not to spend time on them. I focused on Selenium (with HTML) instead.
Proof: Both Adobe Flex and Microsoft Silverlight were dead.
5. CT Server vs CI Server (such as Hudson/Jenkins, Bamboo, TeamCity)
Status: self-verified, but not yet widely accepted
Base: CI servers, such as Jenkins/Bamboo/TeamCity, are fine for executing unit tests. I am talking about executing automated UI tests here, which is far more challenging. CI servers simply lack the necessary features to execute UI tests.
Proof: So far I haven’t seen a single successful CT with Jenkins / Bamboo / TeamCity. While I have been doing CT well which enables the release of my web apps to production on a daily basis with my own BuildWise (won the 10th Fukuoka Ruby Award in Japan in 2018). Some of the BuildWise features are similar to the Sandcastle, the internally developed and used CI/CT server at Facebook (see “Continuous Integration at Facebook”).
Software Development Trend ⇒ Continuous Testing
6. Selenium WebDriver vs Cypress, TestCafe, …,
Status: Selenium WebDriver is verified for sure, but the failure of Cypress-type frameworks is yet to be widely accepted. I noticed more noises against Cypress this year.
Proof: “Microsoft Verified”.
No concrete proof to show yet, except seeing every Cypress test automation failed. Selenium WebDriver 4, according to its creator Simon Stewart, would introduce a feature (which Cypress uses a non-standard protocol) that will make people want to forget Cypress happily.
Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)
Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
7. BDD: RSpec vs Gherkin (Cucumber, SpecFlow, …)
Status: self-verified, proven in many of my clients.
Proof: Despite the hype, the adoption of Gherkin remained static, as showed in the “STATE OF TESTING Report 2021” (even dropped from 20% to 17% last year). My article “Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?” was well-received, and featured in Medium’s largest publication: Start it up.
8. Testing Tool: Independent Tool to support Selenium such as TestWise vs Vendor tool mixed framework including RFT, QTP, Ranorex, Sahi Pro, and Coded UI
Status: self-and-expert verified and “Microsoft Verified”
Reasoning: The test framework shall be free and open-source.
Status: TestWise is doing well with faster and better support for more scripting languages. While RFT is pretty much dead, QTP was sold to Micro Focus and is now a selenium level sponsor of Selenium WebDriver. All projects I knew that used Ranorex and Sahi failed as well, I rescued a couple of them by using Selenium WebDriver. Not only was it faster and more stable, but also there were no more paying license fees.
9. Load Testing: Protocol-based vs Browser-based
Status: self-and-expert verified and “Microsoft Verified”
Status: Many traditional protocol-based load testing tools failed in some high-profile projects. A well-known example in Australia is “Embarrassing Collapses of Census Website due to Poor Load Testing”. At the same time, we have seen an emerging new type of load testing using real browsers.
Proof: I did not realize this until 2013, though still well ahead of the industry. I created a load testing tool: LoadWise. However, I deprecated it in 2014 as it was not suitable for testing dynamic web apps (using AJAX for example). Though LoadWise was still commercially feasible (with interests) at the time, I shelved it.
10. API Testing: SoapUI vs HTTP Request-based with a scripting language
Status: self-verified, proven in many of my clients.
Status: Every single SoapUI project I have seen was complex and in a most execution-failed state. The reason is simple, SoapUI is not flexible. for example, what is the date after tomorrow's date? (in Ruby: Date.today.advance(days: 2) . There are also many other drawbacks such as version control issues (comparing test scripts in a code/testing IDE), difficulty running in a CT server,… etc. Due to these setbacks, the SOAP UI tests will not be maintainable after a while.
Proof: I have always been using raw Ruby for API (Soap/REST/…) tests at very high efficiency. Another great benefit of using a scripting language is that we could achieve the synergy of API tests and UI (Selenium) tests in the same test project. Once, a tester showed me a test in ReadyAPI (a more expensive version of SoapUI), I asked her how long it took her to develop in ReadyAPI. Her answer was “4–5 hours”. I informed her that I could get it done in under 20 minutes by using Ruby, and in a format that she would understand. To her surprise, we did it.
For more on API testing, please check my book “API Testing Recipes in Ruby”.
11. Desktop App Testing: Appium with WinAppDriver vs QTP, TestComplete
Status: self-verified and “Microsoft Verified”
Base: I have never believed that the tools like QTP/TestComplete can do real test automation. When I developed TestWise (a desktop app), I used open-source AutoIT3 to do test automation. It is useable with limitations, far from ideal. But I stayed away from commercial tools such as QTP. Eventually, my waiting paid off. With the release of WinAppDriver in 2019, I started to write Desktop End-2-End testing in RSpec to test TestWise in TestWise. Currently, in TestWise end-2-end regression test suite, there are 272 Appium tests.
Proof: Verified by Microsoft (2018). The company shall know more than other companies about the desktop app in Windows.