My Core Test Automation Stack Largely Unchanged for 17 years
If our testing target (Web and Mobile) has not changed for decades, why should QA Engineers change (for the sake of change)?
A repost of my past article on Medium
I embarked on my test automation journey in 2005; as of 2024, it has been nearly two decades. I don’t mean to boast, but I’ve accomplished quite a bit in the realm of end-to-end test automation:
Authored 14 books on E2E Test Automation and Continuous Testing
Invited to speak at international software testing conferences, including the premier StarWest.
Implemented successful E2E test automation (web, API, desktop) in a number of software projects. Rescued some too.
TestWise IDE I created received acknowledgements worldwide.
My BuildWise CT Server won the 2018 Ruby International Award in Japan.
I developed (and have been maintaining) several highly acclaimed apps in my spare time, thanks to my E2E Test Automation and Continuous Testing process.
Regarded as one of the world’s top bloggers on software testing.
Table of Contents:
· 1. Pre-WebDriver (2005–2010)
· 2. Web Test Automation Stack
· 3. Desktop Test Automation Stack
· 4. API Testing
· 5. Performance Testing
· 6. Load Testing
· 7. Mobile Test Automation
· 8. I am more open-minded than most.
· Summary
1. Pre-WebDriver (2005–2010)
I started with JWebUnit (Java), then quickly switched to real E2E (in browser) testing with Watir.
Tech. Stack (Pre-WebDriver):
Automation Framework: Watir
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: Cruise Control (with customization)
2. Web Test Automation Stack
Tech. Stack:
Automation Framework: raw Selenium WebDriver
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise
Example: Selenium Suite for WhenWise web app
368K test executions for ~6 years.
Example: Selenium Suite for ClinicWise web app
I have another even bigger suite for the ClinicWise app, with 611 selenium tests (and 618,327 test executions). Below is an execution history report for ~8 years.
For more, check out my articles:
3. Desktop Test Automation Stack
My Desktop Test Automation started in 2009, and I tried autoit3 with limited success. I changed the tech stack after seeing Microsoft deprecated its own Coded UI Tests and recommended using Appium + WinAppDriver for Desktop app automation in 2018.
Tech. Stack:
Automation Framework: raw Appium + WinAppDriver
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise
Example: Appium Suite for TestWise Desktop App
73K+ test executions over 4.5 years.
4. API Testing
While I haven’t explicitly worked with an official job title, API Test Automation Engineer, I have used API testing at most jobs in the form of either API testing or using APIs to assist Web Test Automation.
Tech. Stack:
Automation Framework: N/A. Just use Ruby to invoke HTTP
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise
Example: API Testing Ruby Recipes
For more on my API Testing approach:
My Universal, High-Efficient and Free Approach to API Testing
A Story of 100X API Testing I Implemented in a Matter of Days
My daughter’s API testing series: SOAP Web Services, RESTful Services, GraphQL
4. Performance Testing
Traditionally, performance testing is conducted by a separate team towards the very end of software development. Using the BuildWise CT server, I implemented “Continuous Performance Testing”, which runs performance tests early and frequently.
Tech. Stack:
Automation Framework: Ruby Mechanize or Selenium WebDriver
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise
Example: WhenWise Performance Testing
For more, check out:
5. Load Testing
Like Performance Testing, Load Testing is not my specialty. In 2016, while working as a contract test automation consultant (functional), I solved a key technical load testing challenge that bothered this large tech company’s performance testing team for a year. The solution is simple: Using 10 BuildWise Agents (in VMs, which this company has the infrastructure), and I completed the proof of concept within a couple of days, the task was accomplished the task within weeks.
Since then, I have added more load-testing features in BuildWise and used them with my clients and my own projects with great outcomes. In 2022, I published the book Practical Performance and Load Testing.
Tech. Stack:
Automation Framework: Ruby Mechanize or Selenium WebDriver
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise with Multiple Agents
Example: WhenWise Load Testing
6. Mobile Test Automation
My primary End-to-end test automation is on testing web and desktop apps. I have tried some Appium to drive iOS apps and could put up an impressive demonstration or presentation. However, long-time readers know I don’t consider that real test automation.
It is not me not trying to do mobile test automation. The primary reason is that I don’t have mobile apps, i.e., no practical needs. I did attempt to join mobile app teams (as a contractor), twice. The first one turned out to be a simple wrapper of the web app; therefore, web testing worked fine. The other did not happen due to the impact of the COVID-19 pandemic.
Anyway, I believe my core tech stack also works for Mobile Test Automation, as my daughter is writing a mobile test automation book with this tech stack.
Tech Stack:
Automation Framework: Appium + XCUIDriver/ExpressoDriver
Test Syntax Framework: RSpec
Test Scripting Language: Ruby
Testing IDE: TestWise
Continuous Testing Server: BuildWise
Example: My Daughter’s book “Practical Mobile Test Automation with Appium”
The use of Appium should not be controversial, as it’s the de-facto choice in the mobile automation space. Some readers might think: “With the above test automation types, you provided proof. But without proof of execution history for a sizeable suite, I have doubts”. If you have doubts, stay tuned for my daughter’s book.
7. I am more open-minded than most.
People have different opinions. I don’t expect everyone to agree and adopt my tech stacks. Frankly, I don’t care.
Being objective is a core attribute of a Software Tester. I am more open-minded than most test automation engineers. Again, I say this with proof.
Scripting language
I am probably more objective than most, as My Selenium Recipes Book series covers all of Selenium WebDriver’s five official programming languages.
Cucumber
Many readers know that I am against using Cucumber or any Genkin BDD framework in E2E Test Automation from my articles:
Yet, I have done extensive work to add supporting Cucumber in my TestWise IDE.
My BuildWise CT server supports Cucumber from its first version.
Playwright
TestWise 7 added support for Playwright.
Summary
As you can see, my core tech stack for E2E test automation is unchanged. Moreover, there is a lot of synergy among different types of E2E testing (Web, Desktop, API, and Mobile): all scripting in Ruby using TestWise IDE, running regularly, in a continuous manner, in BuildWise.
Related reading:
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps real
- Practical Performance and Load TestingBenefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
Why Ruby is the Best Scripting Language for End-to-End Test Automation?
Why JavaScript Is Not a Suitable Language for Real Web Test Automation?