WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions
Share WhenWise’s CT process, which enables daily deployment to production.
Today (2021–03–05), I reached Level 4 of AgileWay Continuous Testing Grading for my WhenWise app: 500 end-to-end/user-story-level (Selenium WebDriver) tests in the regression test suite.
This is my second time reaching Level 4. The first time was for ClinicWise on 2016–07–29.
Parallel Test Execution
A big challenge of executing automated end-to-end tests (in browsers) is the long feedback, as running many UI tests will take a long time. In the case of the WhenWise regression test suite, it will take 4+ hours if running on a single machine. The only practical solution is parallel execution.
Running this suite in 9 BuildWise agents (concurrently) will only take ~35 minutes. Besides time-saving,
Test execution is much more stable via Auto-Retries
Most of the 12 false alarms (shown in the build report, top-right) were due to the chromedriver v87 not being compatible with self-updated Chrome v89 on one build agent (#11 Win10). BuildWise auto-retried the failed tests on another agent and passed!
Related readings: “Why Auto-Retry of Test Execution in a Test Framework is Wrong?”
Cross-platform & Cross-browser Testing
From the above screenshot, you can see that the tests were on different platforms: Windows, macOS, and Linux.
For some other projects, I configured agents with different browsers for cross-browser testing. For WhenWise, I just used Chrome.
Total test case executions: Nearly 300,000.
No Defect Tracking at all
WhenWise is developed and maintained in my spare time, along with other apps such as TestWise, BuildWise, SupportWise, and ClinicWise. With Continuous Testing, I don’t need to do defect tracking, not at all.
Test Script Framework
Raw Selenium WebDriver (Automation) + RSpec (syntax), no fancy ‘own framework’. I don’t use test recorders, not at all (despite that I created one myself).
Related readings:
“Please, not another Web Test Automation Framework, just use Selenium WebDriver”
“Step by Step Showing Why Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework”
Testing and CI tools
The test scripts are pure selenium tests in plain text, they will work with other tools. Tools are for efficiency, and I am very efficient with the help of these two tools (disclaimer: I created them. You may use these tools in free mode).
Metrics?
None at all. I only care whether the build (to run the whole regression test suite) passes or not. If not, №1 priority is to resolve the issues. If there are changes made, I will deploy the new build to production on that day, upon getting a green build on the BuildWise CT server. There are only a handful of cases that I could not get a green build on the day. Then I will work on those first thing the next morning.
Sample Test
Below is one of 500 WhenWise tests: “A Business signs up with validation”.
The test scripts follow the “Maintainable Automated Test Design”, using Reusable Helper Function and Page Object Model.
The style of automated test scripts has been verified in the ‘battlefields’. I have been writing thousands of automated tests in this style since 2006 for clients and my own projects. When working in the consulting role, I was always able to implement a few key automated tests and run them in BuildWise on the first day. So don’t believe a so-called fancy ‘new test tool/framework’ (such as Cypress, see the reasons), raw Selenium WebDriver has been, is, and will still be (in the foreseeable future) the best (and the easiest to learn, don’t believe fake testers’ labeling ‘steep learning curve’) test automation framework for web apps.
For more ideas on test script design, refer to my “Practical Web Test Automation” book. Here I just point out two attributes:
Business analysts/customers can understand the test scripts, after a quick explanation
One test step (by line) for each user operation, is important for debugging and efficient maintenance.
Test Script Stats
+------------+---------+---------+---------+--------+
| TEST | LINES | SUITES | CASES | LOC |
| | 23365 | 304 | 510 | 18381 |
+------------+---------+---------+---------+--------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 9036 | 159 | 1483 | 6862 |
+------------+---------+---------+---------+--------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 801 | 5 | 61 | 618 |
+------------+---------+---------+---------+--------+
| TOTAL | 33202 | | | 25861 |
+------------+---------+---------+---------+--------+Page class count: 159. The total number of test steps: 25,861
A green build means each of the 25K+ (some repeated execution of one step) test step pass!
Build History
The CT process detected many issues/defects, then I fixed them very quickly. (Again, I never used a Defect Tracking System for my own software development). I cannot emphasize the importance of this CT process for my app development. (note: the last 6 builds took a shorter time because I added 3 more BuildWise agents)
If you are interested in my approach and prefer finding out more in a structured way, check out my books:
“Practical Web Test Automation — Test web applications wisely with Selenium WebDriver”
“Selenium WebDriver Recipes in Ruby — The problem-solving guide to Selenium WebDriver in Ruby”
“Practical Continuous Testing, make Agile/DevOps real”
Related reading: