Another Web Test Automation Milestone: 400,000+ test executions over 7 years for 583 Automated End-to-End (UI) Selenium Regression Suite
Real E2E (UI) Test Automation and Continuous Delivery are never about feasibility, but rather, technical capability.
On November 6, 2017, I published a post on LinkedIn: “500,000 Selenium UI test executions for ClinicWise in BuildWise CI server”.
ClinicWise underwent active development from 2013 to 2017, resulting in numerous regression testing cycles during that period.
Starting in 2018, my web development efforts gradually shifted to WhenWise, a customer-centric booking application tailored for a variety of businesses, including dental practices, yoga studios, tutoring services, driving schools, tennis court hires, and more.
Today, on my BuildWise CT server, the number of E2E (UI) test executions for the WhenWise E2E regression suite reached 400K, over the past seven years.
The screenshot below shows:
“The total number of Automated End-to-end (UI) tests?”
583. The number changes over time, of course. Apart from dozens of API tests, all user-story-level E2E UI tests. The below video shows starting a test execution in TestWise IDE. (By the way, Cypress is unable to perform this test, due to one of its several limitations).
“How long does the regression testing take to run?”
33 minutes with 6 BuildWise Agents.“33 minutes, impossible for 583 E2E UI tests?!”
Running on a single machine would take over 2.6 hours. By utilizing six BuildWise agents (Mac Mini machines), the execution time is reduced by 78.3%.“Even 2.6 hours on a single machine is suspiciously short. Test data management for E2E takes time greatly”
I don’t view test data management as a separate concern. I implemented "The Simpsons" Data Reset Pattern in the WhenWise app, with carefully planned test data seeding. This approach makes test data management straightforward, resulting in very fast test execution.“What are test automation frameworks?”
raw Selenium WebDriver + RSpec (Ruby), I have been using them unchanged since 2011.
“How can you maintain these test scripts (so efficiently)”
See the article, “Maintainable Automated Test Design” and “Page Object Model is universally applicable in web test automation”. Plus, TestWise, the next-generation functional testing IDE, is super helpful.
“How many test steps?”
I wrote a script that does the counting: 31011,A green run, as shown before, means each of every 31K+ test step passes.
One might ask, “Does regression testing with this suite catch defects?” Absolutely.
Take today’s change, for example. I decided to restructure WhenWise's free plan to make it more accessible for casual businesses, allowing them to use it for free and take advantage of new features like Quick-QR-Code-Booking. Readers with a software development background would know this is not a trivial change—one that would likely break something. And it did, 32 test failures on the first regression run after I pushed the code change.
Then, I performed fix-and-then-another regression run cycles. It took five regression runs to get the GREEN. Yes, I did five rounds of comprehensive regression testing today. I pushed the build to production after getting the GREEN on the BuidWise CT server. Yes, I have been conducting daily production releases for over a decade.
Five months ago, I published an article, titled “Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for Large/Complex Apps”, to demonstrate that true end-to-end (UI) test automation is achievable. The test suite was just the WhenWise one.
At that time (as of 2023–08–10):
Number of Test case executions: 359,158
Test case count: 569
Test Steps: 29,845
As astute readers have noticed, the test suite has expanded. It’s not just an increase in size—there have also been changes, naturally evolving alongside the application updates. Maintenance is the key to E2E test automation, many software companies fail E2E test automation because lacking the capability to maintain the test scripts. My advice: seek help from a mentor.
I have consistently used the same technology stack for web, desktop and mobile test automation, developing/refinding/debugging Selenium/Appium with RSpec in the TestWise IDE and running tests on the international award-winning BuildWise CT server. By the way, you can start using the same tech stack for your work or own side-hustle projects, for free.
Below is an article I previously published, showcasing the execution of over 300 Appium + WinAppDriver end-to-end UI tests for validating TestWise, a Windows native application.
See my daughter’s articles on Mobile Test Automation.
Related reading:
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps realWhy Software Testing is Not Effective in Most Software Teams?
Facebook and I Shared a Similar Approach to E2E Test Automation and Continuous Testing
Benefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
What is my Agile Way? Production Deployment within the first week, then every working day
How do I OverCome the E2E Test Automation Challenges Posted in DHH’s Article?