Case Study: How Much Money Does Real End-to-End Test Automation Save? *

Many seasoned software professionals might say, "The savings could be significant!" But just how significant? This case study will leave you astonished.

While it is widely accepted that test automation, especially end-to-end functional testing, saves money, many software professionals have not realized or experienced this in practice.

I will compare a government testing task with my personal project in this case study to provide valuable insights.

A Government Regression Testing Task

In 2013, I joined a government agency as a contractor Test Automation Engineer. The testing team consists of 10 testers, a mix of permanent employees and contractors. Of course, they have all been doing manual testing.

The system is old, and the test cases have been documented well (after over 2 decades). In other words, we know exactly how many tests to conduct in one round of full regression testing. The magic number was set to 3000, and, according to the test plan, the team has one month (30 days) to complete it.

In other words, calculating the cost of a full regression test is fairly straightforward. With a team of 11 members (including the test manager) and an average salary of ~A$80,000 (my guess) per person, the total cost for a full round of regression testing comes to approximately

Money Cost: A$75,000

Time Cost: 30 days

My Recent Three Rounds of Regression Testing for WhenWise Completed in 3 Hours

Now let’s see the cost of a full round of WhenWise (my app) automated E2E regression testing, which I conducted three times yesterday morning (2024-11-13).

The whole process took 3 hours exactly (7 AM to 10 AM), including fixing a few minor issues and stabilizing test scripts, and a deployment to production.

The final one (100% pass of 573 Selenium WebDriver + RSpec tests, with over 30,000+ test steps) only took 29 minutes.

Comparison

Assuming I am working as a principal engineer at a company with a rate of A$200 per hour, a full round of regression testing for WhenWise—a complex and comprehensive app—would only cost A$100.

BTW: you might find WhenWise useful to book a piano lesson, tennis court, physio, yoga class, …, etc.

Let’s factor the test count difference: 3000 / 573 = 5.24.

A$524 vs A$75,000, What a big difference!