Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part B: Significant Cost Savings
No.1 reason for small and medium software business: save big money!
In this article series:
Part A: Introduction
Part B: Significant Cost Savings
Part C: Superior Performance
Part D: Higher Reliability *
Part E: Greater Flexibility
Part F: Easier Implementation Than Most Thought (upcoming)
Part G: Security Concerns (upcoming)
Part H: Advice and FAQ (upcoming)
A common excuse many software companies use for not implementing end-to-end test automation (mandatory for real Agile/DevOps) is a lack of funding. Twenty years ago, a license for automation tools like HP QuickTest Pro or IBM Rational Functional Tester cost over $10,000. Today, however, the most popular test automation frameworks are free and open-source. For example, in the realm of web testing, the top two frameworks—Selenium WebDriver and Playwright—are both free.
Why didn’t I mention the declining #3, Cypress? Some readers may know that although Cypress claims to be open-source, it imposes restrictions that effectively push users toward their commercial test execution service: Cypress Cloud.
As you can see, the cost has shifted from developing test scripts to executing them, in terms of monetary expense (excluding human time).
Therefore, cost is a major determining factor in deciding between an On-Premises Continuous Testing Lab and a Cloud-based solution. My advice is On-Premises for the reasons below.
1. Cypress Cloud’s Demise: Primarily Due to Its High Cost
Let’s start with a fact: Cypress, whether you like it or not, has made a significant impact on test automation and is well-known. However, its Cypress Cloud service is struggling, and the company is facing difficulties, including being unable to secure Series-C funding and laying off a considerable a number of staff (see my article: 'Cypress.io is Dying').
It’s disappointing to see a supposedly 'open-source' project impose restrictions on test execution — a clear sign of its decline. The reality is, customers aren’t willing to pay for Cypress Cloud (besides, it is no good anyway), its business plan starts with $3199 per year.
2. Cloud option cost goes up quickly with more test executions, extras.
Commonly, the advertised price for cloud-based services is merely a starting point. For real Continuous Testing—running full regression suites frequently—the actual costs can rise quickly. For instance, Cypress’s $3,199/year plan includes 120,000 test results, which breaks down to just 328 per day. Beyond that, it's $5 per additional 1,000 results. Furthermore, there is a long list of limits and extras.
Let me illustrate with a real example of Continuous Testing. On 2025-03-04, the test execution results for my WhenWise project (~600 Selenium tests) on BuildWise exceeded 4,000. (Yes, just one day). This is an End-to-end UI test suite. For API tests, the result count will be a lot bigger.
This is just one of several apps developed by a solopreneur, me. If I were running a small software company (which I have no interest in—I much prefer being a solopreneur), reaching 20,000 daily test results would be quite common. At that scale, using the Cypress Cloud option would cost me at least $100 per day, which adds up to over $30,000 a year.
I set up my on-premise Continuous Testing Lab (with BuildWise CT Server) back in 2012, starting with just one existing iMac and two Mac Minis. Since then, I’ve only added four Mac Minis (each under A$1,000 = ~US$650), in total, I've spent ~US$2,000 on CT Lab hardware in the past 13 years. As a comparison, it would be US$500K with Cypress Cloud (of course, I wouldn’t be that silly to choose Cypress, here, just a comparison in cost). What a difference!
3. Cloud-based was an easier option, but not any more.
I have experience migrating on-premise app servers to the cloud over two decades ago, so I’m well aware of all the touted benefits of going cloud (one the them is cost-saving). However, with advances in hardware and infrastructure-as-a-service technologies, there's now a growing trend of moving away from the cloud to significantly cut costs. DHH’s well-known post on this topic—complete with detailed figures—is worth a read.
4. Thanks to advancements in hardware and software, setting up an on-premise Continuous Testing Lab is now both affordable and straightforward.
Here, I focus only on the cost factors:
Mini PCs are cheap. M4 Mac Mini, in particular, is a perfect build machine (for both the agent and the server), only costs $599.
The consumer-grade LAN (even Wifi) route speed is much faster.
Consumer-grade RAM and memory are much faster.
UTM virtualisation software on Mac is free.
Related reading: