Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part A: Introduction

Setting up your own on-premises reliability and performance continuous testing lab is much easier, more flexible, and more cost-effective than most people think.

Image credit: https://x.com/hehehenrihenri/status/1913161609507205549. Please note: the above setup is for AI, the actual CT labs for most software companies require much much fewer computers.

In this article series:

• Part A: Introduction

• Part B: Significant Cost Savings

• Part C: Superior Performance

• Part D: Higher Reliability (upcoming)

• Part E: Greater Flexibility (upcoming)

• Part F: Easier Implementation Than Most Thought (upcoming)

• Part G: Security Concerns (upcoming)

• Part H: Advice and FAQ (upcoming)

Recently, I upgraded my Continuous Testing Lab, which now includes the M4 Mac Mini.

One reader’s comments (on LinkedIn):

Before diving into this topic, I want to clarify the distinction between Cloud CT providers and the two types of on-premises setups:

• Cloud CT Provider: The infrastructure is fully managed by a commercial service provider. Example: Cypress Cloud, SmartBear LoadNinja. You have little control or visibility of the infrastructure.

• Physical On-Premises: The machines are physically located within your organization.

• Cloud On-Premises: Instances hosted on platforms like AWS or Azure, but integrated into your private network.

There are numerous advantages to choosing an on-premises setup:

• Significant Cost Savings (upcoming)

• Superior Performance

• Higher Reliability

• Greater Flexibility

• Easier Implementation Than Most Thought

• Security Concerns

The Continuous Testing Lab is not exclusive to the world’s top tech giants, like Facebook’s.

Image source: “Continuous Integration at Facebook” presentation

By the way, Facebook started with a simple rack too, in 2012.

Image source: “Continuous Integration at Facebook” presentation

The Continuous Testing Lab I am referring here applies to most software companies, from one-person start-ups (like myself) to large tech companies.

But first, let me clarify my approach to CI/CD and Continuous Testing, which may differ from conventional perspectives.

I frequently run a large number of user-story-level automated end-to-end (UI) tests as part of regression testing. For instance,

• ClinicWise: 610 Selenium WebDriver tests

• TestWise: 330 Appium + WinAppDriver tests

• WhenWise: 577 Selenium WebDriver tests.

A recent run of WhenWise Regression Testing in Buildwise CT Server

Since 2012, I’ve adopted the practice of “daily production releases” for all my applications. If the E2E test suite passes successfully on my Continuous Testing (CT) server, I push the new build directly to production.

I’ve observed that many so-called “CI/CD” pipelines only execute a small suite of unit tests (white-box testing), with some even skipping integration tests (still white-box). However, without comprehensive end-to-end testing, there is no true Continuous Delivery, because delivery means deploying to production.

If someone wants to debate 'On-Premises' versus 'On-Cloud' for Continuous Testing, I'd first recommend evaluating the testing volume—looking at factors like test types, number of test cases, execution time, and frequency. Below are my factors:

• 500+ user story-level automated end-to-end (UI) tests — designed to be understandable and executable by end users
  (Tech stack: raw Selenium WebDriver + RSpec)

• Several full regression test runs on an active development day.

• Parallel execution in BuildWise reduces total regression testing time to under one hour.

• Immediate deployment to production upon successful E2E test suite completion.

So, our definitions of 'CI/CD' or Continuous Testing might be quite different.

---

The Resources Zhimin Created to Help Achieve Real End-to-End Test Automation & Continuous Testing.