Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part F: Easier Implementation Than Most Thought
If using the right toolset.
In this article series:
Part A: Introduction
Part B: Significant Cost Savings
Part C: Superior Performance
Part D: Higher Reliability *
Part E: Greater Flexibility
Part G: Security Concerns (upcoming)
Part H: Advice and FAQ (upcoming)
With all the clear benefits of an on-premise Continuous Testing Lab (see Part A-E), some might think, “This sounds too good to be true—it must be difficult to set up.” Not at all. In fact, it’s surprisingly easy. My daughter set one up in just a single morning during her internship.
In my consulting work, I typically get a few key business tests automated and an on-premise testing lab up and running a small suite of automated E2E tests on Day 1. Some might doubt this, saying, “Impossible! Getting lab machines in a company usually takes days—often weeks.” I did it on my computer using virtual machines. For the parallel part (three build agents), I simply used the free Oracle VirtualBox.
Of course, the choice of Continuous Testing (CT) server determines the setup. I’ve been using BuildWise—the CT server I developed, which won a major international award.
Unlike other CT solutions, BuildWise follows a radically simple, effective philosophy that gives users maximum control. It avoids features like workers, Docker, sharding, and framework-level retries—things that look good in demos but often complicate real-world usage (and fail). The BuildWise approach is straightforward yet powerful: the BuildWise server assigns automated tests to multiple build agents for execution—a model so clear that even end users can understand it.
A Continuous Testing Lab is made up of the following key components:
On-premise Hardware
Required Software – for example, a Chrome browser
Continuous Testing Server: BuildWise (free and open-source)
Continuous Testing Agents: BuildWise Agents
(a free mode can be used for commercial purpose)
Let me explain each of these components one by one.
1. On-premise Hardware
I prefer using Mac Minis (for both the server and agents)—it's powerful, quiet, reliable, and very affordable ($599).
At work, you'll likely begin with a proof of concept, using whatever resources you have available. You can set everything up on your own computer—even a laptop—using Oracle VirtualBox (since 2025-03, VMWare is free too).
2. Required Software
End-to-end testing replicates real user interactions with your application. Your tech support usually have a quick process to prepare a clean machine (or VM) with all network access and the essential software, e.g. Chrome Browser.
After that, as a test automation engineer, you just need to install a few extra tools to run the automated tests.
With my preferred tech stack for Web Test Automation (raw Selenium WebDriver + RSpec), I usually just need to run a single command on the server and agent machines.
gem install --no-document rspec selenium-webdriver3. Continuous Testing Server: BuildWise
Setting up the BuildWise server is incredibly simple. The installation package is just 1.4MB, and the only requirement is Ruby—which is already included on macOS. With that, installation is just a single command.
Next, you configure it for your project. BuildWise follows a unique philosophy that sets it apart from most CI/CD servers. Instead of enforcing a rigid structure, it gives users maximum control through a single build script file (Rakefile).
This design choice may not make commercial sense—after all, it prohibits locking users into proprietary configurations or creating unnecessary dependencies on the tool itself. But for software engineers, that freedom is a major advantage.
Besides, AgileWay (the company behind BuildWise) provides examples (on Github) for various automation framework configurations, which mostly are directly useable.
Furthermore, BuildWise server comes with a dozen of same demo project setup.
Check out my daughter’s article: “Set up BuildWise CT Server Step by Step”.
4. Continuous Testing Agent: BuildWise Agent
A CT testing lab consists of multiple build agents. A agent machine is set up as an end-user machine. Then, just install two pieces of software:
Test automation library, e.g. Selenium WebDriver, Playwright or Appium
BuildWise Agent
BuildWise Agent is a lightweight desktop application (available for macOS, Windows, and Linux) that executes test scripts assigned by the server.
You can use multiple BuildWise Agents in free mode, fully featured, with just one limitation: each session can last up to two minutes (similar to Zoom’s model).
Related reading:
