Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part D: Higher Reliability *

Highly reliability CT setup saves money and time.

In this article series:

• Part A: Introduction

• Part B: Significant Cost Savings

• Part C: Superior Performance

• Part D: Higher Reliability *

• Part E: Greater Flexibility (upcoming)

• Part F: Easier Implementation Than Most Thought (upcoming)

• Part G: Security Concerns (upcoming)

• Part H: Advice and FAQ (upcoming)

Some advocates of cloud-based solutions might acknowledge the objective benefits of Parts A, B, and C. However, when they come across "High Reliability" as a point, they may respond with, “I don’t see any reliability issues with cloud-based servers.”

Oh well, let’s see whether I can change your view.

Shared-CPU Plan Lacks Reliability

When it comes to a server instance, your manager is usually willing to invest in a premium plan—like a dedicated server—to guarantee reliability.

But will your manager sign off on paying for dedicated server instances for 25 build agents? I highly doubt it. The Shared-CPU (or VM) plan is a common choice.

A shared CPU plan

With a VM plan, dozens of virtual machines share the same physical server, competing for CPU and memory. This means your performance can be affected by other users on the same host. But E2E test execution is especially sensitive to system performance—just consider how even minor slowdowns can cause delays or flakiness (in automated test script execution) when interacting with modern, dynamic websites.

This might not be very noticeable when you only have a small test suite—say, around 20 tests. But when the suite grows to 500+ tests, the additional maintenance overhead caused by inconsistent test results becomes a real pain—trust me, you won’t like it. In fact, many E2E test automation attempts failed because of that.