Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part E: Greater Flexibility

How on-prem CT labs offer more control and flexibility for automated E2E testing.

In this article series:

• Part A: Introduction

• Part B: Significant Cost Savings

• Part C: Superior Performance

• Part D: Higher Reliability *

• Part E: Greater Flexibility

• Part F: Easier Implementation Than Most Thought (upcoming)

• Part G: Security Concerns (upcoming)

• Part H: Advice and FAQ (upcoming)

Different from unit testing, E2E tests often require dependent software (at specific versions) and third-party services, such as

• browser driver matches the Chrome browser

• SMTP server

Readers who have only used specific CI tools (like Travis) and no-good CT platforms (such as Cypress Cloud) often don’t realize how limited those tools are—until they experience the power of the international award-winning BuildWise Continuous Testing Server, which is both free and open-source.

Note: I'm referring here to real end-to-end (UI) test automation, real Agile, and real DevOps — the kind that empowers software teams to deliver high-quality releases to production daily. (See my article: Definition of End-to-End Test Automation Success.)

The flexibility of on-premises (or in-house) systems is certainly greater than that of cloud-based ones — we all know that familiar “at home” feeling.

So, what specific flexibility does an On-Premises Continuous Testing Lab offer? (By the way, fake test automation engineers fear specifics.)

1. Supports many testing types

More often than not, test automation engineers perform more than one type of E2E testing, such as:

• Web Test Automation

• Desktop Test Automation

• Mobile Test Automation

• API Testing

• Performance Testing

• Load Testing

In such cases, it can be bogged down by red tape or become costly to get your commercial continuous testing vendor to accommodate requests like, “I want to run a small suite of mobile tests against iOS version X”.

2. Supports multiple test automation frameworks in different languages.

With the same BuildWise CT server, I have a number of build projects for the following automation framework + language combinations:

• Selenium WebDriver + RSpec (Ruby),

• Selenium WebDriver + PyTest (Python),

• Selenium WebDriver + Cucumber (Ruby),

• Selenium WebDriver + JUnit (Java),

• Selenium WebDriver + MSTest (C#),

• Appium + RSpec with iOS or Android (Ruby)

• Appium + RSpec with Windows Desktop Driver (Ruby)

• Playwright Test (TypeScript)

• …

3. Maximum flexibility

As mentioned earlier, E2E testing relies on supporting infrastructure, such as:

• MailCatcher: Testing Emails in Automated Test Scripts with a Fake SMTP server

• Certificates: Using a locally trusted certificate can reduce costs.

• A specific browser version (with auto-updates disabled) paired with a specific matching driver version.

With on-premises set up, you can install and configure infrastructure easily.

With a cloud-based CT vendor, you may become locked into specific technologies—such as Docker containers (which are showing signs of decline). Any deviation from the default setup can be costly and often unreliable.

---

Some readers might say, “Sure, an On-Premises Continuous Testing Lab offers more flexibility, but…” Let me stop you right there—there are no buts here. As covered in other parts of this article series, on-premises CT delivers significant cost savings, stronger security, greater reliability, and far superior performance.

---

Related reading

• My eBook:
  - Practical Web Test Automation with Selenium WebDriver
  - Practical Continuous Testing: make Agile/DevOps real
  - Practical Performance and Load Testing

• My Universal, High-Efficient and Free Approach to API Testing

• Set up, Develop Automated UI tests and Run them in a CT server on your First day at work

The Resources Zhimin Created to Help Achieve Real End-to-End Test Automation & Continuous Testing