Why I Prefer an On-Premises Continuous Testing Lab Over the Cloud? – Part C: Superior Performance

Performance matters a lot to executing Automated End-to-End tests, another strong reason to go for on-premise.

In this article series:

• Part A: Introduction

• Part B: Significant Cost Savings

• Part C: Superior Performance

• Part D: Higher Reliability (upcoming)

• Part E: Greater Flexibility

• Part F: Easier Implementation Than Most Thought

• Part G: Security Concerns

• Part H: Advice and FAQ

The speed of E2E test automation is crucial in Continuous Testing, as it directly impacts efficiency. However, some people mistakenly associate Continuous Testing (CT) only with the execution of unit tests during Continuous Integration (CI). To clarify, here’s a rough, illustrative formula (though not precise) that highlights the difference in typical test execution times: go up ~20x factor each level.

Unit Test:  0.01 second

Integration Test: 0.5 second

End-to-End Test: 20 second 

Imagine your end-to-end (E2E) test suite consisting of 200 tests. Running the entire suite on a single machine would take approximately 1.5 hours. Achieving a flawless run for all 200 tests (involving 6,000+ test steps) is no small feat—any single test error or failure often necessitates rerunning the entire suite.

In this article, I’ll focus specifically on one critical factor: the overall test execution time. In other words, we want test execution to be as fast as possible.