Why Is a Continuous Testing Server Rarely Seen? *

Shouldn't every team that claims to be Agile or DevOps have one? Yes, in theory. However, human factors …

DevOps is one of the hottest software buzzwords. You must have seen the following drawing (with slightly different variations).

image credit: https://www.uffizzi.com/blog/continuous-testing-in-devops

Test automation is, of course, the foundation of DevOps, commonly referred to as Continuous Testing. The term “Ops” implies that automated tests are E2E tests. Unit and Integration testing are considered a part of coding. TDD, right?

Then, there comes a problem. How do we run automated E2E tests in a continuous process?

The CI servers, such as Jenkins, Bamboo, TeamCity, and Travis, are not good answers. Why? They are built for executing unit and integration tests, i.e., white box tests. Executing a suite of E2E (via UI) tests is a completely different game.

To qualify as a Continuous Testing Server, it must effectively address the following characteristics of E2E Testing.

• Brittle

• Long feedback time

Please note that a CI server does not need to worry about the above unless unit testing is done wrongly. (see Unit Testing Clarified)

CI servers can be made to execute E2E Tests, but not well

Technically, most CI vendors could implement features that support executing automated E2E tests. I even successfully implemented one (customization of CruiseControl) back in 2008. But, those CI tool companies wouldn’t do it, for two reasons.

1. They Lack End-to-end Test Automation Knowledge themselves.

Here, I just show a quote (more in this article).