What does Real Continuous Testing Result History Look Like? *

Identify Real CI/CD or Continuous Testing Quickly.

This full article is exclusively available to Premium Subscribers of this newsletter.

Years ago, a CIO of a large tech firm, fooled by her chief agile coach, proudly announced during a town hall meeting that the team had successfully implemented automated end-to-end (E2E) testing for their main project. She enthusiastically shared that 980 automated E2E tests were executed nightly with a flawless 100% pass rate. Listening to this, I turned to the testing lead beside me and remarked, "That can't possibly be true." (This assessment is based on my observations of the company's self-labelled agile practices.)

It didn’t take long for my prediction to be validated. Against my advice to let it slide, the testing lead decided to investigate further. It was soon uncovered that the so-called chief agile coach, together with a group of fake agile coaches and faux DevOps engineers, had rigged the system. They had configured the TeamCity (CI server) to always display a “Success” status 😱, regardless of the actual test outcomes. In reality, over 700 tests were failing every single night.

If that poor CIO had read this article, such a humiliating incident might have been avoided. Real Continuous Testing is extremely rare; however, identifying what is not real is quite simple.

Below is a real one, a test execution history of my WhenWise app’s E2E test suite, consisting of 577 user-story-level Selenium tests, in the award-winning BuildWise CT server.

Let me quickly explain what the two colours mean in the above chart:

• GREEN

  All tests pass.

• RED
  Any test failed.

Below are my four assessment factors to determine whether Continuous Testing is truly implemented.

1. Execution History Cannot be All Green

2. Execution History Cannot be All Red

3. Only a short gap between the next Green and the Red

4. Too Few E2E Tests

1. Execution History Cannot be All Green