Continuous Testing Clarified

Continuous Testing is the most important process in DevOps; Yet most so-called ‘Agile’ or ‘DevOps’ projects did not have it or got it wrong.

A repost of my past article published on Medium in 2021

According to Wikipedia, continuous testing is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” The keywords are “automated tests”, “delivery pipeline”, “immediate feedback” and “software release candidate”. (The reason I quote Wikipedia, a non-academic reference, for definitions: because Continuous Testing is new, and there is a lot of confusion over it. So I reach for a common understanding at Wikipedia)

Table of Contents:
· Continuous Testing explained in simple words
· Continuous Testing is the Holy Grail of Software Development
· Software Development Trend ⇒ Continuous Testing
· Continuous Testing vs DevOps
· Continuous Testing vs Continuous Integration
· If CI is implemented properly, no need for CD or CT
· Continuous Delivery is really about testing
· Reality Check of CT/DevOps

Continuous Testing explained in simple words

Like many other formal technology definitions, the above (on Wikipedia) sounds right but is not quite clear. Let me interpret this:

• “executing automated tests”, “business risks”
  The automated tests are at the user story level: testing business features. For example, for a web app, the CT process will run a set of automated test scripts to drive the app to verify business functions, in a browser.
  
  Comparatively, unit testing is conducted by programmers only, that’s why unit tests are also called programmer tests.

• “pipeline”, “immediate feedback”
  In this pipeline, Customers/Business Analysts and programmers are waiting for feedback, and more importantly, ready to act on the feedback. Modification follows feedback, i.e., if there are test failures, a new build (with potential fixes) will be triggered to run another execution of automated tests to ensure the quality.

• “software release candidate”
  The software is in a ready-to-release state if it has passed all automated functional tests, with little or no human activities in pushing the latest release to production.

Here is my interpretation of Continuous Testing: “Run automated end-to-end (UI) as regression testing, frequently on new builds. If all tests pass, the software is ready for a production release. If there are test failures, the team must act quickly on the feedback.”

For the benefits of CT, please check out this article series.

Continuous Testing is the Holy Grail of Software Development

I didn’t invent using ‘Holy Grail’ to describe CT (as much as I would like to take credit for, I heard it from a conversation years ago, but pitifully, could not remember the source), but I think it is a perfect metaphor for Continuous Testing. Yes, it is a big claim, and many of you probably are doubtful. I hope to convince you with the Practical Continuous Testing book.

Here I share one customer’s comments after I helped implement the continuous testing process, which enabled the project to push updates to the production server on a daily basis. This product owner agrees: “Continuous testing is super valuable and super rare, despite many heard of it, but very few saw it, just like the Holy Grail”.

Prior to practising test automation and CT, I have worked as a senior software engineer (Java/C#) for a number of years. I did not create a single my own product in spare time. Now have a handful highly-acclaimed software products, one even won an international programming award. Check out this article: Reflections of Software I Created over the Last 14 Years in My Spare Time,

When someone asked my secret, My answer: “I found the holy grail (of software development), that is Continuous Testing”.

Software Development Trend ⇒ Continuous Testing

Please read this article for details.

Continuous Testing vs DevOps

“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.” — Predicts 2017: Gartner Report