Software Development Trend ⇒ Continuous Testing
The trend in software development is Continuous Testing — executing automated end-to-end regression tests in a CT server, multiple times a day. CT simply enables DevOps.
What is Continuous Testing?
Continuous Testing (CT) is the critical component of DevOps, which probably is the hottest terminology in software development in recent years.
So what exactly is Continuous Testing? According to Wikipedia, CT is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” The keywords are “automated tests”, “delivery pipeline”, “immediate feedback”, and “software release candidate”.
The reason I quote Wikipedia, a non-academic reference, for definitions: because Continuous Testing is relatively new and there are a lot of confusions over it. So I reached for the common understanding at Wikipedia.
Like many other formal technology definitions, the above sounds right but not quite clear. Let me interpret this:
“executing automated tests”, “business risks”
The automated tests are at the user story level: testing business features. For example, for a web app, the CT process will run a set of automated test scripts to drive the app to verify business functions, in a browser. Only a comprehensive end-to-end user-story-level regression test suite can ensure quality releases (to production).
Comparatively, unit testing is conducted by programmers only. That’s why unit tests are also called programmer tests.“pipeline”, “immediate feedback”
In this pipeline, Customers/Business Analysts and programmers are waiting for feedback, and more importantly, ready to act on the feedback. Modification follows feedback, i.e., if there are test failures, a new build (with potential fixes) will be triggered to run another execution of all automated tests to ensure no regression issues.
“software release candidate”
The software is in a ready-to-release state if it has passed all automated functional tests, with little or no human activities in pushing the latest release to production.
Here is my interpretation of Continuous Testing: “Run automated end-to-end (UI) as regression testing, frequently on new builds. If all tests pass, the software is ready for a production release. If there are test failures, the team must act quickly on the feedback.”
Continuous Testing is the trend
Let’s examine some hot software terminologies (that teams use every day) over the last two decades.
1997 — JUnit by Kent Beck and Erich Gamma
1999.07 — “Refactoring” book by Martin Fowler, a process of refining code design backed up by comprehensive automated unit testing.
1999.10 — First Agile Book by Kent Beck: “Extreme Programming Explained.”
2000.10 — “Continuous Integration” paper by Martin Fowler
2001.02 — “The Agile Manifesto” was written by Kent Beck and 16 other wise software development practitioners
2002.11 — “Test Driven Development: By Example” book by Kent Beck Introduction
2007.07 — “Continuous Integration” book by Paul Duvall, Steve Matyas, and Andrew Glover
2009.01 — “Agile Testing” book Lisa Crispin and Janet Gregory
2010.08 — “Continuous Delivery” book by Jez Humble and David Farley
∼2014 — Behaviour Driven Development (BDD)
∼2018 — DevOps and Continuous Testing (CT)
Quite clearly, the software development trend is moving towards frequent releases backed up by automated functional testing.
Unit tests ⇒ Functional End-to-End tests
“The first time ever that ‘end-user satisfaction’ is the top objective of quality assurance and software testing strategy” — the first item in World Quality Report 2018–19 Findings
2. Adhoc test execution ⇒ Repeatable process to run all automated tests
3. Programmers only ⇒ The whole team
4. Testing for better code quality ⇒ Testing for overall quality and frequent releases
This movement shall not come as a surprise, as being able to push out software releases frequently with high quality is every software project pursuing. You might have heard of “Quality at speed” and “Ship Early, Ship Often”.
In fact, no matter what development methodology your project adopts, Agile (Scrum, exTreme Programming, Kanban, SAFe, …, etc)or or even WaterFall, Continuous Testing is beneficial.
It is worth noting that the practices above (such as unit testing, refactoring, functional testing, and CI/CD) are complementary, rather than replacements. For example, in a software team that embraces a whole-team-involved continuous testing process, programmers are still encouraged to do automated unit testing with TDD.
My Experience
How Continuous Testing will improve software development? I will share the development of my own apps (as might not be appropriate to disclose my clients’).
Between 1997–2007, I worked as a Java Programmer (contractor, mostly). I did not create a single own software. After I started practising test automation (raw Watir, then raw Selenium WebDriver) and Continuous Testing (customized Cruise Control, then my own BuildWise), I have created and been maintaining 8 software apps, most of them with commercial customers; one finalist and one winner (2nd prize) of an international programming award.
For ClinicWise alone, one architect who worked on a similar system in the UK before, told me “this is the work of a team with 25+ people”. He could not believe I created (and still maintaining) ClinicWise in my spare time. ClinicWise is just one of my apps. My secret to high productivity: Continuous Testing.
For detail, please read my article “Reflections of Software I Created over the Last 14 Years in My Spare Time”.
Action
Assess the CT maturity of your current project against AgileWay Continuous Testing Grading. If Level 0 or 1, set up one Continuous Testing server today. Yes, today, as there is no other software development activity more important than CT, as Lisa Crispin (author of the classic ‘Agile Testing’ book) pointed out (see below). I completely agree.
Whenever I speak to a conference session or user group meeting, I always tell people, “If you aren’t doing continuous integration now, go back to your office and drop everything and get your CI going. It isn’t hard to do, … A programmer can do it in a matter of days or less. There’s no excuse to not do CI.”
— Lisa Crispin's blog article “The Team’s Pulse: CI/Build Process” (2010–08–23)
Please note, the “CI Lisa referred to” is “Continuous execution of automated functional tests (Watir/Selenium)”, more like CT nowadays.
This article is an excerpt from my book “Practical Continuous Testing”.