Continuous Testing Clarified
Continuous Testing is the most important process in DevOps; Yet most so-called ‘Agile’ or ‘DevOps’ projects did not have it or got it wrong.
This article is one of the “IT Terminology Clarified” series.
Continuous Testing, according to Wikipedia, is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” The keywords are “automated tests”, “delivery pipeline”, “immediate feedback” and “software release candidate”. (The reason I quote Wikipedia, a non-academic reference, for definitions: because Continuous Testing is new and there is a lot of confusion over it. So I reach for a common understanding at Wikipedia)
Table of Contents:
· Continuous Testing explained in simple words
· Continuous Testing is the Holy Grail of Software Development
· Software Development Trend ⇒ Continuous Testing
· Continuous Testing vs DevOps
· Continuous Testing vs Continuous Integration
· If CI is implemented properly, no need for CD or CT
· Continuous Delivery is really about testing
· Reality Check of CT/DevOps
Continuous Testing explained in simple words
Like many other formal technology definitions, the above (on Wikipedia) sounds right but is not quite clear. Let me interpret this:
“executing automated tests”, “business risks”
The automated tests are at the user story level: testing business features. For example, for a web app, the CT process will run a set of automated test scripts to drive the app to verify business functions, in a browser.
Comparatively, unit testing is conducted by programmers only, that’s why unit tests are also called programmer tests.“pipeline”, “immediate feedback”
In this pipeline, Customers/Business Analysts and programmers are waiting for feedback, and more importantly, ready to act on the feedback. Modification follows feedback, i.e, if there are test failures, a new build (with potential fixes) will be triggered to run another execution of automated tests to ensure the quality.“software release candidate”
The software is in a ready-to-release state if it has passed all automated functional tests, with little or no human activities in pushing the latest release to production.
Here is my interpretation of Continuous Testing: “Run automated end-to-end (UI) as regression testing, frequently on new builds. If all tests pass, the software is ready for a production release. If there are test failures, the team must act quickly on the feedback.”
For the benefits of CT, please check out this article series.
Continuous Testing is the Holy Grail of Software Development
I didn’t invent using ‘Holy Grail’ to describe CT (as much as I would like to take credit for it, I heard it from a conversation years ago, but pitifully, could not remember the source), but I think it is a perfect metaphor for Continuous Testing. Yes, it is a big claim, and many of you probably are doubtful. It is my hope to convince you with the Practical Continuous Testing book.
Here I share one customer’s comments after I helped implement the continuous testing process, which enabled the project to push updates to the production server on a daily basis. This product owner agrees: “continuous testing is super valuable and super rare, despite many heard of it, but very few saw it, just like the Holy Grail”.
Prior to practising test automation and CT, I have worked as a senior software engineer (Java/C#) for a number of years. I did not create a single my own product in spare time. Now have a handful highly-acclaimed software products, one even won an international programming award. Check out this article: Reflections of Software I Created over the Last 14 Years in My Spare Time,
When someone asked my secret, My answer: “I found the holy grail (of software development), that is Continuous Testing”.
Software Development Trend ⇒ Continuous Testing
Please read this article for detail.
Continuous Testing vs DevOps
“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.” — Predicts 2017: Gartner Report
If I ask you the hottest term in the software development industry in the past two years, many will say “DevOps”. Frankly, I think the term DevOps is quite vague (as opposed to ‘10-minute build’ and ‘pair programming’), therefore, it is open to interpretation.
Continuous Testing is the key (and most important) process of DevOps. You must have seen this DevOps image.
However, the below one is more accurate, as Continuous Testing is the pipeline with all the team members involved.
I have heard a few DevOps talks, however, they left no marks on my brain. For one project I witnessed in 2019, the executives got sold by the ‘impressive talk’ by a ‘DevOps talking expert’, and engaged the consulting company to implement DevOps. These consultants were busy talking, presenting, planning a roadmap, and introducing new software, …, for a few months. The result was a total disaster, in the end, the teams were told to revert back to the old way. The reason is simple: the foundation of DevOps is Continuous Testing. It is easy to understand, just imagine a pipeline producing poor-quality products in a factory. As we know, the quality problems in a pipeline magnify in order of magnitude.
Let’s switch the focus to DevOps’ objective (instead of its definition). I resolve to Wikipedia: “It (DevOps) aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably”. This sounds quite like the objective of Continuous Integration, doesn’t it? Except with an emphasized focus on quality releases and feedback to the team. For example, if you set up a Jenkins or TeamCity to build software and run a few unit tests (i.e. programmer tests), you might call it CI, but it is incomplete in terms of DevOps, as it does not include regression testing (at the functional level) for releases.
I don’t mind DevOps at all. As a matter of fact, I have been developing software this way (releasing high-quality software frequently with comprehensive automated testing) since 2007, and have shared my experience in numerous presentations. Only at that time, the term “DevOps” and “CT” did not exist yet.
Continuous Testing vs Continuous Integration
We cannot talk about Continuous Testing without comparing it to “Continuous Integration” (CI in short). Continuous Integration is “where members of a team integrate their work frequently, usually each person integrates at least daily — leading to multiple integrations per day”. In this famous CI article’s original version (published in 2000. By the way, that’s how CI started), Martin Flower used “often talked about but seem to be rarely done” in the first sentence. Based on my observation over the last two decades, this still remains true: the term CI is favoured by “talkers”.
CI Reality
I remember at one CITCON (Continuous Integration and Testing Conference) in 2009, a delegate talked about why he attended the conference: “I want to know how other projects are doing CI? The closest CI experience I ever encountered was that one machine was assigned to do it, and then ticked the box. No one touched the machine again.” Many agreed with him.
A decade later, most software claimed “doing CI” is no more than building (code) and deployment (package), with little or no execution of automated tests …
Here I highly recommend a great presentation: “Continuous Integration at Facebook”.
If CI is implemented properly, no need for CD or CT
CI has been so messed up in practice that it is becoming meaningless. That is why a new term comes up “Continuous Delivery” (CD in short, which later quickly lost its meaning as well), somehow people find it fancier to say these two terms together “CI/CD”. In every project I visited over the last decade, agile coaches/architects talked a lot about CI/CD and did not do hands-on test automation, their continuous integration processes were all embarrassing failures.
Once I worked at a software company, they had a Bamboo CI server with a number of projects, which seldom ran and virtually no sign of executing automated tests. However, they claimed they were providing CI consultancy to one of the top four banks in Australia.
If CI’s main purpose is to build a releasable software package, this has been achieved years ago (before CI concept existed) with build scripts, like an Ant task generating a deployable war file (back to J2EE days). Triggering a build from a web interface and seeing build results (on the CI server) is nice, but don’t you think there is not much to brag about? The purpose of CI is to ensure quality releases by running automated tests against release candidates. The testing, in particular, End2End user story level testing, is the main part.
Some people might say “Continuous Testing” could be the next ruined ‘talker term’. Yes, that could well be true, and probably already is. At this moment, we have run out of terms, sadly. I will settle with the term “CT” for now, because of its emphasis on testing.
So what is the relationship between CT and CI? Simply put, CT is a part of CI, the most important and difficult part. If a CI process is implemented properly, there is no need for “CD” or “CT”.
Continuous Delivery is really about testing
The origin of ‘Continuous Delivery’ is the book with the same title by Jez Humble and David Farley, published in 2009. Some will debate the differences between CI, CD and CT. Frankly, I don’t think it is necessary. The core of all three is the same: executing automated functional tests as regression testing.
Don’t just take my word for it, let’s hear the views from the authors of the Continuous Delivery book and a highly claimed authority in this field. In an interview in October 2019, Lisa Crispin, the co-author of the Agile Testing book, said this: “They (Jez Humble and David Farley) asked me to be a technical reviewer for their manuscript (Continuous Delivery book) … I read it. It’s a book about testing, you know, the whole book is really about testing. That’s the heart of continuous delivery. Jez Humble is very supportive of my saying that.”
Reality Check of CT/DevOps
Despite all the hype of CT (and previously CI/CD) and DevOps, the reality is 99%+ software teams at level 1 or 0 (a term I learned from the movie: Kungfu Panda) on the AgileWay Continuous Testing Grading. If you are not convinced, try to answer the two questions below:
When was the last time your project pushed a release to production?
How often do you do that?
In the context of DevOps, the correct answers for the above are “Yesterday or Today” and “At least once a day” (if changes were made).
According to this Forrester Study: Continuous Testing Separates DevOps Leaders From Laggards, IT executives often are not aware of immature/bad/fake Continuous testing practices in the company.
I have my reasons for using 99%+. Alan Page, the first author of “How We Test Software at Microsoft” book, said this at Test Talk PodCast #44, March 2015: “95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
Alan’s view has remained unchanged since 2008 when he wrote this on his blog:
“For 95% of all software applications, automating the GUI is a waste of time. For the record, I typed 99% above first, then chickened out. I may change my mind again.” — Alan Page’s Blog
Alan used ‘99%’ there, I added ‘+’ because most software companies won’t match Microsoft on
quality of software test engineers
resources (both technically and financially)
Furthermore, Continuous Testing adds more challenges, by running the whole test suite as automated regression testing frequently.