CI/CD (Pipeline) Clarified
There is really no need for the terms such as CD, CI/CD, CI/CD Pipeline or CT, if software professionals do CI properly. Automated End-to-End UI Testing is the key.
This article is one of the “IT Terminology Clarified” series.
Nowadays, nearly everyone working in software development hears one or more of the following terminologies on a daily basis:
CI (Continuous Integration)
CD (Continuous Delivery)
CI/CD
CI/CD Pipeline
CT (Continuous Testing)
It is a bit confusing, isn’t it? This article will clarify.
Table of Contents:
· Why so many similar terms?
· Continuous Integration
· Continuous Delivery is really about testing
· Why “CI/CD”?
· CI/CD Reality
· CI/CD Pipeline
· CT (Continuous Testing)
· Why CI is good enough if done properly?Why so many similar terms?
First, if a software project does CI properly, there is no need for all other terms. In other words, most projects did CI badly.
The Toyota Corolla is one of the most popular compact cars n Australia, the current model is the 12th generation. In comparison, the names of competitor models from Ford, Holden and Mitsubishi kept changing. Why? Those cars were not good (quality, design, …). Therefore, those car manufacturers released a new model with a different name.
By the same token, after CI was done badly, the management and tech leads are happy to switch to another name, such as “CD”, “CI/CD”, …, etc.
Continuous Integration
Continuous Integration is “where members of a team integrate their work frequently, usually, each person integrates at least daily — leading to multiple integrations per day”. In this famous CI article’s original version (published in 2000. By the way, that’s how CI started), Martin Fowler used “often talked about but seem to be rarely done” in the first sentence. Based on my observation over the last two decades, this still remains true: the term CI is favoured by “talkers”, but few do it well.
For readers unfamiliar with CI, check out my article: Continuous Integration Illustrated, the drawing and notes I created in 2007.
Continuous Integration, according to Martin Fowler, includes testing and deployment.
“It stresses a fully automated and reproducible build, including testing, that runs many times a day.”
“Once Ant has done the compiling and deployment”
- — Martin Fowler’s original CI article
Please note the date of CI’s birth, the year 2000. The technology stack was primarily focused on the Java platform and web development, and there was no real automated functional testing in the browser then, but still possible with browser simulation using HtmlUnit. But the goal was clear: integrate software and run automated functional testing often, and once passing, do the deployment.
Continuous Delivery is really about testing
The origin of ‘Continuous Delivery’ is the book with the same title by Jez Humble and David Farley, published in 2009.
Let’s hear the views from the authors of the Continuous Delivery book and a highly claimed authority in this field. In an interview in October 2019, Lisa Crispin, the co-author of the Agile Testing book, said this: “They (Jez Humble and David Farley) asked me to be a technical reviewer for their manuscript (Continuous Delivery book) … I read it. It’s a book about testing, you know, the whole book is really about testing. That’s the heart of continuous delivery. Jez Humble is very supportive of my saying that.”
So, Continuous Delivery is really about functional testing, automated End-to-End Functional Testing. This makes sense, as only good functional testing can prevent bad software deliveries. The term ‘continuous” here means executing automated functional testers in a continuous process. (You cannot do manual functional testing continuously)
Why “CI/CD”?
Continuous Integration (CI) has been so messed up in practice that it is becoming meaningless, soon the new term “Continuous Delivery” (CD) in quickly lost its meaning as well), somehow one ‘genius’ somewhere found it fancier to say these two terms together “CI/CD”. Frankly, I have to admit that it is easy to say and does sound better (more catchy). Therefore, the term CI/CD became popular quickly. Who doesn’t like the idea of “release early, release often?”
Claiming doing “CI/CD” (fakely) has two benefits over just CI or CD separately:
If someone questions the output of CI does not contribute to frequent software delivery, the architect can empathize “CD” in the meetings.
If someone questions the delivery frequency is not improving, the architect could say: “our team is focusing on CI, and once matured, will move on to CD”.
In summary, it is a good term for playing office politics.
However, without automated End-to-End testing, CI/CD has no meaning. Without comprehensive/reliable automated regression testing, responsible software teams dare to release a build/fix to production. That is a fact.
CI/CD Reality
In every project I visited over the last decade, agile coaches/architects talked a lot about CI/CD and did not do hands-on test automation. Their so-called “CI/CD processes” were all embarrassing failures.
Based on my observations over the last 10 years, while many projects were doing “CI/CD”, the “Delivery” means nothing to them. Manual testing was still the primary form to conduct functional testing.
Of course, there was no lack of attempts to execute Automated End-to-End tests in CI servers, but most failed (using a wrong automation framework such as Cypress, or with a wrong Syntax framework such as Gherkin, or in a wrong scripting language such as JavaScript , …, etc). I noted down the pass rate of so-called ‘overnight builds’ in the software projects I visited last 10 years. The highest pass rate of the Automated End-to-End test suite was a miserable 48%! The target rate is 100% on a daily basis, like this one.
CI/CD Pipeline
The term DevOps appeared around 2010, it gradually became popular. Along with it, we see the following drawing often.
This look like a PipeLine in a factory, isn’t it? Another “genius” added “Pipeline” to “CI/CD”, to make CI/CD more relevant to DevOps.
CT (Continuous Testing)
CT is focused on executing automated End-to-End Tests. Again, shouldn’t exist if people are doing CI properly.
For more, please check out my article: Continuous Testing Clarified.
Why CI is good enough if done properly?
Some readers would argue that there are differences between “CI”, “CD”, “CI/CD”, …etc. I normally don’t like the debate on terminologies, especially since these can be proven (hands-on) quickly, such as my daughter’s article: “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.
The section below serves as FAQ.
1. CI does the integration, not deliver to UAT or Production. That’s CD.
Wrong! Please re-read Martin Fowler’s original CI paper, which clearly stated “Deployment” (see above). Some might debate the deployment target, System Test or Production server. Let’s look at another absolute authority, the book started Agile (in 1999), by Father of Agile: Kent Beck.
Facebook releases twice a day (see below). By the way, I have been doing daily production deployment (often, multiple times a day) for all my apps since 2012. . It is not only possible, frankly, now I don’t know do software development otherwise.
Desperate engineers might argue: “Kent is a legend, of course. However, that book was written 22 years ago, XP is out of date”. Wrong too, Let’s see what Robert Martin, another authority in Agile and co-author of Agile Manifesto, wrote in his new book (2019):
After all, “Release Early; Release Often” is always welcome. Please note, there was no term “CD” back in 2000.
2. CI runs unit/integration tests, not E2E tests, which belong to CT
Wrong! Without automated End-to-End regression testing, how dare a team push updates to production every night? (see above)
Not Convinced? Let’s see the below from a great presentation, “Continuous Integration at Facebook”.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Clearly, Automated End-to-End tests (in WebDriver) are a part of Facebook’s CI.
3. CI/CD Pipeline is about feedback to all team members
That’s what CI does too. The purpose of Facebook’s CI process is to provide feedback.
When automated End-to-End UI tests run regularly in your CI process, there are many benefits to all team members: Executives, Managers, Business Analysts, Developers, Testers and Customers. Furthermore, Teams can use automated UI tests to assist with: Showcases, Training staff and test data generation.
See, calling it “Pipeline” does not mean anything. Make it so, but first, how many automated User-Story-Level End-to-End UI Tests do you have?
4. Our CI/CD runs good coverage of unit testing, which is enough. No need to run automated End-to-End UI Tests.
No. it is not enough. CTO of ibood.com posted this image on LinkedIn: “Unit tests all green, no integration tests”.
Modern apps are dynamic and UI savvy. We all know that 100% unit-test-covered code can still have many functional errors. Don’t search for excuses, as an engineer, you need to have a repeatable process to ensure the end-product’s quality. Learn automated End-to-End UI testing , today.
5. We think running API tests in CI/CD is enough.
Still not enough, unless your app has no UI. I remember a recent world quality report (maybe 2021), which listed ‘testing from user’s perspective as priority’ first time. This makes sense.
Don’t make things more complicated than it is. If your users are using your product in browsers and smartphones, then you must test the software that way.
6. But we don’t have the capability to develop/maintain automated UI tests.
Learn it, like a real engineer. It is not only possible but can be fun, also it will make your team much more productive. You have already made a good first step: acknowledge that, good on you.
Related reading:
My eBook: “Practical Web Test Automation with Selenium WebDriver”
My eBook: “Practical Continuous Testing: make Agile/DevOps real”
Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.