DevOps Clarified
DevOps is not about deployment with Docker/K8. It is a set of practices that enable frequent high-quality production releases. Most of DevOps efforts are on Continuous Testing.
This article is one of the “IT Terminology Clarified” series.
DevOps has been a buzzword in the IT world since 2018. According to Wikipedia, “DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.”
“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.” — Predicts 2017: Gartner Report
Name-wise, DevOps is not an ideal one as the original meaning is largely lost. However, DevOps is so popular that some others have started to copy its form: such as GitOps, QAOps, SecOps, ITOps, DevSecOps. A quick test: If you replace ‘Ops’ with the original word “Operations”, do these terms make sense? Obviously not. All these are just marketing tricks. But from this, we can tell that many people don’t understand DevOps. This article will try to clarify it.
Virtually every software project claims to do DevOps nowadays, and there is project personnel whose job title is “DevOps Engineer”. However, from my observation over the last five years, nearly every software company or project was doing fake DevOps.
After an iteration of Dev (development), you deploy to production (which is what DevOps actually does according to Scrum. Real DevOps usually means daily production releases). But are the customer operations still fine?
Table of Contents:
· What does DevOps really mean?
· I implemented real ‘DevOps’ in 2008
∘ A government project I led between 2008–2010
∘ My own projects
· On what basis for “most were doing Fake DevOps”?
· Self-Check whether a company is doing fake DevOps
· No solid Automated End-to-End Regression Testing = Fake DevOps
∘ No real CT process → fake DevOps
· DevOps Engineer’s work is not about Deployments, such as AWS/Azure or Docker/Kubernetes
· How do I do deployment?What does DevOps really mean?
Below are the keywords I extracted from the DevOps definition:
practices (a set of)
shorten the systems development life cycle
continuous delivery
high software quality
From the above, the goal of DevOps is quite clear: push software updates to production with high quality quickly and frequently. I believe few people will argue about that. And my following content is judged based on this goal.
I implemented real ‘DevOps’ in 2008
Some readers might question my eligibility to talk about DevOps. I have been implementing real DevOps (the goal, not the term) since 2008, fourteen years before. My opinions are based on the real experience of implementing several client projects I led and my own apps (in my spare time), all with daily-production releases.
A government project I led between 2008–2010
When I led this project in 2008, we were comfortable with production releases at any time with a build that passed all our regression test suites which consisted of 250 Automated E2E user-story level tests (the framework was Watir + RSpec).
With the team’s high-level production and morale, I believe we practically implemented DevOps then when the term DevOps did not even exist.
The Continous Integration server we used was CruiseControl which was probably the only available one at that time. I added several CT modules, such as parallel testing and intelligent ordering, into CruiseControl.
CruiseControl is the first CI Server not designed for running UI tests. Facing the challenges and failing to find answers from googling, I came up with the design of parallel testing and intelligent ordering, and implemented them. Later, I designed my own open-source CT Server: BuildWise, which won Ruby Award in Japan in 2018.
The table at the top right corner of the above image showed test results, over 200K, which were saved into the database. In the bottom right, a pair of physical lava lamps were connected to the build server. On a green build (passing all regression tests), the green lamp was turned on. Otherwise, the red lamp was on.
These two lava lamps (connected to the CruiseControl server to show build statuses) were the heart of our development process.
My own projects
Below are recent automated regression testing reports for my three apps (shown in the BuildWise CT server):
WhenWise (web app)
548 Selenium WebDriver tests (2018 — present)
340K+ test case executions
TestWise (desktop app):
307 Appium + WinAppDriver tests (2019 — present)
67K+ test case executions
ClinicWise (Web app):
611 Selenium WebDriver tests, (2013 — present)
618K+ test case executions
On what basis do I claim that “most were doing Fake DevOps”?
It is based on the comparison of top companies and my own work (see above).
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
“LinkedIn took a huge risk pausing all development for two months as it switched to a turbocharged new system known as “continuous deployment.” The gamble paid off big: LinkedIn now releases new web and app features twice per day, compared with once per month before. Amid a barrage of new products, its stock has more than tripled in less than two years.” — the subtitle of Wired article: “The Software Revolution Behind LinkedIn’s Gushing Profits”
“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs.”
In the field of technologies or engineering, we judge the credibility of any product by the results (compared to DevOps’ goal). Facebook and LinkedIn have the capability to push updates to production twice a day, which is DevOps. A software company that has a team of 5 DevOps engineers means nothing if all team members do regression testing manually (often under the flag of sound-better exploratory testing), or it still takes a few months to push a release to production.
Check if your company is doing fake DevOps
Let’s go back to DevOps’ definition and ask some simple questions. I provided my answers (in italic).
practices (a set of)
Can you quickly list one DevOps practice in which most team members are involved?
(No meetings if the build has been broken in Continuous Testing Server; Automated End-to-End regression testing via UI in CT server; Multiple full regression testing every day; Daily releases to production; No defect tracking; … )shorten the systems development life cycle
How often does your team push update to production? Daily, weekly or monthly?
(For my own web apps ClinicWise, SiteWise, WhenWise, and TestWisely, ~95% of customer feature requests and defects were implemented/fixed within 24 hours, the customers can verify in production today or tomorrow)continuous delivery
Do you run automated End-to-End UI tests in your CI/CD process? (just unit/API testing is not enough)
(Once I got a green run of automated E2E regression tests in the BuildWse CT server, I immediately pushed the new build to production)high software quality
Is every user story covered by automated End-to-End tests?
(True ‘Done, Done’. A user story is not done until it is verified by automated end-to-end tests (via UI); No defect tracking)
Question: how many outstanding defects are there in your Jira or defect tracking system? What’s the defect limit, in your opinion, in a real DevOps team?
No solid Automated End-to-End Regression Testing = Fake DevOps
A question: “Do you think comprehensive automated E2E regression testing (that runs well daily) is a must for a real DevOps?”. The answer shall be “Yes”. Let’s focus on this must-have practice (the most important one) now.
For people who disagree, I will cover that shortly.
No real CT process → fake DevOps
After seeing my automated regression testing reports, some might think “the daily regression testing reports do not necessarily mean you are doing real DevOps?”.
First of all, I have been pushing updates to production every day if there is a change to code/infrastructure changes, for over a decade. This sounds more like DevOps, doesn’t it? “Release Early, Release Often” is a dream of CIOs, and I have been doing it for several complex apps since 2012, and the client projects I have led since 2008.
DevOps Engineer’s work is not about Deployments, e.g. AWS/Azure or Docker/Kubernetes
A common mistake is that people misunderstand what a DevOps Engineer’s work is. For example, the below is from a DevOps Engineer job ad.
Previous project experience migrating on-premise to the Cloud(AWS/Azure)
Experience with container-based technologies and their deployment tools (Docker, Kubernetes, Chef)
Experience using Terraform, Vault, Artifactory, Gitlab
These are deployment tasks that are not DevOps! A DevOps Engineer is to support Continuous Testing. The reasons:
1. DevOps (or Agile) is generic (applicable to all software teams, regardless of tech stacks), and I spent < 0.1% effort on deployments.
Shocked? I develop and maintain my apps (TestWise, ClinicWise, BuildWise, SiteWise, WhenWise and upcoming TestWisely) in my spare time. If I have to worry even a little bit about deployments (for internal testing servers and production) on a daily basis, I certainly would have given it all up a long time ago.
2. Deployment Script doesn’t change often compared to code, test scripts or requirements.
Code and test scripts change frequently, but not deployment scripts.
if it ain’t broke, don’t fix it!
3. The deployment scripts (including IAAS) are mostly reusable.
Deployment scripts (including infrastructure as code) are reusable across many projects that use the same tech stack. For example, I copied my own deployment scripts (for Java and Ruby on Rails) to several client projects and only made minor customization. I had them working in a matter of hours.
4. DevOps definition does not mention deployment anyway.
The closest term should be “continuous delivery”, which is CD (the term that existed a decade ago)originally from the same-titled book. The authors of the Continuous Delivery book agreed: “The whole book is really about testing”, check out the full story in this article: Continuous Testing Clarified.
DevOps is about providing quick feedback from Automated E2E testing, in order to achieve ‘Release Early, Release Often’ of high-quality software to production. — Zhimin Zhan
How do I do deployment?
Deployment is a very minor task in a real Agile team. For all my web apps, the deployment is done via one command.
deploy.cmd productionSome might not believe it could be that simple. That is what I thought in 2005 before a renowned coach (in Java Celerity List) showed me. Yes, it was possible via scripting, even with complex J2EE apps. I have been doing this way ever since, for J2EE, .NET and Ruby on Rails apps.
Simple ≠ Easy
Because deployment is always so simple in my company, there was never a dedicated role or even a dedicated task. Being simple is just one attribute of deployments in real DevOps teams. The other attribute (of deployment) is being fast, very fast!
In a real Agile/DevOps team, besides the production server, the team members need to push new builds to many different server environments frequently, such as local, test, and integration (multiple for parallel testing) many times a day. For example, the deployment (to production) for the WhenWise app is typically around 15 seconds.
In terms of deployment technology, I prefer old-school Stack Script over Docker/K8. If you could achieve simple, reliable and fast with Docker/K8, go for it. Agile/DevOps never specify what type of deployment technologies to use as long as the deployment process is quick, reliable and repeatable.
If you are open-minded and not fixated on a specific IAAS vendor or deployment technologies (such as Docker), you will find it easy to achieve fast, reliable and simple deployments.
To put it simply, 95+% of DevOps is Continuous Testing (CT), i.e., Automated End-to-End Regression Testing via UI; The effort on deployment shall be neglectable.
Related reading