“Daily Production Releases” Clarified
“Daily Production Releases” is mandatory for real Agile/DevOps.
This article is one of the “IT Terminology Clarified” series.
Below is one reader’s comment on my article “Release Early, Release Often” Clarified”:
That is fairly common feedback. This article will answer this question and clarify “Daily Production Releases”.
Table of Contents:
∘ 1. “Daily Production Releases” can mean multiple times a day.
∘ 2. “Daily Production Releases” is a capability and the team’s confidence in regression testing, not a schedule
∘ 3. “Daily Production Releases” means production deployment on the same day when a change is made.
∘ 4. Automated End-to-end (via GUI) regression testing is the enabler of “Daily Production Releases”
∘ 5. How “Daily Production Releases” is possible?
∘ 6. “Daily Production Releases” means a complete change to the software development process.
∘ 7. Most CIOs, Managers, Tech leads, and Sofware Engineers chickened out in front of “Daily production releases”1. “Daily Production Releases” can mean multiple times a day.
The “Daily” in “Daily Production Releases” does not mean exactly one release per day. For example, Facebook “releases twice a day”; for simplicity, people call it “daily releases”, rather than “4-hour or 12-hour releases”. So, the ‘daily’ is not a literal date (once every 24 hours).
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Some people stretch the concept a bit, calling “one release every 2 days” daily-production releases. I think it is fine too. If there is no code change at all, why bother with another purposeless redeployment?!
The empathize of the “Daily” means a much shorter (~ 1 day) production release cycle (compared to the common 6 months), which will totally change SDLC.
I am sure you have met many fake agile coaches talking about Agile, but nothing really changed or is worse than before the so-called ‘agile transformation’. But, fake eXtreme Programming (XP) coaches are rare, why? In XP, Daily production deployment is a requirement.
Fake agile coaches, fake Scrum Masters, and fake SAFe Agilists would disagree and snob at XP. Wrong! see the below quote from an Agile Manifesto coauthor and world-renowned agile authority.
Some might say: “we are doing scrum, ‘daily production release’ is not a part of Scrum”. OK, let’s relax a bit, does your team do “production release every sprint (typically 2 weeks)”? If not, fake Scrum, according to the creator of Scrum.
2. “Daily Production Releases” is a capability of regression testing, not a schedule
“Daily Production Release” is a capability rather than a fixed release schedule.” — Zhimin Zhan
It is OK to plan a specific time for a 6-month release (by the way, a release means a production release in this article) in Waterfall, but it would be unnecessary to set a specific time for daily production releases, such as 6 AM every day in the morning, at least for most software products. If missing 6 AM, it is not a big deal, we just do it in one hour or two.
So don’t be fixated on the time, as long as it is frequent (from a few hours to a few days).
A production-ready build means the team has high confidence in its quality, specifically, the build passes the quality assurance. A bad release is often much worse than no releases. (Remember Window Vista?)
So, “Daily Production Releases” is really about the capability of full regression testing, not actual deployment. For all my web apps, deployment is done via one script mina deploy , takes between 15–30 seconds. The full regression testing (running automated end-to-end (UI) tests) in the BuildWise Continuous Testing server takes about 30–50 minutes.
3. “Daily Production Releases” means production deployment on the same day when a change is made.
If there is no change to code and infrastructure, there is no point in doing a production deployment.
Now I can answer the “daily production release is not required in my app or many industries”.
“Not Required”
It is not required because few software companies can achieve that, most IT professionals never witnessed one. Few people thought “high-speed rail” was required until Japan did it in the 60s.
Saying “not required” is really a sign of limited knowledge. Check out the LinkedIn story: The Software Revolution Behind LinkedIn’s Gushing Profits (2013). LinkedIn, in Silicon Valley, couldn’t get it before successfully luring Kelvin Scott.
Let me put in a sentence that might help understand how rare the real senior test automation engineer is. Image this job ad for Uber Drivers: “a former Grand Prix Winner”.
If there is a code change, even a simple typo, Some engineers may want to fix it ASAP for a software company that admires perfection.
“Not for my app or this industry”
Efficient and high-quality product releases are generic, and independent of the type of software. Let’s say, for company A, one production release per year on Jan 1st. On D-day, after deployment, an end-user found a critical defect. The development team quickly acted on and created a fix in 2 hours. Then a test manager said: “sorry, our regression testing takes 2 weeks. We will do deployment next year.”
Code changes happen all the time, and to release to production requires full automated regression testing, which is the pain point. Most (>99%) software teams do not know how to do proper automated regression testing. Quite often, some tech leads blindly thought they knew a bit about test automation, which made things worse. They shall seek professional help from a real test automation coach (rare) instead.
4. Automated End-to-end (via GUI) regression testing is the enabler of “Daily Production Releases”
Let me ask you a question first: “How long does a round of full regression testing take in your project?”.
If the answer is “1 week”, then surely “daily production releases” is not possible.
Averagely speaking, a properly-managed software team needs to perform 2–4 full regression testing rounds (and one pass) for a production release.
The answer is not “8 hours” either, shall be much shorter than that. Why? you cannot assume that the result of regression testing is always “ALL PASS”. In fact, on the opposite, good regression testing detects regression errors. After developers fixed the identified regression defects, there would be another build, thus, another round of regression testing. Then, possibly a new set of regression errors! , … I think you get the picture.
Therefore, a round of comprehensive regression testing shall only take an hour or less to achieve “Daily Production Releases”.
Test Automation
manual testing simply is not quick and reliable enough.
Comprehensive
ideally, every user story is covered by automated tests (Done, Done in Agile)End-to-End Testing (via GUI)
Unit testing and integration testing are not enough. A simple comma in JavaScript or CSS might cause web pages to fail to load. You do need end-to-end testing via (GUI)
Some might wonder how “End-to-End Testing” fits in SDLC and release, besides the above Facebook representation, read the below (from LinkedIn).
LinkedIn’s test automation enabled frequent releases.
“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs. Once the code passes the tests it is merged into trunk and cataloged in a system that shows managers what features are ready to go live on the site or in new versions of LinkedIn’s apps.”
— The Software Revolution Behind LinkedIn’s Gushing Profits (2013)
5. How “Daily Production Releases” is possible?
Yes, it is. As a matter of fact, once you have experienced one, you will know that is how natural software development shall be.
The technical how is comprehensive Automated End-to-End (via UI) regression testing, running multiple times a day in a continuous form. For more, check out How to Implement Real Automated Regression Testing?
A few words on the test execution time. The majority of software teams attempted test automation, failed and gave it up quietly. The reason: they were unable to maintain when a test suite grew to ~50 test cases, while the application changes constantly. Growing -test-count means a lot more maintenance effort, why? A run will take a long time. The solution: Run the whole end-to-end test suite in a Continuous Testing sever such as BuildWise, please note, not Jenkins or Bamboo, which are CI servers.
Here is a recent test report of my WhenWise app, its automated end-to-end regression suite consists of 558 raw Selenium WebDriver tests.
As you see, a run of this test suite (over 25000 steps) takes about 200 minutes on a single machine. By running the tests in parallel on 5 build agents, the total execution time is reduced to 46 minutes. For more on Continuous Testing, check out my book “Practical Continuous Testing: make Agile/DevOps real”.
6. “Daily Production Releases” means a complete change to the software development process.
Most IT engineers work on 3+ month release schedules. Now, somehow, magically, your team is doing real ‘daily production releases’. Consider the following activities you were used to:
Daily stand-up meeting
Spring planning
Velocity Chart
Retrospectives
The work of the Release Manager
Defect Tracking process
Regression Testing process
…
Surely, there will be changes, right?
Wired used “completely overhauled” to describe the changes made in LinkedIn.
“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day”
— The Software Revolution Behind LinkedIn’s Gushing Profits (2013)
7. Most CIOs, Managers, Tech leads, and Sofware Engineers chickened out in front of “Daily production releases”
Senior executives like to talk about how much they desire “Daily production releases”. Warning, don’t take that seriously. From my experience, the reality is often the total opposite: those people fear “daily production releases”. “Release often” is OK, but how often can be interpreted differently, but not “Release daily”.
How come? There is an ancient Chinese idiom that describes this mindset: Lord Ye Loves Dragon. I also shared one story in that article.
For readers who are still doubtful? Considering this scenario. If all wishes become true, your company suddenly is doing production releases twice a day, what will your CIO, senior VP, programme director/managers, architects and principal software engineers do every day? Surely, it is hard to justify several project development meetings every day. Generally speaking, human beings fear changes, especially ones that bring unknown impacts. Talking about, not doing, is safer.
Fact: even in many Dictatorship Countries, democracy is a positive word. The leader in power might talk highly about his country’s democracy policies, but no general voting.
Further reading: