“Release Early, Release Often” Clarified
Solid Automated End-to-End (via UI) regression testing is the enabler for “Release Early; Release Often”
This article is one of the “IT Terminology Clarified” series.
Most IT professionals have heard IT executives talking about “Release Early; Release Often” a lot, typically in company town hall meetings. However, this phrase has become an ideal slogan, which is meaningless in everyday life.
Most CIOs like the idea of “Release Early; Release Often”, but they have never experienced it. At the bottom of their hearts, they are not even sure if they really want it. This Chinese idiom (Lord Ye Loves Dragon) describes this kind of behavior very well. Without the strong determination of the CEO/CIO, the middle tier: senior VPs, managers, fake agile coaches, fake scrum masters, fake DevOps leads, and architects, would just do some fake work under the banner of “Release Early, Release Often”. For most of those people, this is just a job, and fearing change is normal.
If you are CIO/CEO and really want to achieve “Release Early; Release Often” and would like a real life case study at a large IT organization. I recommend Wired’s article: The Software Revolution Behind LinkedIn’s Gushing Profits, a good read.
I will share my thoughts on this slogan in this article.
Table of Contents:
· A Story
· Bad Release is worse than no Release
· Release Early: How Early? The first month or the first week?
· Release Often: How Often? Every Sprint or every day?
· How to Implement “Release Early; Release Often”?A Story
Almost a decade ago, I worked as a test automation consultant at a large tech company. At a company town hall meeting, the CEO and CIO spoke passionately about the ‘Agile Transformation’ the company was currently doing. Inevitably, both of them mentioned “Release Early; Release Often” a few times.
The company invited a guest speaker who was the CEO of an Agile Consulting company (the headquarter in Melbourne). He was a passionate speaker, in Steve Ballmer’s style.
Instead of talking about “Developers”, this CEO repeatedly said, “Automation, Automation, Automation” and “Testing, Testing, Testing” (please note that this was a cunning trick, and explanations will be followed later). Towards the end of his speech, he told us a story: recently, his consultants helped an organization achieve 45 production releases a day”. The audience applauded, including my project manager Mark.
As Mark was sitting next to me, I whispered to him: “It must be a lie!”. Mark looked at me surprisedly, “Why? ”(by the way, Mark has worked with me on two projects, both of which achieved production deployment every sprint thanks to the daily automated regression testing I implemented)
I replied: “A simple math. 45 production releases per eight hours, that is, ~10 minutes per release. How long will the regression testing take? What are the reasons for a release? If his company could do good automated regression testing, surely he would talk about it a lot and show stats. But he didn’t. What is the value of deploying again and again without code changes? ”
Mark understood instantly and agreed with me. Like many other people, he was influenced by the atmosphere and forgot about logical thinking.
I continued: “If 10 minutes is pure deployment time, it will be meaningless. The typical deployment time for all my web apps is about 20 seconds. If we follow his logic, we will do thousands of production deployments per day, but for what? The main cost of a production release cycle is automated regression testing (via UI)! He did not mention end-to-end test automation at all but only said the words ‘testing’ and ‘automation’ repeatedly and separately. That’s cunning”.
After the town hall meeting, this agile consulting company was engaged to help improve the agile process of the company’s flagship product, i.e. to make it ‘Release Early; Release Often”. There was no more news after that. To my knowledge, there had not been any changes implemented (the teams, with the exception of mine, were still doing manual testing) by the time I left the company.
A few months after, I had a LinkedIn chat with a former colleague who was working at a large bank in Melbourne. I learned from him that the agile consulting company’s CEO was previously this bank’s division CIO, who had terrible track records: buggy and behind-schedule releases. As a matter of fact, he was let go of the bank with shame. However, he figured out the software industry's needs, set up his own company, and talked himself up to fool others. I have to admit that he was an excellent salesman. He started his company with two associates to offer ‘solutions’ to those tech companies in need. According to him, his company was one of the top 10 fastest-growing Australian IT companies by BRW and now with over 100 employees.
Bad Release is worse than no Release
The foundation of a software release is quality. Without sufficient quality, there is no point in talking about software releases. This is common sense.
Remember the Windows Vista fiasco? I bet that Microsoft executives wished they did not make that release. According to the article ‘The top five reasons why Windows Vista failed’, the NO.1 reason was “It broke too much stuff” (i.e. poor regression testing)
The quality of software development is reflected in Software Testing. Hence it is also called “Quality Assurance”.
Release Early: How Early? The first month or the first week?
The main problem with the “Release Early; Release Often” slogan is that it lacks a specific target. In real Scrum, a production release after Sprint #1 is two weeks. Even factoring Sprint #0, the preparation iteration is 28 days tops.
Some “Certified Scrum Masters” might wonder “How can that be possible?” Yes, it is possible. I released ClinicWise, one of my web apps, after 40-hour coding/testing. For more, check out Case Study: Real “Release Early; Release Often” ClinicWise Development.
Compared to “Release Early”, “Release Often” is much harder to achieve.
If a software team can “Release Often”, then “Release Early” is given. The team can fix/update quickly. They don’t worry about ‘non-perfect’ releases. — Zhimin Zhan
Release Often: How Often? Every Sprint or every day?
Agile processes such as eXtreme Programming (XP) and Scrum clearly defined the production release frequency.
Please note that Kent Beck, father of Agile, wrote “daily (production) deployment” in his classic book over 20 years ago.
“Daily Production Release” is not a theory. In fact, it is a must and normal in real Agile/DevOps teams.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Some might say: “That’s Facebook. We're just a small or medium-sized company.” There are more reasons for small companies to do so. It is a question about capability and determination and is not really related to the cost. As a micro-independent software vendor (MISV), I set the goal of daily production release when starting my own company. Honestly, I couldn’t imagine developing all my software without practising daily-production-release. Check out my other article: Did You Push Software Updates into Production Yesterday?
How will you implement “Release Early; Release Often”?
IT executives must make it clear that Automated End-to-End (via UI) Regression Testing is the top priority in SDLC, not time-wasting activities such as velocity based on estimated user story points.
Some developers might argue that “if we do good unit testing and integration testing, it will be set for release early; release often”. Wrong! Unit Testing and Integration Testing are the responsibilities of developers, not testers’. A user story without automated acceptance (end-to-end via UI) tests, by Agile’s definition, is not ‘Done, Done’. 100% good unit-test coverage (rarely implemented) would help the general quality of the code but offer limited protection for the business features. With web apps being highly-dynamic nowadays, an extra comma in JavaScript or CSS might cause the website to be totally unusable.
YOU DO NEED AUTOMATED END-TO-END (via UI) TESTING. If you are a CIO and your principal/chief software engineers or solution architects tried to avoid automated end-to-end testing, be alert! The most likely reason is that they are incompetent software engineers.
“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, software legend, in a podcast (2018)
Learn from LinkedIn’s Chairman, and go to lure the best tester (see the Wired article below). If you work for a small/medium-sized company, seek help from a professional test automation coach.
If an IT executive wonders how to achieve automated end-to-end testing, check out my other articles:
If someone wants a quick and easy solution, there is one: engage and listen to a real test automation coach, as LinkedIn Chairman did.
“LinkedIn now releases new web and app features twice per day, compared with once per month before.
“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs”
“Much of LinkedIn’s success can be traced to changes made by Kevin Scott, the senior vice president of engineering and longtime Google veteran lured to LinkedIn in Feb. 2011”
“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and app”— “The Software Revolution Behind LinkedIn’s Gushing Profits”
Related reading: