Case Study: Real “Release Early; Release Often” ClinicWise Development. First production release after 40-hour development; Then Daily Production Releases
What does a real “Release Early; Release Often” look like?
This article shares my experience in developing my web project: ClinicWise.
Table of Contents:
· My Confidence came from …
· Release Early: First Production Release after 40-hour work
· Release Often: At least once a day if there were code changes
· Some Stats
· ClinicWise => WhenWise
· FAQMy brother is a dentist working in a public hospital. In early 2012, he and his best friend planned to start a private dental clinic in China. They spent nearly all their money on building renovation, equipment, and staff. One day, my brother called to ask my opinion on a quote they received for their IT system. From memory, it was about a once-off US$20,000 and 10% ongoing cost (plus support) each year.
I knew they had difficulty in making the once-off payment at that moment and they were worried the ongoing fees might be too high (please note that these quotes were in US dollars). To make matters worse, the clinic had scheduled to open only in a few weeks’ time. When my brother described the system (used at his hospital) “not good”, without much thought, I proposed that I would create a dental practice management web app for them.
At that time, almost all clinical practice systems were Desktop apps. Therefore, a web app for this purpose was an innovative idea. I didn’t realize that then, because it was the only type of app that I was comfortable to develop.
My Confidence came from …
I was quite confident with my developing efficiency, which was backed by my recent two exercises. At that time, I had already switched my day work from Programming to Test Automation. For the two recent projects, I coded the teams’ (3 and 5 developers) 3-month work within about three days (in my spare time). I knew my code worked fine (and better) because it passed the automated test suites.
“Realization: Coding is just a very minor effort in a good SDLC, the majority of effort is Automated End-to-End Regression Testing” — Zhimin Zhan
Some readers might find it hard to believe, that is 100X development efficiency! I don’t care about comparing to others. For me, I have to be highly productive, because developing my own apps solo, in my spare time.
Prior to 2005, my development prodcutivty was about the same as other senior contract software engineers. What made so? Check out My Test Automation Journey.
However, I did not know anything about the clinical practice system. If I knew how difficult it was, I probably wouldn’t have proposed to do it.
I was very confident then that I could do it. My confidence came from the following:
I no longer worked as a programmer at a day-time job
Test Automation work is pretty much universal, so I did not carry the work burden from work to home.Test Automation
Continuous Testing
Ruby on Rails framework
It is much easier than Java and JS (e.g. Angular.js) web frameworks, and it is intuitive and fun to use.
Release Early: First Production Release after 40-hour work
I timed the process. After 40 hours of requirements gathering (from my brother), coding, and testing (again, testing was my main effort), I made the first release of the production. Some may wonder how it could be possible.
Firstly, I will tell you some facts about me at that time.
working full-time as a test automation consultant
writing my third book “Selenium WebDriver Recipes in Ruby”
refining and maintaining TestWise
actively developing BuildWise (later international-award winning)
working on ClinicWise at night and at weekends
raising two teenage children
To get all these done, I had to be super-efficient. If the first production release had taken longer than one month, I would have already given up.
I guess your next question is How?
From the project management perspective, it was quite simple. I only worked on the absolutely necessary features at the start and developed them most simply and minimally, but made sure they did the work required.
For example, the first release contained the following:
One user
admincan log in (authentication)
Multi-user support and Authorization were only implemented a week later.The admin user can make a booking, in a minimal format
No recurring appointments, …The admin user can take treatment notes
Just one text area to record the notes. The sectioned notes were implemented a few weeks later.The admin user can create invoices (based on appointment) and record payments
No invoice printing (hand-written receipt first)…Only in Chinese
Multi-language support was implemented three months later.…
I believe you get the message. Below is the first test invoice created on the production server (2012–03–25). You can see that the quantity was missing, which was quickly fixed in the next build.
By the way, the UI was far simpler than the screenshot on that date. CSS was added later.
From previous work, I noted down the common reusable components, such as
User Authentication
Infrastructure setup
One-command deployment
Testing infrastructure setup, such as Testing Emails in Automated Test Scripts with a Fake SMTP server
Automaton Test script Project and Continuous Testing setup
Frankly, 40 hours was a bit long. I accomplished the first production release much shorter with my later web apps: WhenWise and TestWisely.
Some readers may remain doubtful. I will address some of the doubts in the FAQ section.
Airbnb
In 2007, Airbnb was created quickly by three founders using Ruby on Rails.
Release Often: At least once a day if there were code changes
Unlike software development in many organizations, production releases for me is a daily thing. More often, it happens multiple times a day.
After the first release, every night (for a long period of time) I received feedback from my bother, and then I worked on the fix and new features. Most of the time, the new requirements/changes were implemented in production the next morning.
Many more features were implemented and updated within one month after the first production release, naturally. A new release was pushed to production every night (if changes were made). Experienced software professionals would understand the challenges: the system has production data now!
The foundation of “Release Often” is to run Automated End-to-End Regression Testing in a continuous manner. The logic is quite simple too: every user story is truly ‘Done, Done’, i.e. covered by at least one automated end test script. All test scripts PASS means that the software is doing fine and the software build is ready for production.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day.”
— The Software Revolution Behind LinkedIn’s Gushing Profits, Wired, 2013
Once you get used to it, daily production release is normal. Check out my other article: Did You Push Software Updates into Production Yesterday?
Some Stats
1. A full automated regression testing (via UI) report on the BuildWise CT server
ClinicWise’s automated regression suite consists of 611 raw Selenium WebDriver tests which run in our BuildWise CT server (all at one go). If I get a green build (ALL PASS), I push the new build to production. I trust my automated regression testing.
2. Build History
3. Total Test Case Execution
There were 618, 327 test executions over eight years (the actual number would be bigger because BuildWise excludes cancelled builds)
As a comparison, the average test execution was 3,000 per month for the whole team in a large government testing team I once worked that consisted of ~ ten manual testers.
ClinicWise => WhenWise
I built ClinicWise exclusively for my brother, and later attracted some other users, including the local practices in Australia (I added internalization). Overall, there was no marketing for ClinicWise. About a year later, there were a few competing products on the market, and they did marketing aggressively.
While I am not against marketing, I avoid doing that myself purely for personal reasons. I did receive a few business interests, some tried and failed, some wanted to invest in my company, … etc. Long stories short, ClinicWise was not commercially successful. I am totally fine with that as I created it for my brother anyway, and his clinic staff have been using ClinicWise happily.
One business lesson I learned is that ClinicWise is still not challenging enough technically. A company with ~25 developers might be able to develop it. I want to focus on far more challenging apps such as TestWise and BuildWise.
I stopped taking new customers for ClinicWise in 2021 but will continue to support existing customers. I migrated many of the functionalities to WhenWise, a new online service booking app. WhenWise was my daughter’s idea when she was on an overseas camp trip. I liked the idea and thought it would appeal to a wider customer base. You can book my training and one-to-one test automation coaching there.
FAQ
1. You could NOT get the database structure defined in the first go!
That’s correct, I did not, could not, and had no need to. In fact, ClinicWise had hundreds of database migration changes. Ruby on Rails framework comes with database migration support. Check out my other article, My Innovative Solution to Software Development: Versioned Database Migration.
Your current doubt probably is “How can he have the confidence to change database structures frequently?”. The answer is automated End-to-End regression testing (via UI).
2. Did you use Jira or similar to manage the project?
Not at all. I never had the need for that.
3. Surely your apps would have defects, How did you do defect tracking?
Yes. ClinicWise (and my other apps) had defects. With test automation, these were much less than the industry average. But I didn’t do defect tracking at all. The answer is still Test Automation. Check out my other article, Why Don’t I Use Defect Tracking? No Need, I do real Continuous Testing.
4. You could NOT always implement the new user story on the same day!
No, I could not always, but I did a large percentage of the time. The classic book “Implementing Lean Software Development: From Concept to Cash” has the answer to this question.
If the customer sees that the business features can be implemented the next day, they will provide realistic and immediate useful requirements. The problem in most software projects in terms of requirement-gathering is that the customers fear that they may miss providing some requirements Therefore, they will add as many requirements as possible (which generates lots of waste). This issue will go away once “Release Early; Release Often” is truly implemented.
5. Are you afraid of going wrong with daily production releases?
No, not at all. I constantly see fear in the teams that are doing six-month release cycles. Yes, something could go wrong (even after my comprehensive automated end-to-end regression testing). I just need to fix it quickly (code and automated tests) and run the regression testing in BuildWise again. Maybe within an hour, the issue will be fixed in production with another update.
Related reading: