Benefits of Continuous Testing (Part 1: to Wise Executives)
Big financial rewards and a relaxing lifestyle
Continuous Testing (enables software teams to push software updates to production daily, not fake CI/CD talks. Check out “Continuous Integration at Facebook” and AgileWay Continuous Testing Grading) is the heart of the software development process. It benefits all stakeholders of a software project.
Testers (automated & manual)
Executives get financial rewards
The biggest beneficiary of a successful software product, financially speaking, is the owner and/or executives of the company. A good process of developing successful products is often more important than the products themselves. However, few software executives realize the full value of frequently pushing out software releases (multiple times per day) which is only possible with a good Continuous functional testing process in place.
It is easy for executives to say “we are going agile”, “Steve Jobs once said …”, blah, blah, blah. We know there is only one Steve Jobs (who is known for being obsessed with quality. See his interview on quality). Most executives, who just want to keep their jobs, lack a real desire to make a difference. As this Wired magazine’s article pointed out, LinkedIn ‘completely overhauled’ its development and release process. Wow! However, if you think about it, it is so logical and it shall have always been like that.
The software revolution behind LinkedIn’s gushing profits [Wired]
…, It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day.
… Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs.
Engineering (in our process: software engineering, and in our titles: software engineers/testing engineers) shall mean something. In other industries, any serious business must have a repeatable process to ensure product quality, and the quality control process drives production. Our software industry probably is the only exception.
For software development, by its nature, what really matters are two things:
Development efficiency → Release Early
Nathan Myhrvold has pointed out the vast productivity difference between average and good programmers. Naturally, it is an executive’s interest to find and keep 100x (or more) programmers. These 100x (+) programmers write automated tests. You might have heard of TDD (Test-Driven Development). Here, I will emphasize that the challenge is not about writing automated tests. It is to keep all tests green, in other words, Continuous Testing. Kent Beck, the father of Agile, now recommends “test && commit || revert”.
Maintenance efficiency → Release Often
Software updates are normal (compared to others, such as your car or washing machine). This simply means that a highly-efficient quality control process is a must for a highly-successful software company. That process is Continuous Testing.
In reality, most executives in the software industry are quite short-sighted. They usually possess leadership traits: high EQ, interpersonal skills, …, etc. However, these are not enough for the software business which is quite different from others.
The real understanding of software development, for most people, can be only gained from years of testing and coding at a very high level. Of course, not everyone can be Mark Zuckerberg. A sensible way is to find a master software engineer (not an easy task though, like ‘Kevin Scott was lured to LinkedIn’) and follow his advice on engineering practices.
For all executives who have the ambition to make a difference, I recommend this insightful Forbes article: How Facebook Beat MySpace.
Some might say “I just run a small IT firm, and have no desire to be next Google/Facebook. Continuous Testing is a nice idea, but …”. Wrong! CT is more important for small companies.
Once I consulted at a small Start-up. The owner told me that she hasn’t had holidays for 3 years and worked on most weekends.
I asked her: “even if this new version turned out exactly as you hoped, will you be able to afford a long mind-free holiday?” She hesitated, probably never thought about it.
Then I said: “my guess is probably not. After release, everything is going very well, you will get more customers, therefore more feature requests/changes. And you will hire some new programmers. All these will add uncertainties. A major issue, such as a wrong database update, probably will ruin the product”. She nodded.
I continued: “The time that you can relax will be after a solid Continuous Testing process is in place. You could really enjoy skiing (her favourite) in Switzerland knowing all builds must pass the comprehensive regression testing, automated, of course.” She agreed.
On my last day there, I pointed to the parallel testing lab that I helped set up (1 build server + 3 build agents, all mac minis) and said to the owner: “You now have an automated regression suite of 180 selenium tests, running several times a day, ~30 min build time. As you can see from the build history, the test execution has been quite reliable. Still, it needs constant attention. Getting a green build every day is the top priority”.
She promised that “Sure, I would not be stupid enough to make the mistake again”. However, sadly, she made the mistake again. About a year later, I bumped into a former colleague who told me that the company bankrupted. The reason: too many quality issues and the two business owners had to work as manual testers as well.
I asked: “what about the test automation?”
The colleague answered: “Abandoned a couple of months after you left. We (programmers) liked the process. However, the boss didn’t allocate us time to maintain automated tests. They wanted to focus on delivering new business features, and worry about automated tests later. Therefore, more and more tests failed in the CT server, then nobody cared about it anymore”.
My Experience
I never considered myself an executive, but rather an Independent Software Vendor (ISV) with several software products. Besides productivity tools such as TestWise and BuildWise, I created several web apps such as WhenWise, ClinicWise and SiteWise. At first, I created these web apps for my own (including family member’s) needs, later a few friends and acquaintances wanted to use them as well. In other words, I have fee-paying customers who are running businesses relying on my apps.
This sounds like a lot of work, right? Yes, CT can greatly improve a team’s productivity (will be a separate article). It does not impact my family life much. I sometimes still worked in a full-time contracting/consulting role during the day. I do spend time with my children, watching TV shows and playing PS4 games. The reason: Continuous Testing. In 2018, our family went on a 6-week long holiday in Asia. During that time, I received a few feature requests. I managed to push a few updates to production as well but running the CT on my portable test lab: MacBook Pro.
Advice
If you are not determined, don’t do CT. For courageous executives who do want to implement real Agile/DevOps:
Searching for a real Continuous Testing Coach like Kevin Scott (whom LinkedIn boss did successfully find). Please note there are many fake ones there. To filter out fake ones is easy, a real one can make visible results/progress on the first day such as setting up CT servers and executing tests, and maintaining that rhythm onwards.
How to lure him/her to work or consult at your company? Financial rewards are generally not the main interest of real CT coaches.
Follow his/her advice blindly, but with clear expectations within a short time (in days). I recommend trying one project first. Be ready to “completely overhaul” your existing development and release process, such as dumping JIRA, Jenkins, Docker, …, etc. I am not suggesting replacing these tools/technologies for sure, but rather as examples of what “completely overhaul” might mean.
Monitor the CT process daily, and make sure everyone is aware of it.
Challenge
Expect strong resistance from the staff, especially the middle tier!
This article is an excerpt from my book “Practical Continuous Testing”.