A Good Software Testing Culture Can be Easily Broken

Lessons learned from Google’s “$100 billion error” yesterday.

---

This article is one of the Stories series.

Yesterday, Google’s “$100 billion error” made the news headlines.

The Register

CNN

Let’s see the official statement from Google:

“This highlights the importance of a rigorous testing process, something that we’re kicking off this week with our Trusted Tester program,” a Google spokesperson told CNN in a statement Wednesday about the factual error. “We’ll combine external feedback with our own internal testing to make sure Bard’s responses meet a high bar for quality, safety and groundedness in real-world information.” — CNN news

The two keywords here are “Quality” and “Important”. But wait, I remembered one quote from the “How Google Test Software” book.

The “$100 billion error” is just the opposite example of this quote in the book “How Google Tests Software”, don’t you agree?

I searched “Alberto Savoia” and “Patrick Copeland” (both wrote the forward of the book, and Patrick Copeland was the top of food chain for Google’s QA Engineering, reporting directly to Larry Page, the CEO and co-founder) on LinkedIn, and both left Google (in 2012 and 2016).

This shows that a good testing culture can easily be neglected and broken after key people leave the company. Sooner or later, “YOU HAVE TO PAY FOR IT!”

Over my consulting/coaching, I have seen this repeatedly. Let me share you a story.

Many years ago, after I rescued one software project and set up a daily automated End-to-End (via UI) regression testing process (1 BuildWise CT server with 3 mac mini build agents, running 148 selenium tests, ~30 mins build time). I trained and coached the engineers so that they technically could maintain the suite.

“Once we have a solid continuous testing process, i.e. daily execution of automated end-to-end (via UI) suite, we don’t need to settle for “if it ain’t broke, don’t fix it”. The team will have the courage to enhance and innovate, without fear.” — Zhimin Zhan

On my last day (a good coach leaves when the staff is ready), I told the owner of this start-up company, “Ensure all end-to-end tests pass by the end of every working day, if not, do so the first thing the next day”.

The owner nodded her head repeatedly, and said, “Definitely, I will not be that stupid to make the same mistake again”. She did it again. Within weeks, she instructed the team to ignore test failures on the CT server, and focus on delivering new features. I heard this from the engineers I trained, they were frustrated. While me not being there, maintaining work will take longer, still, they believed in the process (enjoyed the benefits) and wanted to keep the build green (passing all end-to-end tests) daily. But the boss told them not to run daily regression testing.