Be aware of the “fake it until you make it” mindset in Test Automation and CI/CD
Be a real IT professional!
This article is one of the “Be aware of Fake Test Automation/DevOps Engineers” series.
The first big news in the tech world this year is that “Elizabeth Holmes, the former CEO of Theranos is found guilty on four out of 11 federal charges”. This so-called “female Steve Jobs” will probably spend her next 20 years in prison. According to the New York Times’ report on this, faking is common in Silicon Valley.
“In Silicon Valley’s world of make-believe, the philosophy of “fake it until you make it” finally gets its comeuppance.” — New York Times (2022–01–03)
In doubt? Here is a line from my favourite TV show “Silicon Valley” (Season 4, Episode 9):
Faking Test Automation (UI) and CI/CD are everywhere
With that perspective (faking at top executives), it is reasonable to assume some degrees of faking at lower levels (IT projects or individuals) as well. Readers probably will agree more with my previous comment “E2E test automation in most software projects are faking, intentionally or unintentionally”.
Faking Test Automation, CI/CD/CT, or DevOps is very common. Not convinced? try to answer the following questions:
[Test Automation] How many user stories are covered by automated UI tests?
[Test Automation] Does your project have a comprehensive automated regression testing suite that runs daily?
[Test Automation] Does your team still mainly rely on manual testing?
[CI/CD] What is the average code coverage (unit testing) of last month’s builds on a CI server?
[CT] Do you run automated UI tests in a CT (or CI) server? How long does it take?
[DevOps] How often does your team push updates to the production server? every day or every sprint?
[DevOps] How many minutes do your developers wait for feedback (integration or testing) after a code commit? (Facebook’s goal: 10 minutes)
Please note, while test automation and CT failed in most (>95%) software projects, test automation engineers at those projects can be classified into two categories: TRYING or FAKING. I do know a fair percentage of engineers were frustrated because they were unable to access proper training and mentoring. There is no shortage of deliberate fakers in test automation and CI/CD, those fake engineers (in particular, senior roles) made the job unnecessarily harder (and more expensive) with all sorts of mistakes.
How easily people are fooled by fakers?
Yes, very easily. Let’s see the Theranos's example again.
Biden (as Vice President) visited Theranos’ headquarters in California in 2015 for a public event and called it ‘the laboratory of the future’. [DailyMail]
The Obama administration even invited Holmes to a state dinner in 2015 honouring the prime minister of Japan.
former President Bill Clinton was reverently quizzing her about her thoughts on technology.
Rupert Murdoch is reported to have invested $100m in Theranos between 2014 and 2015 [The Guardian]
many Venture capital (VC) investors.
Obviously, Elizabeth Holmes is a master of telling fake stories, and many high-profile celebrities and companies fell for it.
However, in my opinion, faking in Test Automation and Continuous Testing (CT) can be stopped because it is so easy and quick to identify, almost instantly.
Faking test automation can be exposed immediately
In the classic IT book “Peopleware: Productive Projects and Teams”, there is a classic story on hiring software professionals purely by their answers to questions in interviews.
“It would be ludicrous to think of hiring a juggler without first seeing him perform. That’s just common sense.” — Peopleware, chapter 16.
Frankly, this is somewhat excusable. Setting up a hands-on interview requires some effort (I did one, highly recommended). However, for Automated End-to-End test automation, it is easy, and the result would be quick, objective and clear. The reasons are:
Web technologies (defined by W3C) have largely unchanged
such as HTML, JavaScript and CSS.Selenium WebDriver, the W3C standard-compliant automation framework, is also largely unchanged since its release in 2011.
The above means that one competent test automation engineer shall be able to implement automated tests immediately, i.e. visible progress in a very short time.
To make it easier to understand, let me ask you a question: “When you ask a handyman to install a split air-conditioner, will you allow the technician to complete the work for 1 month and charge the full rate?” Of course, you won’t. You expect the work to be done in a couple of hours. Why? Because all air-conditioners (the same type) are made according to a standard. Customers’ expectations are the technicians have mastered the installation from previous experiences (if this is the technician's first job, he/she would be accompanied by an experienced one).
The same principle applies to automated test creation (test maintenance is a different game, which is far more challenging). I have always been able to implement a few real tests (web apps) on the first day for all client projects. Because all web apps are the same, I used exactly the same formula: raw Selenium WebDriver + RSpec (free and open-source, for more, check this out).
Here is an example of faking test automation that my student told me about. She worked in a government department, which engaged two highly-paid consultants from a well-known software consulting firm. The framework they used was Gauge. Anyway, two months passed, and 10 automated tests were created. Only half were running reliably. A real test automation engineer shall get it done within 1 or 2 days, but took those two high-paying consultants 2 months, and was done badly. Those two fake “test automation consultants” were on Level 0 of AgileWay Continuous Testing Grading.