No “Bug-Free” Programmers; Top Programmers spend 50+% Effort on Test Automation
Learning and practising test automation is a must for good programmers.
Over my 20+ years of work in software development, I have met a few programmers with big egos who claimed that his/her code was 100% bug-free in the previous projects, during the interview or on the first day at work. Obviously, it was not true, actually far from it. Most humans tend to forget their failures and only remember their highlight moments. I bet you have heard the below statements from someone:
“My investment portfolio constantly yields much higher returns than the index”
“I have a strategy that guarantees to win at casinos”
The reality is that if these people could beat the share market or casinos, they wouldn’t have bothered talking to you. They would rather be on a luxury yacht. It is normal for people to describe their achievements and make it sounds like they have never failed when they know there is no immediate and objective verification. In software development, that immediate and objective verification is automated end-to-end regression testing (via UI).
A Story
A few years back, a new senior JavaScript contractor joined the team where I worked as a test automation engineer. He bragged about his “100% bug-free records” in his past projects during the introduction on his first day. A programmer commented after hearing that: “That was my thought until I met Zhimin’s test automation”. The others laughed, and some nodded. This new colleague was smart enough not to say more about this.
A couple of days later, this new “Bug-free” programmer started checking in code. About 30 minutes after his code check-in, the build failed in BuildWise (our Continuous Testing Server). I analysed the test failures and presented him with a few errors on several pieces of paper. BTW, I did not use the defect tracking system on that project.
Continuous execution of automated end-to-end tests (via UI) will find regression issues regularly. This is almost a given, as long as the test suite has a good coverage and run often. Why? because nowadays there are virtually no design and specification phases in ‘agile’ projects. Developers focus on one user story or change request, but often break existing features. Furthermore, modern web apps are dynamic (CSS & JavaScript), easy to get wrong.
On the defect report (on paper), I listed:
What failed? indicated by a screenshot of the app.
Which step? indicated by the test step.
How can you reproduce the error? the failed step line in the automated test script file.
With his permission, I did git pull to get the latest automated test scripts (in raw Selenium WebDriver), launched TestWise and ran a failed test. (His computer belongs to a former team member who left, with the test automation setup). He was shocked to see the error stack trace on his screen shortly after automated test scripts drove the app in a Chrome browser window.
In the coming days, I still detected regression errors caused by him frequently, but the errors were reducing. The reason was that he started to run some automated tests before committing the code changes. Soon, his regression-error-producing rate is about the same as the others. It was normal to have a few regression defects every day (sometimes a lot) for the whole team. Because the feedback was quick (via Continuous Testing), the programmers were usually able to fix the regression errors quickly.
Why Bug-Free Programmers are not possible?
Donald Knuth, the author of the classic book “The Art of Computer Programming” and the creator of Tex, said this:
“I believe that the final bug in TeX was discovered and removed on November 27, 1985. But if, somehow, an error still lurks in the code, I shall gladly pay a finder’s fee of $20.48 to the first person who discovers it. (This is twice the previous amount, and I plan to double it again in a year; you see, I really am confident!)”
You can read more about the “Knuth reward check”. Here I will state the facts:
Not everyone is Donald Knuth, a software legend.
Even so, “Knuth reported having written more than 2,000 checks, with an average value exceeding $8 per check”Commercial software projects are different from research ones
Donald Knuth worked in a research environment (Stanford University), which is different from enterprise/commercial projects with budgets and timelines. Also, generally speaking, people working on commercial projects are less passionate (the main purpose is Money).Graphic User Interface
Every IT professional knows that a software program with GUI is much harder to test. At the same time, it is also where the majority of errors are.Agile
In an agile project (virtually everyone is claiming that is what they are doing now despite that most of them still conduct software end-to-end testing manually) Software requirement changes. This will greatly increase the likelihood of regression issues.
Acknowledging regression errors is the first step in the right direction
For every software project that I worked on/consulted in ( test automation), my automated end-to-end tests picked regression issues on a daily basis. Think about what would happen if there was no automated regression testing.
Many of those issues/defects would go undetected into software builds, or worse, in production. One basic principle in software engineering (we all learned at Uni) is that the earlier a defect is detected, the easier and cheaper to fix it. Many later development efforts would be spent on bug-fixing. There are also many complications, such as tech debts, dare not refactor, blaming, low-moral, …, etc.
So, for a programmer who wants to achieve more, it is important to acknowledge that regression errors are common.
Every top (10X+) programmer I met focused on Test Automation because there they knew the benefits well. At some top companies, such as Facebook, programmers must perform test automation.
“It is really important at Facebook I will never, as a developer, write code and toss over the fence and for someone else to write test for it. It is my job as a developer at Facebook to write tests for my code, and my reviewers make sure I do”. — “Continuous Integration at Facebook”, a great presentation
Solution: Minimize Regression Bugs by Continuous Testing
While we, as software engineers, cannot ensure 100% bug-free with our work, we can minimize the majority of bugs (especially regression errors) with a good engineering process: Continuous Testing.
Continuous Testing, here I mean, is to execute the whole suite of automated end-to-end (via UI) tests frequently (multiple times a day) in a Continuous Testing server.
FAQ
I do unit testing and integration testing, shall that be enough?
No, it is not enough. You need to do automated end-to-end testing (via UI, if your app has UI) as regression testing as well, as it offers the best benefits. Check out Testing Pyramid Clarified.
Take web development as an example, an extra comma in a JavaScript/CSS file might cause one or many web pages to be rendered wrongly. Then the end users won’t be able to use the app. What will be the point of showing them your unit-testing reports if that’s the case?As a developer, I would like to have an automated regression suite, but our testing team is unable to provide it.
You implement them. A software engineer is responsible for his/her work, and there should be an engineering process to ensure quality. Every good programmer I met spent more time on testing (mostly automation) than coding.
Check out Advice on Self-Learning Test Automation with Selenium WebDriver.What is the percentage of your time spent on test automation?
I spend ~70% of all software development efforts (including management, design, coding, BA work, deployment, testing, maintenance, meetings, defects, …) on Test Automation and Continuous Testing. Every working day, I spend 70+% time using TestWise (test automation IDE) and BuildWise (CT server). This makes me at least 10X more productive (I created and maintained all my apps in my spare time).
Related reading:
My eBooks:
Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.