There is no “100% coverage” in Automated E2E Software Testing

It is simply not possible, and neither necessary.

It is well accepted that 100% test coverage (functional, not unit testing) from manual testing is impossible. However, some had unrealistic expectations of 100% coverage in E2E Test Automation. This is very wrong, and this article will explain why.

“It is neither possible or necessary to achieve 100% coverage in software testing”
— Zhimin Zhan

Table of Contents:
· 100% Test Coverage is simply Not Possible
· Test Execution Time?

100% Test Coverage is simply Not Possible

First of all, people saying 100% test coverage don’t understand test automation or software testing.

At a conference talk, I said, “I can easily come up with hundreds of automated test scenarios for a credit card payment user story, just on the single payment page”, here is a partial list:

• Can pay the full amount with Visa Credit Card

• Incomplete credit card info (mandatory check), e.g. no CSC, CC number, Expiry Date, …

• Credit card info not matching, e.g. wrong CSC, card number does not match type

• Invalid expiry date, e.g. already expired or too far in the future

• Amount too small (threshold) or too large for credit card payment limit (set by the app, or/and by the credit card type)

• There is not enough money left on the credit card

• Enter negative amount

• Partial payments (can be a lot of cases here)

• Special characters in the cardholder name (e.g. Japanese or Arabic)

• …

• and different combinations of the above.

These are valid test scenarios. Some might argue a few of them are extreme. About 20 years ago, I did encounter one subtle failed scenario for a payment gateway: when the total length of the merchant name + credit card info exceeded a certain limit, the payment processing threw an error. I found out this strange error by luck by reading the payment gateway’s source code. Then I changed the merchant name from “Brisbane City Council” to “BCC”, it worked!

The point of the above exercises:

“You cannot test enough for software” — Zhimin Zhan

Once we establish the above, we can plan the functional testing and decide what is good enough within the constraints of the given time and resources.

I have helped many software projects to achieve much higher coverage (in terms of business features) than their previous testing practices via Test Automation. Still, test leads and managers often wanted to plan big.

At one large financial company, 1200 was a magic number that Matt, a test manager, often talked about. I was working in a different team, and somehow Matt heard of my work on test automation for the product. By the way, all the previous test automation attempts failed over the last 14 years in this company. ( the company always had a team of automated testers, including contractors and a consultant)

Matt came to talk to me about using my approach to the automated 1200 tests, which were considered good coverage. There was quite a lot of effort spent on identifying those scenarios.

After hearing his repeated mentioning of 1200, I said I was not interested in that. Instead, we should focus on each day the testers can develop and, more importantly, can keep all existing ones running valid. Anyway, later I rejected Matt’s contract offer because his team (not him) insisted on using Java, while I insisted on using Ruby, the best scripting language for automated test scripts.

Test Execution Time?

For people who still cannot ditch the idea of 100% test coverage, try to answer this question: “even if somehow you manage to accomplish that, how long will it take to run the whole test suite?”

Assuming 1200 as the test count for the ‘perfect’ coverage, let’s say the average execution time for an End-to-End automated test is 30 seconds (pretty quick, usually longer for enterprise apps), then a perfect run of the test suite would take 10 hours! Another run of the whole suite is required for

• A single step failure (in 1200 tests)

• Any change to code, configuration and infrastructure

How confident are you in managing that?

Some might argue, “we can run them in parallel in Jenkins or Bamboo.” In theory, yes. But over the last 16 years, I have never seen one working with Jenkins or Bamboo (these two are CI servers, not CT servers, that’s a different topic). Have you considered some tests would fail when running them concurrently? Parallel executing automated E2E tests is easier said than done. Anyway, before making big plans, please show some solid parallel testing proof (with real tests).

The fact is: I have NOT witnessed one single successful test automation attempt, except my clients’ (who develop RSpec tests in TestWise and run them in BuildWise CT Server), that could reliably execute 50+ automated E2E tests on a daily basis. Yes, 50 tests, that is Level 2 of AgileWay Continuous Testing Grading.

Some might ask how many automated E2E tests I could manage (over a long period)? Here are my scores:

• ClinicWise, 611 (Selenium Web) since 2013

• TestWise, 307 (Appium + WinAppDriver) since 2019
  (previously, I was using RFormSpec based on AutoIT3)

• WhenWise, 548 (Selenium WebDriver) since 2018

For all my software products, I never used defect tracking, as there is no need. Surely there were defects (reported by staff and customers), but I quickly replicated them into (or updated) automated tests (this means my regression suite grows, i.e. more protection). All my software products have the capability to push updates to production daily (after getting a green build in BuildWise), thanks to my Continuous Testing process.