Unable to E2E Automated Testing Most User Stories? You Haven’t Tried Hard Enough. Part 1

The reason is simply a lack of capability. Good news: Motivated QA engineers can learn.

The article aims to address a frequently asked question, much like the one I received earlier this morning:

A reader’s comment on my article, “Why I don’t use Test Management Tools at all?”

Firstly, “does not make sense to automate every test scenario” is a correct but meaningless statement. There is no such thing as 100% coverage in testing, even with an unlimited budget (such as moon-landing by NASA, No, there was still a limit). So, like most of the tasks we do in life, there is always cost consideration. But for Web E2E Test automation, I can confirm that most user stories (for web apps) can be automated tested at low cost while yielding great benefits.

When developing my apps, I think this way. I cover every user story with automated UI tests (in raw Selenium WebDriver), and run them (the whole suite) as regression testing. This will detect many regression errors (which is very true). If all tests pass (in a CT server), I immediately push the new build to production.

I managed to achieve that for all my apps, for over 10 years. Yes, there were some challenging ones, but with such great benefits, E2E test automaion is the highest priority in my software development process. So, little comprimse there.With that mindset, I find the “challenging-to-automated-test secnarios” often not hard at all. Of course, I learn and grow along the way.

In Agile, you should at least cover a user story with 1+ automated E2E (via) UI test. Otherwise, it is not “Done, Done”. — Zhimin Zhan

In Agile, we must automate (E2E) every necessary user scenario for regression testing. Please note the word “user”. If an end-user can perform, why can’t we cover it with automated testing? If so, how can a software team maintain and support the app effectively? (you cannot verify the change.)

I know some people like to argue for the argument’s sake, saying “There is no 100% absolute …”. Yes, there are edge cases, but don’t you agree that is an easy and weak excuse for a test automation engineer? From my 16+ years of test automation consulting/coaching, most of the so-called “unable to automate …” cases are purely lacking the capability.

Long-time readers know that I have developed several large software and apps, including TestWise, SiteWise, ClinicWise, BuildWise, WhenWise and TestWisely (in beta). BuildWise won an international award, and all with commercial customers (except for the new TestWisely). I don’t have written manual testing steps for any of the above. If the automated E2E UI test suite (as regression testing) passes in the Continuous Testing Server (BuildWise), push the updates to production or release.

A recent run of WhenWise Automated UI Test Suite, consisting of 570 raw Selenium WebDriver + RSpec tests. Yes, after this green run, a production deployment.

For doubters, accept this fact first, as you can now check out most of my software. I have conducted “Continuous Automated UI Regression Testing” for over a decade. This process enables “daily production releases”.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

This is not only possible, but for me, it is necessary. As a one-person Micro-ISV, I develop and maintain those software in my spare time. I must rely on “E2E Test Automation” (with minor ad-hoc exploratory testing).

So, you know my answer to the question “doesn’t always make sense to automate every test”,

“It almost always makes sense to automate every test scenario. The question is about the endeavour and capability” — Zhimin Zhan