What is a Proper Agile Iteration (Sprint)? Part 4: Continuous Testing
E2E Test Automation is backbone of Agile/DevOps. Without solid automated regression testing, how can it be real Agile?
I share my view of “Proper Agile Iteration (Sprint)” in this series:
Part 5: Showcase (upcoming)
Part 6: ‘Production’ Deployment (upcoming)
Part 7: Sustained Pace (upcoming)
(I have applied the above in some client projects I led and all of my own app development, with great success)
Clarification: The testing in Continuous Testing (CT) means "Automated End-to-End (UI) Testing," not unit testing. Unit testing is typically part of a traditional CI process, such as running JUnit tests (aka, programmer tests) in Jenkins. Managing Automated E2E (UI) tests, however, presents a far greater challenge.
After “Done, Done”, the team get a suite of automated E2E (UI) tests after Sprint #1, maybe 5 or 10. Of course, to qualify a good build to show the customer, all tests in the suite must pass.
A simple rule in Test Automation, originating from JUnit nearly three decades ago, is this: If one test fails, the entire test suite is considered a failure.
With Automated E2E (UI) tests, many give up with excuses, such as “flaky” and “slow”.
Of course, that view is clearly wrong! Surely, it is feasible. Below is a successful implementation real E2E test automation.
Tip 1: You run Automated E2E (UI) tests in a real Continuous Testing server, not a CI Server
A common mistake is that many teams use CI servers such as Jenkins and Bamboo to execute automated E2E (UI) tests, which inevitably will fail. The reason: a wrong tool for the task. You should use a proper CT server, such as Facebook’s Sandcastle and AgileWay’s international award-winning (free & open-source) BuildWise.
Tip 2: Run the whole suite daily
One common behaviour I observed was that the test suite was initially run every a few days, but over time, the intervals between executions got longer and longer. Eventually, the team found it too difficult to address the increasing number of test failures and eventually gave up.
The solution is simple: run the entire suite frequently (essentially, daily regression testing) and prioritize getting all tests to pass (GREEN) by the end of the day. If not, make it the team's top priority the next day.
Why?
Frequent test runs mean that the scope of code and test changes remains relatively small, making it easier to identify the cause of any issues.
If left unresolved, more changes will result in more test failures, and eventually, the team's confidence will collapse.
Tip 3: Test Maintenance will become the main effort
The total number of E2E tests will keep increasing until the final stages of the development phase.
This means that the testing effort is primarily focused on maintaining the automated tests from previous sprints.
Tip 4: Understand the capability limit
Ideally, we want as much automated E2E test coverage as possible; however, in reality, we can’t achieve that due to the maintenance effort.
Over the past two decades, I’ve often heard test automation architects, test managers, and principal software engineers mention impressive targets — typically over 1,000 test cases — when discussing E2E automation goals. In reality, however, in the company’s entire history, they have never managed to reliably execute even 20 automated E2E tests over a sustained period.
Tip 5: Invest to gain more test automation coverage
To effectively manage E2E (UI) test execution—reducing feedback time and improving stability—it makes sense to do some investment.
A fast and reliable parallel testing lab.
A good hardware setup helps. Most software teams don’t need to go to extremes like Facebook; a few M4 Mac Minis are usually enough.
Training and Coaching.
Reality check: Most software companies lack expertise in true end-to-end test automation and continuous testing. The best—and most cost-effective—approach is to learn from a real expert. While this seems like common sense, it's unfortunate that many companies fail to do so.
Sadly, this common sense is rarely recognized by IT executives.
Related reading:
My new book: End-to-end Test Automation Anti-Patterns
