In-Sprint Test Automation Clarified
This term is unnecessary, shall be always doing that in real test automation and real Scrum.
This article is one of the “IT Terminology Clarified” series.
I recently read two articles (#1, #2) mentioning “In-sprint test automation”, its definition is as below:
The user stories developed in the same sprint are verified by automated end-to-end tests.
The benefits of In-sprint Automation are obvious, testing early is always welcome.
Table of Contents:
· In-sprint Test Automation should always be the case in real Agile
· The most effort is not on the stories in the current sprint, but rather regression testing
· How to succeed “In-sprint Test Automation”?In-sprint Test Automation should always be the case in real Agile
However, I think the term “In-sprint Automation” is unnecessary, as already implied in ‘Done, Done’. I have been performing ‘In-sprint test automation’ since 2006, 16 years ago. If not automated testing user stories immediately after they were implemented, that’s Waterfall, IMO, at least from a testing perspective.
So when I saw “In-sprint test automation” was listed as №2 in this “The top 5 software testing trends of 2022” article, I highly doubt the author has ever done one successful test automation. “In-sprint test automation” IS NOT a trend; it should have always been the case. Few people have seen real test automation, because 95+% were doing fake test automation.
Suppose a 10-sprint project and there are 10 user stories for each sprint. If not “In-sprint test automation”, Story #11–20 are not tested in Sprint #2, when will these user stories be tested? Another question, What kind of testing will be performed to mark Story #11–20 done? Manual testing?!
“Development Teams deliver an Increment of product functionality every Sprint. This Increment is useable, so a Product Owner may choose to immediately release it.” — scrum.org
As you can see, “In-sprint test automation” is included in the concept of Sprints.
Can you see it now? people who are now promoting “In-sprint test automation” most likely do not really understand Agile or Test Automation, or both.
How do IT Executives prevent being fooled by those fake test automation specialists? It is easy, check out my 35-Word Functional Test Automation Strategy.
The most effort is not on the stories in the current sprint, but rather on regression testing
“In-sprint test automation” is another hype that can be demonstrated at its very early stage, then started to fail gradually.
Like many fake activities in test automation, once pointed out, it is so obvious. Continuing our 10-sprint example, at Sprint #2, the team needs to manage regression testing of Story #1–10, as well as the newly implemented Story #11–20. At Sprint 8, the regression testing will be for 70 user stories.
Please don’t forget the key characteristic of Scrum/Agile is to welcome changes, even late changes. So, a fair percentage of previously implemented user stories would be affected by later ones.
Also, the success criterion of test automation is always: ALL TESTS PASS. Specifically:
The automated tests for newly implemented user stories: PASS
The automated tests for all previously-implemented user stories: PASS
Many test failures will be detected, and both testers and programmers need to work out the cause, and update test scripts or code accordingly
Any change results in another round of regression testing
To give you a direct feel of this challenge (maintaining regression tests), I will show test automation stats for WhenWise, one of my apps. WhenWise regression test suite consists of 551 user-story-level automated end-to-end UI tests, in raw Selenium WebDriver.
To get a green run (like the above one), each of ~30,000 test steps must pass. Each test step is a user operation that you would see in a Chrome browser.
+------------+---------+---------+---------+--------+
| TEST | LINES | SUITES | CASES | LOC |
| | 26193 | 336 | 561 | 20542 |
+------------+---------+---------+---------+--------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 9883 | 181 | 1625 | 7466 |
+------------+---------+---------+---------+--------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 823 | 5 | 61 | 637 |
+------------+---------+---------+---------+--------+
| TOTAL | 36899 | | | 28645 |
+------------+---------+---------+---------+--------+Now, do you agree “In-sprint test automation” is really about automated regression testing?
How to succeed in “In-sprint Test Automation”?
In case you are asked to implement “In-sprint Test Automation” and wonder how. After understanding “In-sprint Test Automation” is really automated regression testing, you will find this article: How to Implement Real Automated Regression Testing? helpful.
Here, I want to highlight three key points:
1. Test Automation Team must be very highly-efficient.
Along with sprints, more and more work will be on maintaining the existing regression suite. After Sprint 8, is it more like ~90%, the effort of developing tests for new user stories is almost neglectable. In other words, if a team of automated testers are still worrying about test creation, the game has lost already.
Let me give you an example. My 20-year old daughter Set up, Develop Automated UI tests and Run them in a CT server on your First day at work at her first intern job. Her team members were deeply impressed, and she was surprised by their reaction, as I told her earlier this is the absolute minimal productivity for a real test automation engineer.
Tip: Many fake automated testers used a wrong framework such as Cypress and Playwright, or a wrong syntax framework such as Gherkins. I have used many frameworks in many languages, only raw Selenium + RSpec using TestWise tool fits the high-efficiency requirement. Check out my formula (I used over 11 years) that my daughter used.
2. All automated regression tests must be run at least once per day in a Continuous Testing server.
Regression testing is a moving target. Every change of code, test data, test scripts or infrastructure scripts can break unknown business features in the app. The only practical way is to run the whole regression test suite often, and resolve the issues quickly at the highest priority.
A common mistake is to use a CI Server, such as Jenkins or Bamboo, to run automated End-to-End tests. This is wrong (unless some CT features are added, like what I did to CruiseControl back in 2009), should use a Continuous Testing server instead, such as Facebooks’ Sandcastle or BuildWise. Check out this great presentation, “Continuous Integration at Facebook”.
3. There is a limit by capability.
Experienced engineers would know that the effort for maintaining an automated test suite grows exponentially with an increasing number of test cases. The effort of maintaining a 50-test suite is NOT double of a 25-test suite, more like five times.
It takes time for automated testers to grow the capability to manage a sizeable automated regression suite, on a daily base. Check out the AgileWay Continuous Testing Grading, and set a realistic target. For most software teams, Level 2 (50 tests) is already good, better than 90% based on my estimation. If you are ambitious, seek professional help from a real test automation coach.
Related reading: