My Innovative Solution to Continuous Testing: Auto-Retry

A Continuous Testing feature that greatly reduces false alarms and increases the chance of getting a green run.

This is included in the “My Innovative Solution to Test Automation and Continuous Testing” series.

The automated tests here I refer to are Automated End-to-End (via UI) tests for web/mobile/desktop testing, which will be raw Selenium WebDriver/Appium tests for me. Most Automated Testers nowadays have heard of the “Test Retries” feature (in Cypress and Playwright).

I consider myself a pioneer in ‘Test Retries’, although some may perceive it as boasting. As with most assertions, I back them up with concrete evidence. I conceived the idea and successfully implemented it in 2009 (predating the existence of Cypress & Playwright) using a customized CruiseControl Server, the grand-daddy of CI servers. For more, refer to “My Continuous Testing Journey.”

The reasoning behind the “Auto-Retry” concept is simple: if the overall non-test-script-related flakiness rate is 1%, retry it once, and then its stability rate is increased to 99X. Therefore, only needs to be done once. Also, the retry shall not be immediate. Why? If a test step fails due to a heavy server load, retrying 0.1 seconds later is mostly meaningless. It is obvious, isn’t it? That’s why Cypress and Playwright's retry approach is wrong; it shouldn’t be the framework’s concern. It is CT’s.

Cypress’s wrong retries approach, image from its official doc.

Following the deprecation of CruiseControl, I developed BuildWise, a free and open-source Continuous Testing server built from scratch, featuring essential CT features, including “Auto-Retry”. BuildWise server won second prize at the 2018 International Ruby Award in Japan.