Why “Test Replay” in E2E Test Automation is Not Useful in Real Agile Projects? Part 1
This kind of ‘feature’ is good for demonstration, not for Real E2E Test Automation.
The latest Cypress release (2023–09) introduced a new feature, “Test Replay”.
Test Replay: let testers explore, run and maybe even debug tests with a time travel experience complete with watch mode.
The marketing team of cypress.io even wrote an article on its Press & Media page, titled “Cypress Unveils Test Replay, Bringing New Speed and Efficiency to Automated Testing”.
cypress.io is wrong again! This article explains why.
Let me start by briefly touching upon Real Agile Projects: a quote from Kent Beck (father of Agile)’s classic book.
I have been conducting ‘daily production releases’ practice for all my own apps since 2012 (and some clients’ ones). To me, it is so natural to software development. The enabler of frequent production releases is real Continuous Automated E2E (via UI) Testing. (see my formula, 100% free, W3C standard-based and open-source frameworks: raw Selenium WebDriver + RSpec)
The cypress.io marketing team cleverly used the word “New” in a somewhat misleading way. Its competitor ‘Playwright’ has already had a similar feature called “UI mode” for a while.
By the way, I implemented a “View Testing Snapshots” feature (like “Playwright UI Mode”) in TestWise IDE (a next-gen functional testing IDE created by me) 15 years ago.
This is not hard to implement, but soon, I realized it was only good for demonstration (yes, the audience liked it), with very limited real value (see below). So I removed this useless “Test Replay” feature and implemented “Keep the Browser Open after Executing an Individual Test” and “Attach test execution to the existing browser” and associated CT features 👍, to achieve the great efficiency needed for Real E2E Test Automation (example below).
Some Cypress testers might argue ‘Cypress Test Replay’ is different (from Playwright UI mode): “It is a replay of what occurred in CI (Cypress Cloud)”. Long-time readers might recall my article “Why Recording Videos for Automated Test Execution in Test Scripts is Wrong?”. A few years back, some vendors (and fake automated testers) hyped ‘view test execution videos’, and I predicted it was wrong. It turned out my prediction was correct again. This time, I would say, this so-called “Cypress Test Replay” was an improvement, but still no good. Better not to use it.
According to Cypress doc, “requiring them (engineers) to reproduce the issue (failed in CI) locally” seems a bad thing. No, as a testing engineer, the most effective and sure way is to reproduce the issue locally; otherwise, how can a test automation engineer fix and check in the changes with confidence? It is expensive (mostly, about time) to do another run of the whole suite. It is common sense for engineers to verify by solid results, not “it seems …”, right? Engineering means no guesswork.
Also, Cypress’ Test Replay is only available to Cypress Cloud users. This is a sign of pushing testers to their not-good paid service. (As I predicted, see “cypress.io is dying”) .
“Test Replay” is good for demonstration, but not useful in real E2E Test Automation.
Some readers (after watching the Cypress TestReplay video or playing with Playwright’s UI mode) might think, “This is cool! How can you say no?”
My definition of Real E2E Test Automation:
At least 100 E2E (via UI) User-Story Level Automated Tests.
Testing real business scenarios like an end-user. Assuming one user story has one or two end-to-end test scenarios, that’s the minimum, right? Otherwise, how can a project claim ‘Agile’ without “Done, Done”?
Please note, not so-called “Component tests” and API tests. End-to-end!The whole test suite is run at least once daily, and the scripts are up to date with the app, i.e., valid.
Unmaintainable automated tests are of no use at all.The team has high confidence in E2E Test Automation.
A good example is that the team can release to production often after getting a green run on the E2E test suite.
The above is quite reasonable, don’t you think? For readers who doubt that is possible, check out this showcase.
With the above information, it is easy to reason why “TestReplay” is usually not useful. It’s somewhat reminiscent of elements in life, much like the ‘manual mode in an automatic gearbox.’ When purchasing the car, I made this a must-have feature (somehow, I was fascinated about it), but in the last eight years, I’ve likely used it only a handful of times.
In Part 2, I will delve into the reasons behind this assessment: “Test Replay” is not useful in real E2E Test Automation.
Related reading: