Execution Speed of Automated End-to-End (via UI) Testing Clarified

Test Execution Speed is just one factor of many in Test Automation.

running image credit: https://www.vecteezy.com/vector-art/6923598-running-man-abstract-logo

This article is one of the “IT Terminology Clarified” series.

Execution Speed is an interesting subject in Test Automation, here, I mean, end-to-end via UI. For example, web test automation, the most common form of test automation, drives the app in a Chrome browser for testing purposes.

Table of Contents
· The Cypress Lie on Test Execution Speed
· From an overall perspective in test automation, speed is minor compared to other factors
· Speed is important in the developing/debugging phase.
· Speed is not that important in the daily execution phase.

The Cypress Lie on Test Execution Speed

Since ~2018, Cypress’ marketing team has lured many fake automated testers using a big, fat lie. (Check out this article)

A false claim is still on Cypress’s website

I don’t know how Cypress got this conclusion. It is against my observation, besides every Cypress test automation attempt (I witnessed) failed, in the process of rescuing some of them (to use raw Selenium WebDriver), I found Cypress was slower than raw Selenium. Maybe the people before me did not develop selenium tests properly. Check out my article: Optimize Selenium WebDriver Automated Test Scripts: Speed.

Some Cypress defenders would argue otherwise, saying “blah, blah”, but no one showed me a solid and convincing result yet. Yes, there might be cases Cypress was a bit faster on a particular website (due to some optimization, we all know that trick).

As a software testing company, Cypress shall at least show the execution results and the test suite to back its claim, right? None. Just a statement followed by marketing words. Sadly, so many fake automated testers fell for it. If some of them stick with the right track, raw Selenium WebDriver, they might grow and become a real test automation engineer.

There is more than one benchmark test to show otherwise, such as

• Cypress vs Selenium vs Playwright vs Puppeteer speed comparison.
  “Cypress’ high startup time might interfere with high-frequency synthetic”

• On Migrating from Cypress to Playwright
  “My Playwright test suite runs 34% faster than the equivalent Cypress tests on CircleCI. On my local dev machine, Playwright gives a 5x speedup over Cypress.”

I expressed my opinion, “Cypress is actually slower” in “Why Cypress Sucks for Real Test Automation?” article two years ago.

From an overall perspective in test automation, speed is minor compared to other factors

“You see only the trees, but not the forest.” — Korean Proverb

People can easily focus on one part but not see the big picture. There are many more important factors than speed in test automation. Here, I will just list two, that most will agree with.

• Reliable
  There is no point in running automated tests if test execution fails intermittently (by the framework). It a common sense.
  
  Raw Selenium WebDriver, of course, scores the best on reliability for two simple reasons: based on a W3C standard and supported by all major browser vendors.

• Feature-Complete
  Web Testing means the framework shall support all web operations defined a user can perform, with no limitations at all! Cypress is a typical bad example, with a list of severe limitations. I have seen one silly automation attempt to use Cypress to test a website using many frames! That’s before Cypress’s so-called support-frame workaround.
  
  Again, Raw Selenium WebDriver scores the best on feature-complete for a simple reason: based on a W3C standard. All web technologies, such as HTML and CSS, are defined by W3C.

Here is an interesting Chinese idiom story on speed, #20 Opposite effect to one’s intention (南辕北辙).

“Browsers such as Chrome update regularly, and Web technologies will evolve. When that happens, the browser vendors, e.g. Google and Apple, will ensure Selenium WebDriver still works, not caring about Cypress or Playwright.” — Zhimin Zhan

Speed is important in the developing/debugging phase

When a test automation engineer develops/refines/debugs an automated test script, slow test execution can be quite annoying. Because we often need to try multiple times to get it right, and then refactor it. Nearly every script change requires a test execution; certainly, faster execution helps.

You can improve that.

1. Write faster test scripts.
   - Optimize Selenium WebDriver Automated Test Scripts: Speed.

2. A better Test Design
   A well-designed test script is easier to read and, maintain. You find out issues quickly, therefore, reducing the number of executions.
   - Maintainable Automated Test Design

3. A better Test Design or Utilities
   - ‘The Simpsons’ Data Reset Pattern for test data, can greatly reduce test execution time.

4. Using a more productive testing tool, such as:
   - Keep the Browser Open after Executing an Individual Test
   - Attach test execution to the existing browser

Speed is not that important in the daily execution phase

Most software teams are unable to achieve daily automated end-to-end (via UI) regression testing.