Why Cypress Sucks for Real Test Automation?
Avoid this recipe for test automation failure.
Note: The sequel to this article “Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)” is now available.(2021–02–15)
Update: A related article “Why JavaScript Is Not a Suitable Language for Real Web Test Automation?” is now available. (2021–04–15)
Update: A related article “Cypress vs Selenium WebDriver Comparison by Example” is now available (2022–06–22)
Cypress, a proprietary JavaScript end-to-end testing tool, seems to me that it is an improved Selenium v1 (also based on JavaScript), which was a big hype in 2005 (until it was merged with WebDriver in 2009). I believe many testers had not used Selenium v1 before, what I could tell you is that being replaced by Selenium WebDriver was a good thing.
As a test automation & continuous testing coach, I visited many software projects. I have never seen a single success using Cypress, not even Level 2 of AgileWay Continuous Testing grading (in ‘Practical Continuous Testing’ book), that is:
A regression suite of 50+ user-story level automated UI tests
The test suite runs in a CI (or CT) server multiple times a day as regression testing. (otherwise, there was no point doing it)
while coping with frequent application changes (reality, right?)
At least one green build (passing all tests, or with explainable causes) a day.
On a green build, the team has high degrees of confidence to release to the production. (that’s the ultimate purpose of doing test automation, isn’t it?)
How do you score? As you can imagine, Level 3 (200+ tests) will be much harder. All successful test automation implementations I have seen (up to Level 4, 500+ automated UI tests run in CT daily) were all using Selenium WebDriver/Watir + RSpec. I heard of Facebook's test suite is huge, if you are interested in large-scale test automation, check out this F8 presentation by a Facebook engineer. By the way, Facebook uses Selenium WebDriver.
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ”- Katie Coons, software engineer at Facebook, “Continuous Integration at Facebook” presentation at F8 2015.
In test automation history, there were full of false hypes. Just name a few:
HP QTP -> Micro Focus UFT -> Micro Focus LeanFT (to support Selenium WebDriver). QTP/UFT has been a long history known as ‘the leader’ in test automation. Since 2010, I have started questioning QTP (at my conference talks and consulting). The simple reason was, every QTP project I saw was a complete failure. Micro Focus is now a “Selenium level sponsor” of Selenium WebDriver.
Phantom JS (for headless). Between 2010 and 2015, test architects talked highly about Phantom JS. I tried, it puzzled me how could that possibly work? I wrote down my thoughts in my book “Selenium WebDriver Recipes in Ruby.” In 2017, Phantom JS was deprecated, according to Vitaly Slobodin, the main maintainer: “(Headless) Chrome is faster and more stable than PhantomJS. And it doesn’t eat memory like crazy. I don’t see any future in developing PhantomJS.” It turned out that those ‘we tested with Phantom JS’ were just lies. (if the test automation framework is not reliable, ….)
Protractor (JS). It was quite popular (between 2015 to 2019) with the rising of AngularJS, it is now deprecated. Check out my article: “Protractor, another Automation Framework I rightly avoided, is being Deprecated”.
Why is Selenium WebDriver the automation framework for testing web apps? I guess the best answer to that is this announcement from Microsoft in December 2018.
Microsoft acknowledged its tool (Coded UI Test) was no good, (by the way, I met a few hyped about Coded UI Test as well), compared to Selenium (or Appium for desktop app testing). Microsoft finally deprecated Coded UI Test and recommended Selenium instead. In fact, some Microsoft engineers have started recommending Selenium (over its own) since 2016. If you are interested in the reasons behind it, read the “Why we are deprecating Coded UI Test” part of this article by a Microsoft Engineer.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — from the presentation of Damien Sereni (engineering director at Facebook) at Selenium 2013 Conference
I assume that most will agree that Selenium Works. It is a W3C standard (this means: feature complete and no vendor-locking), more importantly, the only framework supported by all major browser vendors (this means: future-proof). Some may argue that it is difficult to learn, Wrong! Selenium WebDriver is the easiest to learn, far easier than others. I mean: raw Selenium WebDriver, not with some added custom layer (often marketed as ‘own test framework’, which always ended badly). I mentored many manual testers. Most of them could write real Selenium tests for work the next day. Of course, mastering one skill takes time, but learning can be quick. (If students have the confidence to practice and grow, the teaching/learning is considered successful.)
Let me repeat, under proper guidance, you can learn Test Automation with Selenium WebDriver within one day. My daughter started writing real and useful Selenium tests at the age of 12, so could your child, see “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes.” The reason: Selenium WebDriver syntax follows an intuitive pattern: “find an element and drive it.”
The ‘intuitive’ is important for developing/maintaining medium or large regression test suites. Fake test automation engineers probably never reach the stage of maintenance. Unmaintainable automated test scripts are useless! However, those fake automation testers just focus on smoke-and-mirrors demo, Cypress is good for them.
This post is about why I think Cypress sucks, not how great Selenium WebDriver is. For more, you may refer to my two books, “Practical Web Test Automation” and “Selenium WebDriver Recipes in Ruby”, both were listed in the best selenium books of all time by Book Authority.
I understand why cypress appeals to some who do not understand real test automation and scripting. I agree, Cypress looks good for demonstration purposes, but not for real automation. The so-called ‘nice features’ of Cypress are either actually not useful when addressing the real test automation challenges, or there is a far better way with Selenium scripting. Let me illustrate:
Nice debugging features.
Both Cypress and Puppeteer are built based on Chrome DevTools Protocol (CDP), a protocol to enable a debugger inside Chromium-based browsers. CDP has some issues and limitations. If you are interested, read this article by Simon Stewart, creator of Selenium WebDriver. Anyway, Selenium 4 will support CDP, and most likely be better than what you can do with Cypress. Even with Selenium 3 (without CDP), testing IDEs such as TestWise offer very good debugging capabilities (see below or this article for more).Record.
The recording is actually bad for test automation, and above all, it is not necessary. Strangely, the recording has been criticized for over 20 years, and it keeps coming back in another form. I once developed a recorder (TestWise recorder) for fun, so I do have a good understanding of it. The harm of using a recorder far outweighs its benefits. My suggestion: don’t use the recording. As a matter of fact, with a good testing IDE and proper training, recorders are hardly needed. (I developed and maintained thousands of automated tests for a number of projects over 15 years, I only remembered only one case that a recorder was helpful). Cypress, in my opinion, is somewhere between traditional record-n-playback and (better) raw scripting. While many people choosing to use Cypress Studio or cypress recorder is not Cypress’s fault (the same goes for Selenium IDE for Selenium WebDriver), the nature of how Cypress works is encouraging testers to choose a convenient locator (rather the most reliable one), this leads to maintenance problems.
“Record/Playback scripts are notoriously costly from a maintenance perspective.” — ‘Agile Testing’ p315, 2009
Wait for elements.
This can be easily implemented in scripting. Testers who attended my selenium training know this:tfTab. Done. By raw scripting, you have full flexibility to control the wait, much better than cypress. See the article ‘Test AJAX Properly and Efficiently with Selenium WebDriver’ (with video).Productive, which is false.
Every Cypress tester I met was very poor in productivity. Selenium WebDriver, on the other hand, with the right tool such as TestWise, can be super-productive. I repeatedly was 10x or more productive than a Cypress tester when working on the same set of tests. Check out another article: “Step by Step Showing Why Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework”, a video is also included.
Executing selected test steps against the current browser is possible with Selenium. As you can see in the above screenshot, the user may freely add/change test steps to run against an attached browser. The animated GIF shows the “run steps against the current browser” action.
Disclaimer: I am NOT trying to promote my tool TestWise here. The test script does attach the browser using the
--remote-debugging-portoption with ChromeDriver. You can view the test helper script here. In other words, what I show is 100% free and open. TestWise is just one tool utilize this, no dependency there. You may add this support to your tool, for example, an extension to Visual Studio Code. Freedom is Good!
Easy to learn, which is false.
Cypress is much harder to learn, compared to Selenium WebDriver. If Selenium WebDriver is taught well, a 12-year-old (verified by my daughter) can learn basic Selenium to be useful in minutes. Check out this article.JavaScript, which is wrong.
Node.js, thanks to the v8 engine, is very fast. For UI testing, the raw speed of a language matters a little. However, JavaScript language is not a good testing language. JavaScript programmers would strongly disagree, isn’t this a problem? We are talking about functional tests here. The audiences of functional tests are mainly business analysts/customers/manual testers. Ruby is a far better choice, and it has long been proven. (Watir = Web Application Testing in Ruby, please note, the authors intentionally put the language in the framework name).
I like scripting Selenium WebDriver tests in Ruby. It is efficient, creative, and fun! I dislike the limitations of proprietary test frameworks. For people who are interested in learning about some of Cypress's limitations, read this article by Alister Scott.
In the true spirit of test automation, whether an automation framework is good or bad, is judged by the result, the cold hard binary result: 0 or 1. It is good to see new test frameworks or tools, and competition is good. In that sense, I welcome and actually tried Cypress, PhantomJS, Selenium WebDriver in all 5 languages, and others. Once I was asked to score available test frameworks, which I didn’t expect, as Selenium WebDriver shall be a no-brainer. Later I came up with a set of criteria (such as reliability, flexibility, maintainability, cost, ease of CT integration, …, etc. like judging for the car of the year) and gave my score for each framework.
Selenium -WebDriver: 99 (out of 100), Cypress — 58, Ranorex — 29, UFT — 42
Update (2021–01–29): Some Cypress testers claim Cypress is fast. I tried a simple example, did not notice its advantage compared to Selenium. But I didn’t want to waste more time on Cypress, so I moved on (sorry, I heard so many lies from so-called ‘Cypress Automated Testers’). Giovanni Rago did a thorough test comparing the performance of tools under different scenarios: Selenium vs Playwright vs Puppeteer speed comparison. This test result shows: not much difference in terms of speed for 4 frameworks, however, Cypress was the slowest on most categories.
Maybe one day, cypress will be more open, actually works, and with fewer limitations. So far, I was not convinced at all. If your cypress implementation enables your project to release the production daily, then you are successful, ignore anyone who tells you otherwise. If not, try Selenium WebDriver with RSpec, and embrace freedom, productivity, and fun!
Lastly, I share an execution of 608 selenium tests in the Continuous Testing server yesterday.
And of course, production deployment after the green build, and I have been doing daily-production-deployment (while changes are made to the app) for over a decade. The test suite has over 30,000 test steps (every step matches a user operation). To get a green build, each of every 30366 test steps must pass.
+--------+-------+---------+---------+-------+
| TEST | LINES | SUITES | CASES | LOC |
| | 27508 | 260 | 612 | 21308 |
+--------+-------+---------+---------+-------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 11697 | 260 | 2065 | 8707 |
+--------+-------+---------+---------+-------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 459 | 1 | 45 | 351 |
+--------+-------+---------+---------+-------+
| TOTAL | 39664 | | | 30366 |
+--------+-------+---------+---------+-------+For this app alone, I have been conducting Continuous Testing for nearly 7 years, with over 600,000 test executions.
The development, testing, and customer support of this ClinicWise app, along with several of my other apps such as TestWise, BuildWise, SiteWise, and WhenWise, are done all in my spare time. It is true that real Agile/DevOps, backed up by real CT, greatly increases a software team’s productivity. Choosing a free and good test automation framework, such as Selenium WebDriver, and avoiding a bad one such as Cypress, is a good start.