Comments on the claims of “7 ways Cypress is different”. All False, Wrong or Lie

Testers: Learn to test a testing framework/tool’s claims

On Cypress's website, it claims (still there, as of 2023–07–03).

Most Cypress testers should have known this: more and more people switched (or are about to) to Playwright. When I say something (as a former research scientist and a testing engineer), I will quote results to support my statement. Below are Google results for “cypress playwright migrate” or “playwright cypress migrate”.

See the movement! There is one Reddit post from the Google results with the title “Cypress.io is about to die, you should migrate your projects”.

“But Cypress is most likely going to close down soon.
…
They were already having trouble converting free users to paid users, but then Playwright came along and literally destroyed them.
…
A lot of folks hated Cypress because it was limited, you had to pay up in order to properly integrate it into your pipeline.
…
The propaganda Cypress did to promote their tool is literally taken out of the Soviet Propaganda manuals.” — source

“Cypress is literally dead. Playwright came and took their lunch.” — another Reddit post said in a funny way.

Here I want to point one thing out, most comments above are referring to Cypress as the company and its behaviour, and only one is about the technical aspect of the tool. Here I am not speculating the future of Cypress.io (a downsizing announcement in Jan 2023 is real though, used the phrase ‘incredibly difficult, but necessary’ for the decision), only see Cypress as a technical failure as a test automation framework/tool.

Let me quote the conclusion of another article, which I read in my Medium feed last week.

“Conclusion
Migrating from Cypress to Playwright can significantly improve your end-to-end testing experience, offering better developer experience, faster test execution, and improved test stability” — from Luc Gagan’s article “Migrating from Cypress to Playwright”

So, the so-called “most developer-loved” is FALSE, i.e. failed assertion, at least lost to Playwright in the JS community alone. In this article, I will talk about Cypress’s “Soviet propaganda” that the Reddit post refers to.

Table of Contents:
· The Developer-Focused End-to-End Testing is Wrong, Obviously
· Claim 1: “does not use Selenium”, WRONG (they should have)
· Claim 2: “Cypress focuses on doing end-to-end testing REALLY well”, FALSE
· Claim 3: “Cypress works on any front-end framework or website”, LIE
· Claim 4: “Cypress tests are only written in JavaScript”, BAD
· Claim 5: “Cypress is all in one”, Not Good
· Claim 6: “Cypress is for developers and QA engineers” Not Really
· Claim 7: “Cypress runs much, much faster.” LIE
· The Missing Claim 8: “Cypress has severe limitations.”

The Developer-Focused End-to-End Testing is Wrong, Obviously

There is a fundamental error in the mindset behind that statement: End-to-end testing is not about the developers, instead, its audience is the whole team.

Since college, I was taught “end-to-end functional testing is a black-box testing, shall be done independently from developers”, and pretty much what I witnessed in my career. (I know some will say something about new Shift-Left testing, read on). In some Waterfall teams, testers are not allowed to talk to developers and conduct testing strictly based on the specification.

Some will say “We do Agile”. I know Agile well.

According to Kent Beck, Father of Agile, a real Agile team must have the technical capability to do daily production releases. I have been doing that for some client projects since 2008 and my own apps since 2012. Some long-time readers have read and seen test reports (running a big number of Selenium WebDriver tests as regression testing on a daily basis). This is not a show-off but rather to talk about real test automation from a successful Agile perspective.

Developer-loved test automation tools are often a bad choice. For example, during 1997–2015, Java was the dominant coding language. Many developers, including me (for a short while), wanted to use it for testing too. Thankfully, I met one world-renown top Java programmer, who pointed out an obvious thing, “test script should be in a scripting language”. I discovered Ruby, Wow! I think some Microsoft C# developers probably loved Microsoft Coded UI (using the same language and visual studio IDE), but it was deprecated in 2018.

Some would argue, “JavaScript is a scripting language”. Not long ago, Angular.js was the most popular web development framework, and its associated testing tool Protractor became hugely popular. Angular developers sure would love Protractor, as it has built-in support with Angular model in the test syntax, Can’t beat that! The result: Protractor was deprecated.

Software Testers’ job is to find developers’ defects. “Developer-Loved” and “End-to-End Testing”, when these two terms are used together, sounds not completely impossible, but does not feel right, does it?

Claim 1: “does not use Selenium”, WRONG (they should have)

The wrong statement “Selenium or Selenium-based is problematic” (see this article for reasons and proofs), we can understand, Selenium was seen by Cypress as a competitor, these are marketing words for a commercial company.

However, when I posted a post on LinkedIn (not in any group, a personal update).

Gleb (Distinguished Engineer at Cypress.io, an authority on Cypress) reshared with a comment, calling “everything I wrote (you are reading or have read) is a joke”. That’s unprofessional at least, right?

I wrote my article on my personal blogger and posted “It is featured in a reputable software testing newsletter” (i.e. was selected by a reputable curator who at least consented with professionalism). Even that, I received this personal attack. Suddenly, I could relate to the “Cypress Soviet propaganda” phrase in that Reddit post. My article refers to both Cypress and Playwright, also, I have made accurate predictions regarding the failures of Phantom.JS, Protractor.js, Cucumber, and several other tools. These opinions are purely technical in nature and should not be subjected to personal attacks, regardless of their accuracy. I have never received such verbal abuse from others. See, there is a difference.

I share the same feeling: ‘Angry’ as Lisa Crispin (author of the classic ‘Agile Testing’ book) when people throw evil mudslingings to Selenium for commercial reasons (Selenium is 100% free and open-source), over the years.

Lias Crispin (author of Agile Testing book)’s comment on bad marketing against Selenium.

All real successful (dozens of) web test automation, I witnessed and heard (trusty source), were implemented in Selenium WebDriver.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Image Credit: “Continuous Integration at Facebook” presentation and “The Practical Test Pyramid” article on martinfowler.com

Check out my articles for more. As a comparison, every Cypress test automation attempt (I witnessed) was a complete failure.

Having said that, Cypress as a tool (not as an automation framework), some of its features do have value. If Cypress reinvents itself to become a Selenium WebDriver tool (like the upcoming JetBrains’ Aqua), that will be a welcoming change, and may survive and make a positive impact.

Cypress testers: If Cypress is a tool for Selenium WebDriver, and hearing the rumours that Cypress.io (the company) might face the bankruptcy, you will be less concerned, right?

Claim 2: “Cypress focuses on doing end-to-end testing REALLY well”, FALSE

Of course, it is not. If it is really well, why do more and more JS testers dump it (so quickly) and migrate to Playwright? Remember, the target, the web, has not changed for nearly two decades. Cypress had enough long time to prove this claim, and it failed.

Again, every Cypress test automation I witnessed was a complete failure.

Claim 3: “Cypress works on any front-end framework or website”, LIE

Any website? NO! I don’t even need to quote references from others (“My Thoughts on Cypress.io” by Alister Scott on 2019–07–16, comment section).