JavaScript Web Test Automation Framework Rankings in JavaScript Rising Stars
My Thoughts on the rankings between 2019–2023.
Early this morning, one mentee sent me this message.
A link to test automation framework ranking on JavaScript Rising Stars.
Compared to 2022 (last year), Cypress dropped from the 3rd to the 5th, with exactly half of the stars.
Before Playwright (the first version was released in February 2020), Puppeteer was the leader.
In 2020 (the same year Playwright was available), Playwright took the crown in Rising Stars, and kept it ever since.
What do we learn from the above data?
1. JS testers keep “trying and dumping” web test automation frameworks.
This behaviour is strange and very wrong, as our target (the web) has stayed the same since 2014 (HTML5 Release date, from a testing perspective, not much changed since HTML 4, 1998). In theory, there is no objective need for automation framework changes.
A real End-to-end Test Automation Engineer just needs to use one Web Test Automation Framework (regardless of language) to test all websites in the last 13 years.
Regarding the web automation framework change rate, once per decade is acceptable. However, we have witnessed nearly every one or two years in the JS testing community. That is crazy!
As a test automation coach, I witnessed and rescued test automation attempts; the failure rate of the above JS automation frameworks is 100%. (See Definition of End-to-End Test Automation Success, and my boosted article, A Tale of a Deceptive End-to-End Test Automation Engineer)
2. Playwright took the JS automation frameworks by storm.
Playwright has topped the JS test framework rising star chart since it became available in 2020. Looking at it from a different perspective sheds light on the shortcomings of those (and many of them) JS automation frameworks. If a web automation framework works very well, why do you want to change it?
As a test automation engineer (since 2005), I only changed the automation framework once: Watir => Selenium WebDriver (available in 2011).
3. The highly-hyped Cypress is going to disappear
Despite an extensive marketing campaign, including the ‘successful’ (in terms of gaining popularity) Cypress Ambassador program, which is becoming an embarrassment badge, Cypress is on its way to disappearance.
It Seems Most “Cypress Ambassadors” Don’t Love Cypress as They Claimed
Money does not lie, People do. A fair percentage are hypocrites.
Check out my article, “Cypress.io is dying”.
Selenium WebDriver has always been the correct choice.
Two recent independent surveys showed a similar result: Selenium WebDriver is the framework of choice for web test automation, by a huge margin. In fact, the later survey shows Selenium is extending its already-dominant lead.
Examining the Playwright’s “stars count” over the past three years reveals a noteworthy trend: despite maintaining its top position in the JS test ranking, the numerical value is, in fact, on a decline.
Combining the two survey results, my statement, “Selenium WebDriver has always been the correct choice since 2011”, is justified.
What do I think of the Playwright?
Before sharing my thoughts, you should know that I added support for Playwright to my tools TestWise IDE and BuildWise CT server.
Why didn’t I do that for PhantomJS, ProtractorJS, TestCafe and Cypress? Because upon brief examination, I thought those were bad and had no future. I don’t want to waste time on them. Having said that, I think Playwright is still falling behind raw Selenium WebDriver in most aspects, such as
Easy to learn
Check out my article, “Why Selenium WebDriver is So Easy to Learn?”Reliable
You can’t beat a W3C standard-based.Syntax (intuitiveness)
Check out my article, “Playwright vs Selenium WebDriver Syntax Comparison by Example”.Support
My experience with WebDriver support is top-class.Scripting language.
Playwright, compared to other JS-initiative frameworks, added support for Python, Java, and C#. But it missed the most important one: Ruby.
There are some Playwright features I do NOT like, such as ‘auto-waiting’, ‘test runner’, ‘reporting’, ‘parallel execution’, .., etc. I will write separate articles on these topics.
Still, Playwright is promising if you, unfortunately, must use JavaScript or TypeScript; I hope it can unite the JS tester community and reduce the ‘trying-and-dumping framework’ frenzy. Over the past six years, fake automated testers using various JS automation frameworks (most notably, bad Cypress) have severely damaged the reputation of Test Automation Engineers and Web Test Automation. Now, there are fewer job ads for E2E automated testers, and with less pay and less respect. This is sad because E2E test automation is in big demand, and good E2E test automation engineers are considered to be “Gold” at top software giants.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Patrick Copeland, Google Senior Engineering Director, in an interview (2010)
The only way to rescue our (test automation engineers) reputation is to do the job well and, more importantly, do it REAL! No FAKING!
As mentioned before, we don’t need that many automation frameworks for testing web apps. Just make it work.
35-Word Functional Test Automation Strategy
Write 1+ Automated End-to-End Test for a user story;
Add it to the regression suite;
Run the regression suite daily and keep them valid!
Starting on the first day and then every day!
When we truly dedicate ourselves to changing based on the testing feedback, the work becomes interesting, challenging and fun.
Related reading: