Selenium vs X, Y, Z, …, all these years
Stop comparing, just do real web test automation with raw Selenium WebDriver. All web automation frameworks compared to Selenium are later proved to be the losers.
This article is one of the “Be aware of Fake Test Automation/DevOps Engineers” series.
Today, I noticed a LinkedIn update on an Applitools’ webinar (2021–12–10): “Selenium vs. Playwright”.
I did not watch it, and I will explain the reasons shortly.
On this webinar info page,
“we have long-time reigning champ: Selenium WebDriver”
then it will be followed by introducing the so-called relatively new framework: Playwright.
Here I want to highlight one perspective many people might have missed, why is it always Selenium vs X all these years?! (The Xs, see below, all proved losers later)
Selenium vs QTP/UFT
Examples: Software Testing Genius (2011), Software Testing Magazine (2013)
Fact: Micro Focus became a selenium sponsor of the Selenium project, and released a product (UFT developer) to support Selenium.Selenium vs Protractor
Example: Applitools (2014), its first sentence, “is Protractor easier than Selenium?, Yes”. Of course, that’s not true.
Fact: Protractor has been deprecated.
Protractor is a wrapper of WebDriverJS, not a framework. This proves my point: Raw Selenium WebDriver (not using JavaScript) is better. Check out my article: “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”.Selenium vs Microsoft Coded UI Test
Example: Spheregen (2015)
Fact: Microsoft deprecated Coded UI Test in 2018, “We (Microsoft) recommend Selenium WebDriver for testing web apps”.Selenium vs Cypress, Puppeteer or TestCafe
Examples: TestGuild (2017), Gartner (2018)
After seeing every Cypress test automation attempts fail, I wrote the article “Why Cypress Sucks for Real Test Automation?” in 2020. At that time, Cypress was still hot. Among the feedback I received, a certain percentage disagreed with me back then. Since this year, more and more feedback has shown that people have started to agree with me and shared their frustration with Cypress. Some even expressed that they wished they had read my article earlier.
From this year, it seems that every JS tester, whom I am aware of, has dumped Cypress and started to embrace Playwright. Some told me, “Cypress had too many limitations” which is not fair to Cypress. Its documentation has clearly listed its limitations. It is the fault of the person (fake test engineers) who chose an automation framework/tool with limited features.Selenium vs Playwright (2021 — )
…
Update (2022–04–29): a new article telling a story of a fake senior Playwright tester in a large tech company: An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’
Now it seems that Playwright is the new favourite of JS testers. Again, people like to compare it to Selenium. In this Applitools’s webinar (2021–12–10), Selenium WebDriver’s status was acknowledged.
“If you’re doing any automated testing, surely you’ve heard of Selenium WebDriver, the most popular web automation framework in the market.”
Have you seen the problems here? Since Selenium WebDriver’s release in 2011, after so many so-called ‘new and better’ frameworks have come and gone over the last 10 years, Selenium WebDriver remains its status. In other words, most (if not all) engineers who have claimed using QTP, Protractor, Puppeteer, TestCafe, or Cypress are actually faking test automation. If so, why should we believe Playwright would be different?
Shouldn’t software projects get real with test automation using Selenium WebDriver like Facebook, instead of juggling automation frameworks (and failed)?
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
If Facebook could implement real DevOps enabled by real Continuous Testing 8 years ago, and web technologies and Selenium have barely changed since, what are the excuses? It is certainly not about the cost, right? (Selenium is 100% free)
One possible weak excuse is “Facebook/Google did real CT because they are the top IT companies in the world. As a small/medium-size firm, we lack the resources (human and infrastructure)”. Wrong! Below is a recent run of WhenWise (one of my web apps) regression suite (a total of 548 raw Selenium WebDriver tests ) in a BuildWise CT server.
Another excuse: “that’s because you are an accomplished test automation coach, speaker and author”. Still not correct, Test automation and Continuous Testing is feasible for any company with visible outcome in days if using the free Selenium properly, with little cost. Here is a CT server my daughter (a uni student) set up and ran 43 selenium tests daily for a Uni group project ( daily production release).
Why a teenager could reach Level 2 of AgileWay Continuous Testing Grading, better than 95% of test automation engineers? The reason is simple: she used raw Selenium WebDriver + RSpec, and followed sensible and proven practices, such as maintainable test design and running tests in a real CT server (not Jenkins or Bamboo, which is a CI server).
Over the last 10 years, I have heard many evil mudslingings against Selenium WebDriver. On some occasions, I proved those ‘engineers’ wrong instantly, such as ‘Selenium unable to automate React app’ or ‘Selenium does not support AJAX’, by developing a real test against their apps using raw Selenium WebDriver in minutes. I have worked in test automation since 2005 and trained/mentored many manual testers to do test automation. Selenium WebDriver excels over others in every way, including being easier to learn. My daughter started writing raw Selenium tests at the age of 12. With proper guidance, your child could do it too. (please note, mastering it, or anything, is not easy. For example, to achieve Level 2 of AgileWay Continuous Testing Grading, it will take about one solid year (which will take longer without good mentoring) of full-time hands-on test automation. But that is about test automation and CT knowledge, not framework syntax)
My daughter recently started her intern job. On her first workday, she created two automated tests. Her team leader was deeply impressed and surprised that a Uni student could get the work done with a speed that the team had never seen as manual testing is the norm in this large company. The team leader asked her what framework she was using. My daughter replied: “Raw Selenium WebDriver.”
Why raw Selenium WebDriver is so easy to learn? Selenium WebDriver’s syntax follows an intuitive pattern “Find Element and Perform Action on it”. Often, fake test automation engineers do it wrong to complicate things, such as creating their own formwork on top of Selenium. Seeing is believing, check out this article “Step by Step showing how to develop raw Selenium WebDriver test scripts in minutes”, with a video.
Q & A
1. What’s your opinion on Playwright compared to Selenium?
I have already shared my view in this article, Playwright with Mocha against Chrome browser. Don’t waste time on Playwright, Selenium is far far better, in every aspect. Check out the AgileWay Test Automation Formula.
2. We tried Selenium WebDriver, but failed as well.
This is because your engineers never implemented real successful test automation. They lacked sufficient knowledge (and practices) of test automation and continuous testing, so switching automation frameworks won’t help. Their experiences in UI test automation were all failures. Just verify them against the Definition of End-to-End Test Automation Success.
Those engineers thought they knew a little about test automation after creating a few hello-world tests, then very soon they couldn’t manage it, with problems such as unstable test execution, hard to maintain, test data issues, running unreliably in Jenkins, …, etc. Does this sound familiar? They usually made these common mistakes:
Automated Test Scripts are not in the Syntax of a Scripting Language, such as Java and C#.
Using a non-genuine automation framework that is not 100% real (such as TestCafe) browser or with limitations, such as Cypress.
Created a wrapper syntax on top of Selenium, claiming ‘designed a new framework’. A classic example is the failed Protractor.
Bad test design that is hard to maintain (see a simple and good approach: Maintainable Test Design)
Run tests in a CI server such as Jenkins or Bamboo. (instead, shall use a CT server)
…
Any one of the above mistakes is enough to ruin your test automation attempt. You need to seek external help from a real test automation and continuous testing coach.
Back to the framework, you will be in a much better position to address the above test automation challenges with raw Selenium WebDriver in Ruby.
3. You are biased, it seems that you did not have much experience with Playwright.
I will not debate that, otherwise, another Selenium vs X article. Test automation is based on the result, objective result. If the test count of your UI test suite is <50 and they don’t run reliably every day, then your definition of test automation success is very different from mine. Check out this article: Definition of End-to-End Test Automation Success.
I will state the indisputable fact: in test automation, the maintenance effort is far more than test creation. Check out this article: “Is Your Test Automation on Track? Maintenance is the key”
I haven’t yet met a single real test automation engineer who uses automation frameworks in JavaScript. Trust me, I really want to see one working. So far, no luck for 15 years. If you wonder why, check out this article: Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
4. You are narrow-minded, only used Ruby, and probably never tried JavaScript.
Wrong. I am more objective and open-minded than most test engineers as my Selenium Recipes Book series covers all five official Selenium languages.
Educative likes my Selenium WebDriver Recipes in Node.js book, and converted it into an interactive course. Just over a week ago, I published an article on Playwright: Playwright with Mocha against Chrome browser.
5. You are saying this because of your commercial interests, your tools support only WebDriver/Watir.
Wrong. First of all, let’s be clear about the commercial interests. 99.99+% people go to work for commercial interests, more or less. Let’s be realistic, don’t compare me, a professional engineer, against the saints.
“Would you coach for free?”
“Yes”
“But do you?”
“No”
- excerpt from Apple’s hit TV show “Ted Lasso”
WebDriver is a W3C standard, a commercial software supporting the industry standard (purely without any modifications) is always welcome, right? You may call software locking people into a proprietary syntax evil. I have been against vendor-locking test automation frameworks and tools since 2009 (at software testing conferences, and publishing books).
I created two tools, with commercial and free licenses (please read below):
TestWise, functional testing IDE
TestWise supports only free, open-source test automation frameworks such as Selenium, Watir and Appium, since its first day in 2007.
TestWise was demonstrated at the Agile 2009 conference’s AAFTT workshop in Chicago, listed as the first testing tool by Agile testing book author Lisa Crispin, and a finalist in the 2010 Ruby Award.BuildWise, Continuous Testing server
BuildWise server is free and open-source. BuildWise received the 2nd prize in 2018’s Ruby Award from Matz, the creator of Ruby, a true software legend.
I have spoken at several international software testing conferences demonstrating test automation and CT techniques using these two tools. Those experts don’t judge my tools with a narrow view.
In fact, if some bother to try them, they will find out
TestWise and BuildWise support a wide range of free and open-source frameworks in Ruby, Python and JavaScript, including the ones I don’t recommend such as Cucumber.
I learned from Kent Beck, the Father of Agile.
“I hated the idea so I had to try it” — Kent Beck
Both TestWise and BuildWise come with a generous (non-crippled) free mode (as in ‘free beer’).
The free mode is only with minor limitations.
- BuildWise Agent: 40 min (like Zoom) uptime; after that, relaunch it (took ~3 seconds).
- TestWise: 15 test execution limit; after that, relaunch it (took ~5 seconds)
I received a couple of thank-you emails from testers in third-world countries, they can use testing tools legitimately and freely.
Finally, we need to be clear about Free Software. I met Richard Stallman, Father of Free Software, in 1998, his words changed me. Free software is about freedom, not the price.
To understand the concept, you should think of “free” as in “free speech,” not as in “free beer.” — GNU
I created TestWise and BuildWise with this Free mindset. I turned down a few commercial proposals for adding proprietary syntax to my tools. I rejected several commercial contracts to provide test automation solutions in Java, C# or JavaScript, even using my tools. My typical answer: “If using JavaScript, it will cost you 10 times more, and 10 times of chances to fail”. Why don’t use raw Selenium WebDriver in Ruby, which is recommended by the Agile Testing book?
6. Do you mean Cypress and Playwright definitely will fail?
What I can tell you is that every JS tester I met does not understand test automation at all. Sadly, they thought they did. There might be some JS testers somewhere who can achieve Level 2 of AgileWay Continuous Testing Grading, that is, maintaining 50+ E2E user-story level automated tests that run every day. Even so, when working in a team, it is going to be hard. One of many reasons: JavaScript is a language that offers various syntaxes, and different people tend to write tests differently, making the test suite harder to maintain.
Please don’t forget, a teenager like my daughter could develop a couple of real tests and set up a CT server to run the automated tests in a CT process on Day 1. I told her this is just entry-level skills, because ongoing test maintenance will be far more challenging. How could she do that? because she has been doing exactly the same way (automation framework: Selenium, syntax framework: RSpec, tool: TestWise) since she was in middle school. (she only did some minor automation exercises during school breaks)
It is worth noting that you have 100% freedom with raw Selenium WebDriver, while with all others, there are some degrees of limitations, Yuck! Check out my article: Crazy Web Test Automation: “Freedom Is Slavery”.
The real question a JS tester probably shall consider is: “If Selenium is such an easy and proven solution, why do I bother with Playwright?”. Maybe in the next year, another new hyped JS automation framework “Playleft” (which claims good support for Shift-Left Testing) become popular, will you switch to it?
Related reading: