Expose a Common Lie, “Selenium is Flaky”, Part 1: Selenium WebDriver is much more reliable than others.
Don’t be fooled by fake automated testers and evil vendors’ marketing teams. On reliability, Selenium WebDriver is much better than any other web test automation framework.
Today, I saw two articles about Selenium, with the lie “Selenium is Flaky”.
Terminology: “Flaky” is just another word for “non-deterministic”.
“Love-Hate Relationship with Selenium Automation
tests written with Selenium can be prone to flakiness and unreliability due to the unpredictable behavior of different browsers.”
“Selenium advantages and disadvantages
3. Flaky tests: Tests written with Selenium can sometimes be flaky, meaning they may fail randomly due to issues such as elements on the page loading slowly or network instability.”
In this article, I will expose this common lie with facts!
Table of Contents:
· Selenium is super reliable, not flaky at all!
∘ A story of me exposing a fake tester: about test execution flakiness
∘ 1. WebDriver is a W3C standard
∘ 2. It has been proven in many large organisations
∘ 3. Testimonies from the world-renowned experts
∘ 4. My Continous Testing Reports
· Why this obvious lie is widely spread?
Selenium is super reliable, not flaky at all!
The lie, in my opinion, was created by bad tool vendors’ marketing teams and spread by fake automated testers, who typically keep cycling failed JS automation frameworks, PhantomJS → Protractor.js → TestCafe → Puppeteer → Cypress. From my experience, all these frameworks have some degrees of limitation or flakiness, but not Selenium WebDriver.
“Selenium WebDriver is the most reliable web test automation framework, not flaky at all!” — Zhimin Zhan
I rescued a few failed test automation projects using Protractor, TestCafe and Cypress, back to raw Selenium WebDriver. Faster, easier to learn (for manual testers), more readable, easier to maintain, far more reliable, in a summary, better in every aspect.
I know what fake testers mean “flakiness” in Selenium tests for dyanmic operations, such as AJAX. That was misunderstanding, I will clarify in Part 2. In the mean time, let me clear this low and obvious lie first.
I was a research scientist (developed a habit of citing at least two independent sources to support one opinion), and now a test automation engineer (switched in 2010 from a senior Java developer), I will provide numbers and facts to support my claim.
Before I show the cold hard facts (which will break the fakers’ lies and hearts), it is worth pointing out the foundation of automated test execution.
If one test step is flaky, the whole test script is not good.
If a few test scripts are flaky, the whole suite is almost certain to fail!
A story of me exposing a fake tester: about test execution flakiness
At one company, a newly-joined tester D of another division came to me for help, and I did. I sent him all my test scripts, including the tutorial ones, along with some materials, and also answered a few questions by email or in person. He thanked me repeatedly.
One day, the test lead of that division invited me for a meeting to discuss test automation before my leaving. This tester D was in the meeting room, and somehow, I found he was nervous. Soon, I knew the reason. When his team leader asked him some questions, he told lies, such as not being aware of my test scripts (which I sent him and answered his questions on that). The mentee, who came to the meeting with me, was furious because she witnessed all my help to D.
I decided to give this fake tester and liar a lesson.
I asked D, “What’s the average reliability for your automated tests? For example, if you run a random test script 100 times, how many will pass?”
D couldn’t answer, “uh, eh…”
I hinted, “maybe 80% or 90%?”
A sign of relief showed on D’s face (he might think I was trying to get him out of the embarrassment, but it was a trap I laid for despicable fake testers, I only used this once), and he replied, “Yes, about 90%”.
I said firmly, “Then, you should stop working on test automation, it is already failed”.
D was shocked, and so were most people in the meeting room, except my mentee.
I continued, “Let’s do a simple math, show one slide (see below)”
“Your team lead, just said the goal is 1200 tests, not the 75 as shown in the slide. If one test reliability is 90% on average, the chance of getting a green build is 1 million times less than winning the lotto”.
D’s face was red, and completely silent for the rest of the meeting.
This story will give you a perspective (for many, a new but objective) of the flakiness of test automation.
Now, the objective facts are why Selenium WebDriver is not flaky at all (the others are).
1. WebDriver is a W3C standard
The reason that our society and most goods (TVs, Routers) are functioning well: based on standards.
All web technologies, such as HTML and CSS, are defined by W3C, and WebDriver is another W3C standard. More importantly, all major browser vendors, such as Apple, Google, Microsoft and Mozilla, all support WebDriver. Moreover, they provide the driver for it, such as ChromeDriver and SafariDriver. My saying, “Selenium has no flakiness (others have)”, is based on this.
As we know, web applications are getting more dynamic and complex, very much dependent on browsers. Those big browser vendors do not care whether the next version will break Cypress or Playwright but will ensure the test harness works with WebDriver.
2. It has been proven in many large organisations
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
3. Testimonies from the world-renowned experts
4. My Continous Testing Reports
ClinicWise (2013–2021): 611 raw Selenium RSpec tests
(total test execution count: 618, 237)
WhenWise (2018 — present): 559 raw Selenium RSpec tests
(total test execution count: 348,958)
Keep reading with a 7-day free trial
Subscribe to The Agile Way to keep reading this post and get 7 days of free access to the full post archives.