Three Big Fat Lies about Selenium WebDriver: “Not Reliable”, “Hard to Learn”, and “No Enterprise-Level Support”

Selenium is the best web automation framework, better than others in every aspect. Everyone who failed with Selenium I met was a fake automated tester.

I have seen many articles/posts with lies about Selenium WebDriver, the best web automation framework. Lisa Crispin, the author of the classic Agile Testing book, observed that too, and those bad sales/marketing people made her “so angry”.

Lias Crispin (author of Agile Testing book)’s comment on bad marketing against Selenium.

In my opinion, the two motives for those evil lies:

• deliberately for commercial reasons.

• from fake automated testers, who do not understand test automation at all.

This article will crush these lies one by one with proof.

∘ Lie 1: “Selenium is not reliable” 
∘ Lie 2: “Selenium lacks enterprise-level support”
∘ Lie 3: “Selenium syntax is not as good as X, and it is hard to learn.”
∘ Why can a ten-year-old learn Selenium so quickly?
∘ Summary

“Everyone can call another person a ‘bastard’, but when I say it, I will prove why that guy is a bastard. “

— Li Ao, a famous author and TV host in Taiwan

Lie 1: “Selenium is not reliable.”

A common statement after this lie is, “Cypress can do a succession of retries”.

Let me show some solid evidence of Selenium’s top reliability (much, much better than any other automation framework).

1. Selenium enables Facebook’s “release twice a day”

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

2. WebDriver is the top tier of Facebook’s Testing Pyramid

If you still have doubts, check out this video clip (from this presentation):

3. My raw Selenium/Appium test suites

I have developed and maintained thousands of raw Selenium tests for several of my web apps.

• ClinicWise (2013-)

• WhenWise (2018 — )

• TestWise ( converted from rFormSpec to Appium in 2019 — )

To give you a feel of how reliable Selenium is, I will show the number of test steps in the WhenWise suite.

+------------+---------+---------+---------+--------+
| TEST       |   LINES |  SUITES |   CASES |    LOC |
|            |   26494 |     340 |     569 |  20794 |
+------------+---------+---------+---------+--------+
| PAGE       |   LINES | CLASSES | METHODS |    LOC |
|            |    9973 |     183 |    1640 |   7527 |
+------------+---------+---------+---------+--------+
| HELPER     |   LINES |   COUNT | METHODS |    LOC |
|            |     824 |       5 |      61 |    637 |
+------------+---------+---------+---------+--------+
| TOTAL      |   37291 |         |         |  28958 |
+------------+---------+---------+---------+--------+

The above metric means this test suite has about 29,000 test steps. To get a green run (in the CT Server), each of those 29,000 test steps must pass. For 1298 runs (as regression testing), I managed a good percentage of green (all test steps pass).

Overall, ~350,000 test case executions in ~5 years.

Please note the pass rate (98%) here is not reliablity, is the overall percentage of test failures. With application changes, the automated regression testing will detect regression errors (i.e. human or app related, not with Selenium). Based on my experience, Selenium WebDriver is 100% reliable.

Now, do you still have any doubts on Selenium’s reliability? The reasons for utmost reliability are simple:

• W3C standard
  (web pages are defined by W3C)

• Supported by all major browser vendors
  e.g. Google, Apple, Microsoft, Mozilla

Some might wonder about retries. In short, retries is app or test script related, not related to the framework. For example, if a test script trying to verify a piece of text after an AJAX operation, but not there yet. This is NOT a reliability issue at all (Selenium does as instructed against the web page). For more, check out my article, “Test AJAX Properly and Efficiently with Selenium WebDriver, and Avoid ‘Automated Waiting’”.

Some desperate fake automated testers might start thinking of other factors, such as speed.

By the way, don’t trust any so-called benchmarks or reports from a small specialized test suite, which is purposely built. A fair comparision is to use a general and large test suite. But I never witnessed a working and regularlly run regression testing suite with over 50 automated end-to-end (via UI) tests in Protractor, Cypress or Playwright.

Of course, like any software, Selenium has bugs, but it is rare, only a handful (affected me) that I can remember for over 12 years. When that happens, it is usually addressed within one day or two. This leads to my crushing on the second lie “Selenium lacks enterprise-level support”.

Lie 2: “Selenium lacks enterprise-level support”

Java is a popular language used in enterprises. I have been coding Java since its first public version: v1.0.2. I never remembered any need for direct support from Sun. The same goes with C#, Ruby, …, etc.

I think you get the message, right? As a core framework, Selenium only offers general support (like Sun did for Java, in releases), in fact, top-quality support anyone can wish for. The selenium team only cares about the product and handles/fixes any raised issue professionally and quickly. Check out my support experience with the Selenium team, I am very impressed.

• Lessons Learned from Test Failures due to a new ChromeDriver Bug(2021–03–15)

• Resolve ChromeDriver Issues in Web Test Automation (2022–06–27)

I know, what these people really mean is “paid support on request” (to help on specific needs, such as Automating Shadow DOM). If so, expressed it clearly. Of course, you can get “enterprise-level support” from third parties. For example, I (and others) offer one-on-one coaching, group training and other services. The real question is will you be ready to pay for dedicated services?

From my over 20+ years working in the software industry, the so-called support and maintenance packages offered by large companies, usually costing tens of thousands of US dollars, were useless. More often, you will be directed to a call centre in India, wasting hours and explaining your problems repeatedly.

Be realistic. You should have an idea of the cost of keeping a real qualified test automation engineer (extremely rare) on the payroll, maybe US$4000/day or more. Do you think a company will put good staff on a job for $10000/year to answer calls?

Also, especially for test automation support, to effectively help a client, the consultant needs to know the test scripts, access to the target server and some business knowledge. Real support/coaching requires a dedicated one.\