Evil Mudslingings against Selenium WebDriver
Selenium WebDriver is proved the best for testing web apps, and it ticks all the boxes.
This article is one of the “Be aware of Fake Test Automation/DevOps Engineers” series.
The follow-up article to “False ‘Selenium WebDriver Cons’” is the follow-up article. The reason for writing this article is that I have heard more mudslingings (false accusations to damage others’ reputations) against Selenium WebDriver recently.
I will give out my conclusion first: Selenium WebDriver is the best automation framework for testing web apps. As a W3C standard, Selenium WebDriver ticks all the boxes. Yes, let me repeat: Selenium meets ALL the requirements that shall be included in test automation (for a web app).
Proofs of Selenium WebDriver’s Success
Selenium enables ‘releases twice a day’ at Facebook
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
For most companies, their scope of testing is no way near Facebook’s.
2. Microsoft dumped its “Coded UI Test” and recommended Selenium WebDriver
3. My experience with Selenium WebDriver
Since 2011, I have been using raw Selenium WebDriver for clients and my own projects with great success. I can show some stats for my own apps.
Check out my article: WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions.
Once we know Selenium WebDriver is not only working but also the best, we can see those mudslingings clearly.
Functional Mudslingings
Here are some:
“Selenium WebDriver cannot test one-page website with dynamic JavaScript?”— a ‘principal software engineer’
“I heard Selenium was unable to test Angular or React apps” — a test lead
“Someone told me that Selenium can’t click a control in a specific cell in a table” — a test engineer
The answer to the above is, “Selenium WebDriver can test all these”, and I can show the running examples of those immediately.
Inapplicable Mudslingings
Some mudslingings went even further for tasks that are beyond the scope of functional test automation, such as:
“Selenium tests are unable to test our workflow”
Surely, we can write a Selenium test that waits for 7 days (for a subprocess to finish), but it won’t be that useful. A wise engineer knows instantly that this has nothing to do with Selenium. Selenium is for E2E tests, like an end-user. Can you do this effectively with another framework or even manually?
The solution to the above is to add testing support to the workflow. I once developed a workflow engine for a government project. The ‘Date’ in the workflow was set externally, for testing.
Again, Selenium is good for anything that is in the scope of functional test automation. The efforts to assess whether Selenium is fit for a web project is a total waste of time. Web apps use the standards (such as HTML, CSS, …) defined by W3C, and WebDriver is the only automation standard defined by W3C.“Our Selenium Cucumber tests are hard to maintain”
Another common mudslinging is that “Selenium tests are harder to maintain”. After reading the above section (Facebook and my work), you shall know that this is a 100% human problem that has nothing to do with Selenium. These engineers just need to expand their knowledge of Selenium. What I can tell you is that Selenium tests, if written well, are far easier to maintain than any other automation framework.
More ridiculous mudslinging is to blame Selenium for their wrong choice of using Gherkin syntax. They confused test syntax frameworks with automation frameworks. To illustrate, I can write two different automation tests to test the same function: one is using Selenium with RSpec; the other is using Selenium with Cucumber.
Gherkin syntax is a very bad choice for test automation. It makes tests much harder to maintain. Please read my article (featured in Start it up): Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
Non-Functional Mudslingings
“Selenium is slow”
Not true at all. People who gave this comment are commonly the fake automation testers who try to promote Cypress/Puppeteer, the so-called ‘new JS-based frameworks’.
Check out this recent benchmark comparison by Giovanni Rago: Cypress was the slowest in most categories, behind Selenium WebDriver. Interested in why? read my article: “Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)”“Selenium is hard to learn”
As a matter of fact, Selenium WebDriver is the easiest-to-learn automation framework. Some people feel that way because they learned it the wrong way and/or used the wrong tools.
Check out my article “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”, with video.“Selenium tests are flaky”
That is nonsense. Selenium WebDriver is the most reliable automation framework. Otherwise, how can Facebook maintain a large number of Selenium tests that enable multiple releases per day? Obviously, it is a human problem. Incompetent ‘engineers’ blame the technology rather than themselves.
Automated End-to-End tests, compared to unit tests, is considered flaky. However, the ‘flakiness’ is usually caused by the application or poor test scripts, e.g. an AJAX operation took longer than usual or the server was on high load. Blaming the automation framework for the above cases is very low. Instead, improve knowledge to write reliable test scripts. Here is a case study with my daughter: Working Automated Test ≠ Good Reliable Test.
There are solutions such as improving test design and execution in CT servers to support auto-retry. A real/passionate test automation engineer will research and learn to make it works.
Why do people say wrong and bad things about Selenium WebDriver?
Purely human issues.
Incompetent and mean people
There is no shame in lacking knowledge of test automation and CT as there is no formal channel (i.e., uni education) to acquire those skills. It is shameful to pretend to be knowledgeable and refuse to improve. Let me share a story.
I remember that one ‘principal software engineer’ declined my offer to show Selenium handling dynamic/AJAX which he thought was not possible. Later, I learnt that he was the main reason for the hugely embarrassing test automation failure in this large financial company. He ‘designed an automation framework’ on top of Selenium, with many wrong practices, which had cost the company millions of dollars. Of course, he sabotaged my work after I implemented real tests from Day 1 on the job.
Not long after I left, I saw a Job Ad from this company for a Senior Test Automation Engineer. One criterion was particularly interesting: “Experience in Java, C#, JavaScript and Python”. The only excluded language was Ruby, the proven solution I did (in a very short time and verified by the test team)!Fear
Failures in test automation are common. Most IT professionals have never seen single successful E2E Test Automation in their whole career. Therefore, tech leads’ minds are naturally conditioned to failure. When human beings are getting used to one state, a natural reaction is to reject anything that might change the state. This is fear, fear of reality that test automation might work.
“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day.”
— The Software Revolution Behind LinkedIn’s Gushing Profits, Wired, 2013
Money
For a tech lead, there are no financial benefits in recommending the free, open-source and proven framework, such as raw Selenium WebDriver + RSpec. The best Selenium is free (as in both freedom and free beer) which sounds good for the company (and its executives) but not necessarily for the tech leads.
“ So, you’re probably wondering why some people use Cypress.
Well, just take into consideration of the fact that the Cypress company spends a lot of money on Marketing and a lot of that is focused on Content Marketing.
This means that you’ll find a lot of articles are written by certain individuals where they claim that Selenium is old and flaky and Cypress is just what you need.
Don’t be fooled by those articles.”
- A post on Quora by Mahendra Kaushal
However, if a tech lead suggests an expensive commercial testing tool, based on some journal’s report, he might get kickbacks (or other forms, such as being invited to attend the company’s conferences, …) from the salesperson. Surely, test automation would fail later on, but who cares? The tech lead has plenty of excuses, such as
“My suggestion was based on a Gartner report”
Check out this article (in 2015) on Gartner’s yearly report. The listed products are largely no longer relevant anymore. “in 2014 IBM (Rational Functional Tester) had the top spot”, how wrong is that!
Remember, Facebook achieved ‘release twice-a-day’ with the help of Selenium at least in 2013. Why didn’t Gardner recommend Selenium? I think you know the answer.“I chose the industry-leading tool”
We all heard this: “Nobody Gets Fired For Buying IBM” (this Forbes article added: “But They Should”. By the same token, I have seen many test automation failures with HP (later MicroFocus) and IBM tools. If it failed, the tech lead could say: “it is HP or IBM’s fault, not mine”.
All in all, if you are a CIO (or CEO of an IT firm), don’t be fooled by the suggestions from your senior technical engineers (with titles such as Software Architect, Chief Engineer, or Principal Software Engineer) who mostly don’t understand test automation and Continuous Testing at all. As a matter of fact, the situation is usually worse because they thought they knew. This is not an exaggeration. Test Automation and Continuous Testing are practical knowledge, which can only be gained by daily practice, not by talking about it such as “test pyramid” or “CI/CD”.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)
I started practising test automation in 2005, and since then, I have been hands-on working with automated test scripts in testing IDE or test executions in a CT server on any working day. Why? If I don’t, some tests might be broken due to the nature of software development (constantly changing); If a test is left broken for more than one day, it would be much harder to fix, as in “Broken windows theory”. Now, look at what your senior technical leads do every day. Mostly on all sorts of meetings, right? I rest my case.
To CIOs/CEOs: you want to achieve “Release Early, Release Often”, right? If it is not happening, the most likely cause is that the capability of your technical team is not good enough. The missing link was E2E Test Automation and Continuous Testing. I don’t see daily meetings (on Automation, CT, DevOps or other aspects) that could help.
For those who still do not believe raw Selenium WebDriver is the most reliable, most flexible, easiest-to-learn, best-supported and future-proof web automation framework, book a 30-min online session with me for A$1 (available once per month). I can prove it with real tests against your application. You can find more detail about the session: 30-minutes Test Automation Coaching for A$1.