Why I Don’t Use Selenium-Grid?

It is too complex and missing key features of CT. I have been using BuildWise to manage automated UI test execution for ~10 years.

Image Credit: https://javascript.plainenglish.io/docker-selenium-grid-setup-7e66d60a926a

I am a big fan of Selenium WebDriver. I have been practising/coaching web test automation using raw Selenium WebDriver since 2011 (yes, when it was first released). Also, I published six books on Selenium, including “Practical Web Test Automation with Selenium WebDriver” and the Selenium WebDriver Recipes series.

But I don’t use Selenium Grid.

“Selenium Grid allows you to run test cases in different machines across different platforms. The control of triggering the test cases is on the local end, and when the test cases are triggered, they are automatically executed by the remote end.

After the development of the WebDriver tests, you may face the need to run your tests on multiple browsers and operating system combinations. This is where Grid comes into the picture.” — Selenium Documentation (home page)

I attempted Selenium Grid several times and found it not useful in practice (OK for demo. I wrote my opinion down in the ‘Selenium WebDriver Recipes in Ruby’ book). The Selenium team also seemed to realise this, “Selenium Grid 4 is a fresh implementation and does not share the codebase the previous version had” (from the official documentation). Selenium Grid v4 is a total rewrite. It is better, but still, I don’t think it is up to the level I expected.

People often use it for Cross-Browser Testing.

From my experience, many projects use Selenium-Grid just for cross-browser testing. That is not surprising, even in Selenium’s introduction of Grid “…, multiple browsers and operating system combination”. Some engineers consider Selenium Grid, a free competitor to SauceLabs and Browserstack.

Selenium Grid 4 UI

However, cross-browser testing has become largely unnecessary, as Chromium-based browser (e.g. Chrome and Edge) dominates the browser market. As a result, fewer and fewer people mention cross-browser testing.

We shall use Selenium, what it does best, E2E (via UI) Functional Testing, preferably running the test suite often as regression testing. In my view, Selenium Grid is not up to that challenging task.

1. Complex
   For every selenium grid node, you need to configure the node either by specifying command line parameters or a JSON file. It is quite complex.

2. Very Limited Control
   Selenium-Grid comes with a web-accessible console (the UI was very poor before v4), in my view, very basic (even with Selenium Grid 4) in terms of viewing test execution status (such as output and history) and control test executions (such as cancellation and reassignment).

3. Lack of feedback
   UI tests take time to execute, more tests means longer execution time. Selenium Grid’s distribution model is to reduce that. Apart from the raw execution time, there is also the feedback time. The team would like to see the test results as soon as a test execution finishes on one node. Even better, when we pass the whole test suite to the hub, it will ‘intelligently’ run new or last failed tests first. Selenium Grid, in my view, falls short of this.

4. Lack of Rerun
   In a perfect world, all tests execute as expected every single time. But in reality, there are so many factors that could affect the test execution:
   – test statements didn’t wait long enough for AJAX requests to complete (server on load)
   – browser crashes (rarely, but it happens)
   – node runs out of disk space
   – virus scanning process started in the background
   – windows self-installed an update
   In all the above cases, re-assign failed tests (i.e. rerun) to another node could save a potential good build.

5. Test Scripts need to be aware of Hub.

driver = Selenium::WebDriver.for :remote, 
 :url => "http://192.168.1.228:4444",
 :capabilities => :chrome)

This is an anti-pattern in CI/CD. End-to-end test scripts should be totally independent of the supporting infrastructure.

The dilemma of Selenium Grid