Why Does Some Failed Test Automation Require a Specific Firefox Version?

A clear sign of failed test automation.

Recently, one software engineer told me that there was a test automation suite that runs on a fixed version of Firefox. I asked, “it must be failed, or it has not been run for a long time”. The reply, “Yes”.

It does not make sense to use Firefox for test automation from a few years ago

According to the latest desktop browser market share stats (StatCounter, 2023–01),

Firefox is merely 6.87% , dropping from 9.18% a year ago (a drop of 25%). It is safe to say, Firefox will hardly be mentioned in browser discussions in a few years' time.

Chome, unquestionably, leads in the desktop browser market. If adding Microsoft Edge (based on Chromium as well, which works the same for automated test scripts as for Chrome), its market share reached 77.48% .

End-to-End Test automation is to verify the apps as an end-user would (the execution setup is not set by the automated testers). Now, most users browse the app in Chrome. Surely, locking on Firefox is wrong. Please note, here, I don’t mean cross-browser testing, just standard automated functional testing, running tests against Firefox instead of Chrome without particular functional reasons.

Why do some Lock a specific Firefox version in test automation?

For the requirement and needs of end-to-end web test automation, there is no excuse not to use Chrome as the target browser. Why some do not use Chrome then? The short answer is “Chrome browse self-updates, about every two months”.

The chromedriver is the software used by Selenium to drive Chrome. ChromeDriver version needs to match the Chrome version. In practice, it is relaxed to This Version + Next Version. For example, ChromeDriver v100 will work with Chrome v101, but not v102 onwards. Otherwise, we will get an error like below.

Selenium::WebDriver::Error::SessionNotCreatedError:
  session not created: 
    This version of ChromeDriver only supports Chrome version 103
    Current browser version is 102.0.5005.61 with binary path /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

While installing/updating ChromeDriver is quite easy and fast, many fake automated testers don’t have a habit of test maintenance, even for a tiny bit of effort (see this post). For simplicity, they just lock the test scripts to a specific Firefox version.

What did this wrong behaviour reveal?

For every test automation attempt I witnessed with locking Firefox, it failed. The outdated browser is not the issue, rather, the test scripts were outdated. Not long ago, I saw a README in a test automation project:

This test scripts works only in Firefox v58.

That’s the version released in 2018! (the latest Firefox is v109). Just think about setting up test automation execution for new staff!

All these non-sense reveal a wrong fundamental view of fake or novice (not by age or years of experience. Check out “An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’”) automated testers.

“Test Automation work mainly means developing automated test scripts that run OK once or twice.” ❌

This view is completely wrong. Even the fakers know about it. End-to-End tests drive the app, and the app changes frequently.

To be fair, this largely is not fake/novice automation testers’ fault. It is a failure of our software education (at Uni or training). Based on my estimation, effort-wise in test automation,

“10% on Creation,

30% on Stablization and Refinement,

60% on maintenance. “

(for more, check out Test Creation Only Accounts for ~10% of Web Test Automation Efforts)

For motivated automated testers who want to convince the management about allocating time to test maintenance, check out the story in my article, “Estimating Test Automation Story Points is a Total Waste of Time”. For stabilization and refinement effort, check out this article, “Working Automated Test ≠ Good Reliable Test”.

How do I handle Chrome browser updates?

Easy, because I run full automated regression testing at least once every working day, in a Continous Testing Server. Triggering a run on BuildWise CT Server (free and open-source, 10th RubyAward winner) with just a click of the “Build Now” button.