Lessons Learned from Test Failures due to a new ChromeDriver Bug

On 2019–06–15, I upgraded my test lab with BuildWise Agent v1.5.1 (including the latest ChromeDriver 75.0.3770.8) and triggered one of my continuous testing process: executing 603 Selenium WebDriver automated regression tests for ClinicWise. The build failed!

The build log showed the failure: the downloaded file was not at the expected location.

I was quite sure that no code changes could cause the issue, since the last successful build. Still, I re-ran the failed tests in TestWise IDE and verified that the file was not saved to the specified folder.

@download_path = File.expand_path File.join(File.dirname(__FILE__), "..", "tmp")
prefs = {
        :download => {
            :prompt_for_download => false,
            :default_directory => @download_path
        }
 }
@driver = Selenium::WebDriver.for :chrome, :prefs => prefs

From the build history on the CT server, I was pretty sure the cause was neither code nor test scripts. It most likely was: Chrome browser (v75) or ChromeDriver v75. I checked ChromeDriver’s release log, no mentioning changes on the chrome preferences there. Then I visited ChromeDriver Google Group, and found one related post:

A user: “ Is chrome following your preferences for this? Currently, my default_directory is not being followed and downloads are landing in the users/CURRENT_USER/downloads directory.”

Google Dev: “This is a known bug in the current release of ChromeDriver 75. Please see https://crbug.com/chromedriver/2943 for more details. We expect to have an update to ChromeDriver 75 tomorrow with the fix. Sorry about the inconvenience.”

Unless I found an alternative syntax with Chome preferences, I would just have to wait for the next releases of Chrome browser and/or ChromeDriver. But in the meantime, I needed to push the build to production, as Daily Deployment (an XP practice) has become my habit. After verifying all test failures (on the BuildWise run) were for the same reason, I added a note to the build.

Then I pushed the new build to production.

One thing I had to live with the Red Lamp (on the BuildWise page and a physical one in my study room) for one more day.

On the next day, I checked the ChromeDriver site, there was a new release of ChromeDriver, which fixed the bug. That’s the beauty of going for WebDriver, a W3C standard. You get great support, for free.

I updated the chromedriver and ran the test in TestWise, pass!

Updated the chromedriver on all build agent machines. Triggered a new build on BuildWise, …, then Green Lamp!

Lessons learned:

1. Every product has defects, make a wise choice of a trusty framework such as Selenium WebDriver. (raw Selenium WebDriver is better, as added top tier could introduce bugs, which I have seen many cases)

2. Web test automation depends on browsers, i.e., mostly Chrome. Chrome (and its driver) updates regularly, be prepared for changes.

3. Run your full regression testing often (daily), so that you know the tests/code/env statuses well. When unexpected test failures occur, you will be more likely to handle them calmly and narrow down the cause quickly.

4. Even for people who are getting used to the real Agile way of software development: releasing new updates to production on green builds, the importance of test automation, and continuous testing is still under-appreciated.

5. Act quickly and proactively, e.g. searching Selenium Github issues or raising one there.

6. Reverse your infrastructure if necessary, in this case, re-install an old version of Chrome in the interim.

Originally published on LinkedIn, June 15, 2019