All those who claimed to be “Automated Testers” and Promoted “Headless Testing” were Fake

At least before 2023! Headless Chrome is a separate implementation.

The title is not “clickbait”. A Chrome Developers article published on 2023–02-22 provides solid proof. Read on.

Recently, one former mentee told me that she had one automated test that only failed when running in headless mode but was fine in normal mode. My advice: “Don’t run in headless mode”.

Table of Contents:
· When an automated e2e test fails only in headless mode … 😱
· Headless is a separate implementation of the browser
· Why “Automated Testers” Promote Headless Testing are Fake?
· Headless Mindset rooted in Programmer’s Unit Testing
∘ You may limit headless testing is a small scope

When an automated e2e test fails only in headless mode … 😱

During coaching, I encountered the situation “test fails only in headless mode” too. It was quite frustrating. Why? The test worked fine in normal mode (I could see it). However, the same test execution failed in headless. Without seeing the app (in the browser), I couldn’t debug it effectively. (Refer to my two other articles: Keep the Browser Open after Executing an Individual Test; Attach test execution to the existing browser)

The reason, I thought, was with the driver itself or a specific version of Chrome. Without “seeing” the app (with screenshots and HTML fragments), I could not raise a support ticket to the Selenium team (their support is excellent, much better than commercial ones).

Anyway, I stopped wasting time on that issue. As for me, I always run my daily automated end-to-end (via UI) regression in normal mode.

Today, I found the official answer from the Google Chrome Team: Headless Chrome is a separate implementation!

Headless is a separate implementation of the browser, before Chrome v112

In the article, Headless Browser Testing Clarified, I told the embarrassing history of Phantom.js headless and the very tiny benefits of running end-to-end tests in headless. Compared to its potential issues, those minor benefits are not worth it.

Before Chrome v112 (not released yet as of today, 2023–02–25), Chrome headless was a different implementation! Check out this article on Chrome Developers.

from https://developer.chrome.com/articles/new-headless/

This explains! My other prediction against the mass (over ten years ago) proved correct again!

Therefore, any “test automation engineers” who advertise “headless testing” as a valid testing method before 2023 are not genuine automated testers and should be considered fakes. Unfortunately, numerous individuals fall into this category.

Why “Automated Testers” Promote Headless Testing are Fake?

The whole “headless shenanigan” has been around for over a decade, every single test automation attempt with headless test automation, based on my observation, was a complete failure.

Some readers may find it overly critical to refer to testers who use headless automation as “fake.” That’s because we have a different understanding of “Testing” (and the Definition of End-to-End Test Automation Success).

How can someone claim to be a professional tester when they are testing something different, even if it’s 99% similar? It’s not justifiable to consider them professional just because the majority (over 70%) are doing it incorrectly.

Some might still want to debate. Let me show you how I conduct end-to-end testing in my projects. Below is a run of the WhenWise regression suite, which comprises 568 raw Selenium tests written in Ruby.

On that day (2023–02–15), I triggered four runs in the BuildWise CT server. The previous three runs failed with genuine test failures: 53, 14, and 2, respectively. That’s normal because my automated regression suite detected regression errors frequently. So, on the same day, 2272 test case executions and ~ 120,000 test step executions. To get a green build, each of those 120K test steps must PASS.