Why Visual Regression Testing is Wrong, Mostly?

Mostly wasteful activity. QA engineers focus on Automated E2E UI Functional Regression Testing, as should be.

A repost of my past article on Medium.

A few weeks ago, one mentee asked me, “What is Visual Testing?”.

I replied, “Did someone want to showcase Visual Testing at work, for regression testing? If so, let me guess, he or she is a developer. ”

The mentee said with a surprised tone, “Yes”.

This article is a write-down of my explanation to this mentee.

Table of Contents:
· What is Visual Testing?
· Why Visual Regression Testing is Mostly Wrong?
∘ The Story
∘ Visual Regression Testing is Expensive and Not Practical
· Why do People waste time doing this kind of useless Test Automation?

What is Visual Testing?

“Visual testing is a software testing technique that evaluates the visual appearance and behavior of a software application’s user interface (UI) or graphical user interface (GUI). Visual testing aims to verify that the application’s visual elements like colors, images, fonts, and layouts, are displayed correctly and consistently across different devices, operating systems, and browsers.” — BrowserStack.

According to this definition, Visual Testing is mainly used to check visual appearance “across different devices”. Visual testing is also known as Snapshot Testing or Screenshot Testing.

I’m not suggesting that visual testing had absolutely no value, but it would seldom be required. I never had a need to use software automation to check the UI appearance across different devices during my 26 years of working in the software industry. Manual visual checking was fine.

Some will say, “With the advancement of testing software, we can now do Visual Regression Testing?”. Really? I doubt it.

Why Visual Regression Testing is Mostly Wrong?

I started telling the mentee a story that happened ~17 years ago, which I wrote down in the “Regression Testing Clarified”.

The Story

Many years ago, I joined a large .NET project as a contract software engineer. Technically, the project was a mess, and many quality (functional) issues were reported. The star programmer Rob, who was a senior technical consultant at a well-known international software consulting firm, proposed automation, especially for regression. A few weeks later, an internal demonstration meeting was arranged.

The newly joined project manager, who worked with me on the previous project, suggested that I show my approach, too. I accepted, which I later thought was a mistake (not technically, but embarrassed my colleagues). For this reason, I remember this session quite well.

In the meeting, Rob demonstrated his approach to regression testing:

• Invoke a test to send input to the application (a workflow product), and save the output to a file, e.g. case1–1209.txt

• On the next build candidate, repeat the above to save to a different output file.

• Run a diff tool to compare these two files; if they are the same, no regression issues.

I was quite shocked. This naive ‘regression testing’ demo was well received by other team members. The tech lead even said, “cool”.

Then I came up and showed my regression testing, which I used for my own work for a while. I picked up a case like Rob’s. Different from Rob’s regression testing approach, mine is simpler:

• Verify every step inside the test scripts

If all steps pass, the regression run of the test case passes! Also, I showed a report of the execution history (daily for ~3 weeks) of my test suite (~50 test cases).

There was a moment of silence after my demo. Clearly, everyone realized this was real regression testing, as it did real verification. Comparing the output approach was no good:

• If there is timestamp info, the output will never be the same

• adding an extra print statement will cause ‘regression failure’

• we have no idea (from examining the output) which step failed.

• there were no assertions, i.e., not testing.

After seeing my demo, the team members probably were embarrassed (by their reactions to Rob’s demo). The PM was wise to notice the awkwardness. He stood up and quickly wrapped up the meeting.

Visual Regression Testing is Expensive and Not Practical