Two Simple Reasons that ChatGPT (AI) Won’t Work in End-to-End Test Automation

It is just hype, like many over the last two decades.

With the hype of ChatGPT, AI in Test Automation has been talked about a lot, some even listed it as the Testing Trend 2023. I think this view is wrong.

Table of Contents:
· “ChatGPT is incredibly limited”
· Recap of What ChatGPT can do in End-to-End Test Automation
· Why is AI (such as ChatGPT) very limited in assisting End-to-End Test Automation?
∘ 1. Test Automation is harder than Coding, and AI is not replacing programmers yet.
∘ 2. Test Creation is only a minor effort in Test Automation
· Summary

“ChatGPT is incredibly limited”

This comment is not mine. It is from a tweet by Sam Altman, CEO of OpenAI:

Sam’s tweet

As a test automation engineer, I consider my job quite important. Do you?

If you treat your work as mine, then “talking about AI replacing test automation” is totally irresponsible and wrong.

Accept the “ChapGPT is incredibly limited” for general use. Next, I will explain why it is even much more limited in the context of real End-to-End test automation.

Recap of What ChatGPT can do in End-to-End Test Automation

I did a few exercises using ChatGPT for testing (in this article), like this simple one below.

ChapGPT only generated generic, hypothetical test steps.

And a test script.

This is a “Hello World” (i.e. simplest) level test case. Yet, to me, chatGPT treated it as writing a made-up story, as its test steps and test scripts are irrelevant to my target site: https://travel.agileway.net ( this site has no dashboard).

Here, I have to admit, I do like ChatGPT’s writing ability, including poems and made-up stories. (For the past week, I have been sending ChatGPT’s retelling of Chinese historical stories, and it’s quite amusing because 60–90% part of each story is incorrect) But hey, we are talking about engineering and test-step-must-be-100%-correct testing.

“A simple rule of end-to-end test automation: if one step of a 100-step test script is not implemented or not working reliably, the whole test script is useless!” — Zhimin Zhan

As you can see, ChatGPT is useless for creating a valid test script (often, invalid ones will consume more time). chatGPT might help answer a specific question, like the example below.

The answer is correct. But you can get the answer on Google or Stack Overflow.

There is a much quicker to get a solution for a generic problem like this. For example, find the recipe in “Selenium WebDriver Recipes in Ruby” and run it.

Quickly navigate the recipe, and run it in TestWise IDE.

My points here:

• ChatGPT definitely won’t create a working complete test script for you.

• ChatGPT does not even create a single step for your scenario (in your target app).

• It only offers a well-known solution to a generic problem.

Why is AI (such as ChatGPT) very limited in assisting End-to-End Test Automation?

Two simple reasons (there are more, but I think these two shall be enough).

• AI, if indeed it works, will achieve programmes-less coding first as development is much easier than testing.

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)

• Test Creation is only a minor effort in Test Automation