ChatGPT is out for over 6 months. Have you implemented real test automation with AI?

Reality Check of the hyped ChatGPT/AI testing.

Since ChatGPT’s release in December 2022, AI in Software Test Automation has been a hugely popular topic. Many claimed, “how great and easy that ChatGPT/AI helped test automation”. I am sure you have seen many posts saying, like below:

Six months is more than plenty for verifying a testing technology, especially web test automation (as the target barely changed over 20 years). Surely, a software tester cannot be against “VERIFY”, right (at least publicly)?

My question: “It has been six months, what is the reality of the end-to-end test automation at your software project? Show us”. I believe the common answer is: “Eh …. {excuses}”.

Let me provide some perspectives on this question in the context of real test automation.

1. ChapGPT has been out for over 6 months, the claimed 500 times more powerful v4 is released in March 2023. So, no excuses on ChatGPT side.

2. My young daughter could implement real web test automation (using raw Selenium WebDriver” in a new project on the first day, see her article “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.
   If you claimed “ChapGPT/AI makes test automation easier and more efficient”, then I expect real results in matter of hours. Do you?

3. End-to-End Test Automation, for example, Web Testing, has been exactly the same over two decades. If one automated tester ever did one real test automation at Project A, he/she can replicate the success in Project B, C, D, …, exactly the same way.

Above all, “Test” in software testing means “verifying”. Singing for a new technology (hypes are very common in the test automation industry) without concrete results means nothing. So far, I haven’t seen a real test execution report (with history) from hyped AI Testing posts. Below is what a real test automation execution report looks like:

A recent run of WhenWise end-to-end suite: 559 raw Selenium Tests

Please note the test case count: 559 here. An experienced automated tester knows about the maintenance effort for a large end-to-end (via UI) automated test suite.

Yes, I have been doing “daily production releases”, thanks to my Continuous Testing process, running full regression testing (end-to-end via UI) daily (if changes were made on the day). Moreover, I have been achieving real test automation success on clients’ and my own projects, since 2011, using the exact same tech stack (raw Selenium WebDriver + RSpec).

I consider myself a quite open-minded engineer. Even I predicted the failure of Cucumber, I spent quite some time on it (adding support in my testing IDE: TestWise, and award-winning CT server: BuildWise) My Selenium Recipes Book series covers all these five languages.