Cowardly ‘Automated Testers’ use “GUI Test Automation is hard” as an excuse to avoid, instead do only API Testing
Come on, be a software testing engineer, do test as an end-user would. API Testing is very limited, compared to end-to-end (via UI) test automation.
In a recent LinkedIn chat, one “Automation Engineer” said this:
“ I happen to consider myself a good automation engineer, and I have worked with many other good ones”
“We aren’t supposed to verify everything with the GUI.”
“I think the test pyramid confirms that the GUI test is the most expensive and should only be done for the things that can’t be verified on the layers below”
This view is fairly common by fake automated testers (sadly, too many of them, considered themselves OK, sigh!). It might sound reasonable, but it is wrong.
Table of Contents:
∘ 1. ‘The World Quality Report: “ensuring end-user satisfaction is now a top QA priority”
∘ 2. Why “end-to-end user testing” is more necessary than ever?
∘ 3. No End-to-End Testing (via UI), how can you verify a user story is done?
∘ 4. “But I say I will test GUI when the parts cannot be done in underlying tiers”.
∘ 5. “GUI Test Automation is expensive”
∘ 6. “How can I learn automated end-to-end UI testing?”
1. ‘The World Quality Report: “ensuring end-user satisfaction is now a top QA priority”
Yes, end-user, i.e., testing as a user would, aka, via GUI. For web apps, this means, automated test scripts drive the app in a web browser (typically, Chrome).
The top finding:
The first time ever that “end-user satisfaction” is the top objective of quality assurance and software testing strategy”
…
A sizable majority of 1700 respondents across 10 sectors and from 32 countries have confirmed this.
Every time I spoke and wrote about “end-user testing”, frankly, I thought it was unnecessary. As the testers in a software team, shouldn’t we test everything, in particular as the user. The need to explain means something wrong with our test automation industry.
The root of the problem is that 99+% don’t know how to do GUI Test Automation in a useful (to the team daily) manner. More specifically, develop and maintain a reasonable-sized and useful automated end-to-end (UI) regression suite. Any IT person, since 30 years ago, could create some kind of automated test scripts using a recorder. But those test scripts were useless, and quickly became outdated and discarded as the testers were unable to maintain them.
It is the failure of our IT education (no test automation courses at Uni); don’t feel bad about yourself. You can still learn.
Software Test Engineers, considering “GUI test automation is mandatory (which is objectively true) ” is the first step to learn and master it. If some still think of excuses, such as “impossible” and “expensive”, be warned, the story of my 12-year-old doing real test automation will crush your self-esteem. Once accepting “GUI test automation is a must for real software testing”, people can learn and follow the test automation coach’s advice, as often it turns out, learn it very quickly and with fun!
2. Why “end-to-end user testing” is more necessary than ever?
I will illustrate by testing web apps, the most common form of software testing. As we all know, modern websites are dynamic and interactive, using AJAX and other JS features. Assume your testing team did brilliant API Testing for all backend services, a developer mistyped a comma in a JavaScript or CSS file might cause an unknown number of web pages to fail to render.
When that happens, will you still say to the customers (or end-users), “we fully tested the app thoroughly”? Of course, you won’t.
So, don’t use “we do so-called API end-to-end testing” as an excuse. Learn and do real end-to-end test automation via UI, testing a real user would.
3. No End-to-End Testing (via UI), how can you verify a user story is done?
I know many software testers would say “but GUI Test Automation is hard”, I will clarify it a bit later. Let me reconfirm the necessity of “GUI Test Automation”.
You might have heard of “In-Sprint Test Automation” and “Shift Left Testing” (by the way, which is encouraged in the World Quality Report). As a test automation engineer, how will you consider a newly-developed user story as “Done, Done”?
If you just did some API Testing, and the user could not use the feature due to an HTML or JavaScript or CSS issue. Is that User Story “Done, Done”, of course, not.
4. “But I say I will test GUI when the parts cannot be done in underlying tiers”.
This is a cunning answer. I never met a single “automated tester” saying that could provide me with regular automated UI Testing execution reports. Michael Feathers, a world renown Agile consultant, told this story better (in 2009).
It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn’t matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say “we tried.” I say, “don’t worry, everyone does.”
Why do people who focus on “API Testing” often avoid Automated UI Testing? For a simple reason, developing/maintaining even a small-sized automated end-to-end UI test suite is much harder than API Testing.
Maintaining automated UI tests requires not only skills, high efficiency, and constant ongoing attention. Because the app (with UI) changes daily, and one simple change could affect an unknown number of tests.
For argument’s sake, there are two test suites: 200 API tests (runs for 20 minutes) and 25 UI tests (runs for 30 minutes). After half-day of development and execute both suites three times.
100% API tests pass; 60% UI tests failed
98% API tests pass; 40% UI tests failed
100% API tests pass; 75% UI tests failed
(Of course, following a good test design such as the Maintainable Automated Test Design can reduce the flakiness of test scripts, still ui tests are much more fragile. There is a solution though, check out my books)
If there are test failures, the automated testers need to analyse and verify. Any changes to test scripts lead to another round of regression testing, because the new change can break other tests too, ….
I think you get the idea. Speaking as an automated test engineer who does both API Testing (authored one book) and End-to-End UI Testing, I don’t even consider API-Testing as an effort, as it is too easy, too little and far less valuable work, compared to UI Testing. I use ‘easy, and little work’ here not to belittle others, is the fact real automated testers need to greatly enhance their productivity, to meet the challenges of maintaining Automated UI Testing.
5. “GUI Test Automation is expensive”
This was a comment when I shared the “Testing Pyramid at Facebook”.
Keep reading with a 7-day free trial
Subscribe to The Agile Way to keep reading this post and get 7 days of free access to the full post archives.