Test Automation Clarified
Test Automation = Automation + QA
This article is one of the “IT Terminology Clarified” series.
Test Automation, according to Wikipedia, “is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes”. This definition is correct but too general. This article will share my thoughts on this.
Many IT people think “Test Automation” is just another term for “Automated Testing”. I disagree.
Automated Testing can be in various forms:
Integration Testing
Functional Testing
Performance Testing
Load Testing
Penetration Testing
…
As you can see, Automated Testing is quite a generic term. Test Automation, in my opinion, is specific to end-to-end functional testing via GUI (if there is GUI).
If you are a programmer, unit testing and integration testing are already a part of your work.
Test Automation is about Automated End-to-End Testing
End-to-End testing is to test the software as an end-user would. If your app has GUI (e.g. web or mobile), API testing or Microservice testing (a fancier name) is not end-to-end testing.
According to World Quality Report 2018–19, “The first time ever that `end-user satisfaction” is the top objective of quality assurance and software testing strategy”.
So, there are no excuses, such as ‘we are fine with just unit testing’, to avoid doing real test automation. I know, the ultimate reason in most software companies is that “there is no one who could do it properly”. Commonly, many have seen so-called promising test automation demos from multiple attempts, which all failed. My old-school suggestion to companies and individuals: “Admit and learn it, from a real test automation coach”.
The Test Automation Wikipedia page lists two general approaches:
Graphical user interface (GUI) testing
API driven testing
Both of the above are end-to-end testing. If there is GUI (such as web or mobile), test scripts drive it. If there is no GUI, test scripts invoke the API.
Not convinced? Considering the below terms that you might have heard of:
Web Test Automation
Example automated test scripts: Selenium WebDriver
Desktop Test Automation (Windows)
Example automated test scripts: Appium + WinAppDriver
Mobile Test Automation
Example automated test scripts: Appium + XCUITest Driver / Espresso Driver
All above imply automating the application via GUI.
API Testing
For software without GUI, its API is the end-to-end testing.
I have worked on several projects like that. I remember for a major release of a government system, all major banks sent their people to conduct acceptance testing of the APIs.
Test Automation Synergy
Astute readers might have noticed that all the above test scripts (regardless of web, desktop, mobile, and API) are
all in Ruby scripting language
all in RSpec syntax framework (automation frameworks are different though)
all developed and run in TestWise tool
This means that we can easily mix all the above test scripts together, such as invoking an API and then navigating the web app to verify, and vice versa. I call this Test Automation Synergy, only possible in end-to-end testing.
Test Automation = Automation + QA
A test automation script (and its variant) can serve two purposes: Automation Assist and Quality Assurance.
Automation Assist
Driving the app to a certain state using automated scripts (without checking). In essence, automated scripts are productive tools that can provide many benefits:
Quickly navigate the app (Programmer/BA/Manual Tester)
Generate test data scenarios (BA/Manual Tester)
Create training videos or train newly-joined team members
Quality Assurance
Automating the app (via GUI) and then performing checks is software testing but in a much faster and more efficient way (compared to manual testing). Test Automation is the foundation of Agile and DevOps, to which few will argue. However, few software projects can do real test automation in reality.
Test Automation’s Audience: the Whole Team
“The whole team” includes the project manager, customers, business analysts, developers, architects, manual testers, automated testers, and support staff (maintenance). When I said this (at presentations), people might be surprised at first. Then I asked a scenario question:
Suppose you were a support programmer after the app went live. You attended the handover training and spent a few weeks with the development team. Also, you had access to hundreds of documents on Confluence.
Not long after, a list of change requests came in. As we all know, there will be feedback after the software is used by real customers. To respond to some of the feedback, software updates are necessary.
Now the question, What will help you greatly with the job of software change/update?
The answer is: “a suite of Automated End-to-End Regression Testing”. Not confluence, the content on that is most likely out of date already.
Without Automated End-to-End regression testing, there is a high chance of introducing regression issues with any software updates. The situation is worse in the maintenance phase because:
You don’t have access to a group of programmers who coded it
The business analysts who had knowledge of feature decisions left
There is no dedicated testing team
After production use, your change might corrupt the production data!
Besides supporting programmers, test automation can benefit all other team members. The reason is simple: automation scripts drive the app, the product the team is building.
Test Automation with other Hot Terms, such as Continuous Testing, Shift-Left Testing,…
Regression Testing
Regression Testing in a real agile project runs all automated End-To-End tests frequently. Check out this article for more: Regression Testing Clarified.
Shift-Left Testing
Develop and run automated end-to-end tests as soon as a user story is implemented. To achieve this, a good collaboration between developers and testers is required. For more, check out this article: ‘Shift Left Testing’ Clarified.
In-Sprint Test Automation
Develop automated End-to-End tests for all user stories in the current sprint, and make sure they (plus all from previous sprints) all PASS. Check out this article for more: In-Sprint Test Automation Clarified.
Continuous Testing
Due to the nature of software development, automated end-to-end tests can easily become obsolete (with software changes). Therefore, we need to run all the tests frequently in a continuous fashion. That is Continuous Testing. Check out this article, Continuous Testing Clarified.
“Done, Done” in Agile
A user story is not “Done, Done” unless there are at least one automated end-to-end test for it. Without the safety net, this feature might be broken by a future check-in. For more, check out “Done, Done” in Agile.
In Testing Pyramid
Test Automation, a.k.a, automated end-to-end testing via GUI, is the top tier of the Testing Pyramid, the most valuable one.
For more, check out Testing Pyramid Clarified.
Manual Testing
Test Automation can help manual testers conduct better manual testing; Ambitious manual testers can learn and grow to be automated test engineers. Check out Test Automation Coexists Well with Manual Testing.
Exploratory Testing
Test Automation and Exploratory Testing compliment each other. Check out Without Solid Automated End-To-End Regression Testing, Exploratory Testing is Largely Compromised.
Load Testing
Traditional load testing tools, such as Load Runner, are not suitable for testing modern & dynamic web apps. With advancements in hardware and cloud computing, the new trend is to execute automated functional tests (i.e. test automation) in parallel on multiple build machines to achieve ‘real’ load testing. For more, check out my new book: Practical Performance and Load Testing.
Related reading:
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps real
- Practical Performance and Load TestingAdvice on Self-Learning Test Automation with Selenium WebDriver
Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.