Without Solid Automated End-To-End Regression Testing, Exploratory Testing is Largely Compromised
Test Automation and Exploratory Testing complement each other.
A repost of my past article on Medium in 2022
Table of Contents:
· How is manual testing conducted on my own projects?
∘ Automated End-to-End Testing (via UI)
∘ Manual Testing
· The problems with Exploratory Testing when there is no automated E2E regression testing
· Exploratory Testing compliments Automated testing too
· FAQExploratory Testing is not a new term, “coined by Cem Kane in 1984” (Wikipedia). During my 20+ years’ experience in software development, few software projects had formally planned Exploratory Testing initially but might perform “manual testing” under the flag of ‘Exploratory Testing’ later. I have observed a pattern.:
Detailed Manual Test Plan first
This would mostly be irrelevant later.Test Design (manual testing)
Manual testers create test designs (typically in Spreadsheets or some kind of vendor tools such as QualityCentre) and this phase usually takes some time.Start Test Execution
It went OK for a few iterations, then …Drop the test design (mostly), just focused on regression issues and high-priority features.
Basically, entering a fire-fighter mode. To make it sound better, some call it ‘Exploratory Testing’.
“Exploratory Testing” from a fake tester
About 10 years ago, a tester called Nick who claimed that he had developed 20,000 IBM RFT tests joined a government department (which was about to adopt RFT). Nick never wrote a single working automated test in any framework or tool during my six months’ working there. I will share a few laughable Nick’s lies in a separate article: Lies told by fake automated testers.
One day, after seeing an execution of automated test scripts (in Watir), he said: “I conduct Exploratory Testing by designing all test steps first in Excel, then recording each execution results back into spreadsheet.” I said: “That is not Exploratory Testing”.
Nick insisted: “I watched a video by XX (this is a known expert, we both presented at StarWest 2013), I am doing Exploratory Testing”.
I immediately googled “XX Exploratory Testing” and found one authoritative page. Nick’s face sank. The page clearly stated: “test execution (in Exploratory Testing) is implemented without formally authoring test steps”.
Nick walked away silently. After being exposed knew nothing about test automation, this fake automated tester thought that using the term “Exploratory Testing” sounds better than manual testing.
How is manual testing conducted on my own projects?
I took Google’s approach to simplify the terminologies in testing.
Google uses small, medium, and large to classify the types of tests.
— Source: “How Google Tests Software” book, page 12
For functional testing in my projects, there are only two terms: Automated and manual (in fact Exploratory Testing). Please note that I am focusing on the black-box functional testing here, i.e, the top tier of the Testing Pyramid.
Automated End-to-End Testing (via UI)
I follow the “35-Word Functional Test Automation Strategy”, which is
Write 1+ Automated End-to-End Test for a user story;
Add it to the regression suite;
Run the regression suite daily and keep them valid!
Start on the first day and then every day!
Yes, this would take quite some maintenance efforts. With the help of TestWise (testing IDE) and BuildWise (CT server), I managed to do quite well for several of my medium-large scale software products, in my spare time.
Manual Testing
My wife and sometimes my teenage daughter conducted manual testing for my software. Some may call it ‘Exploratory Testing’, though I don’t call it that way. The term, meaning-wise, is good 🙂, but I find it hard to pronounce.
As we have never had manual scripted testing based on spreadsheets, ‘Manual Testing’ = “Exploratory Testing” here.
As you can see, my most-loved people are doing manual testing for me, how could I belittle manual testing? I treat them with respect because I only ask them to perform manual testing after I’ve got a green build on the BuildWise CT server, that is, passing all automated end-2-end regression tests.
Issues with Exploratory Testing when there is no automated E2E regression testing
In Agile (real or fake) software projects, software updates often happen. Therefore, Regression errors happen a lot. I know that because I have worked/consulted/coached automated regression testing for over a decade. I believe most people who work on software development will agree too.
Without good support from Automated End-to-End regression testing, Exploratory Testing is severely handicapped.
1. Waste of Time on testing broken builds
Many manual testing efforts would be wasted after finding one critical error in the middle of a testing round. Fact: regression errors occur a lot. Then, manual testers must redo all the testing for the next bug-fix build.
2. Too slow, and too much effort to get test data
A common problem with testing is that people can demo a perfect showcase of a particular tool/practice as it will be a simple and well-prepared scenario. In reality, the testing steps in a typical enterprise system are far more complex. Let me illustrate with an example why Exploratory Testing is not effective without test automation’s support.
In an Enterprise Insurance system, submitting an insurance claim manually would take 8 minutes or more (if filling all required data fields correctly). Some manual testers might think that is not too bad. However, many following testing scenarios are based on that, such as Reviewing, Approve, Reject, Payment, Refund, …, etc. Please note that there are many signing-out and signing-in with different user accounts. One core End-to-End scenario can easily take 30 minutes or more (again, if every step is performed correctly) to complete.
Can exploratory testers work effectively without automation support? I haven’t yet met a single one. There is a morale factor here. How can we change that? Automation!
Automation script can shorten one insurance claim submission to ~2 minutes. Have a thought of productivity gain here. Moreover, you can perform other tasks during that two minutes. The time saving is higher than 75%.
Please note, it is automation, not test automation, as there is no planned testing activity here and the manual tester controls the testing. Automation scripts support manual testing in two ways:
Automation scripts drive the app (in a browser) to the desired state.
Then the tester continues manual testing from there. In this mode, manual testers get the test scripts (git pull), then find and run one test script. This is very easy to do in TestWise, or even Visual Studio Code if it is set up properly (see here).Automation Script might create some test data for later use.
The test data, such as a new insurance claim number, can be created with the help of automation scripts. Execution of automation scripts can be done by others, or even in a Continuous Testing server (I will write a separate article on this)
Exploratory Testing compliments Automated testing too
As Business Analysts and Manual Testers understand business logic well, the defects and issues found by them can be turned into (or update) automated test scripts, which help grow the regression suite.
FAQ
There are techniques in Exploratory Testing…
This is not my area of expertise. However, most will agree that the effectiveness of freestyle exploratory testing is highly dependent on the person. As an engineer, I prefer a system, such as Automated End-to-End regression testing.
From my experience, focusing on Test Automation is a good approach, and manual testing is a good supplement. The fact that I have never used a defect tracking system for all my apps (with commercial customers) is working proof.
2. Are you suggesting manual testers learn test automation?
No. Manual Testing is always required. Some motivated manual testers expressed their interest in learning test automation, and I trained them.
My one-day Web Test Automation with Selenium WebDriver training (now offered online) is based on my training experience with those manual testers.
Among those manual testers (who usually have no programming or automation experience), many liked my training, and some even worked as automation engineers later. Regardless, they could use automation to assist in their testing.
Related reading:
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps realBenefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.