Practical DDT: Randomized data scenarios in one automated test script + Continuous Testing
Introduce a practical DDT technique to have a random input data combination in one automated UI test.
This article is included in my “How to in Selenium WebDriver” series.
A reader of “Data-Driven Testing Clarified” asked me how I do DDT. The drawbacks of traditional DDT, as I put in that article, are being difficult to maintain and debug. Therefore, I will try to avoid it in practice unless necessary. In this article, I will share my way of handling multi-data scenarios in one automated UI test case.
To be realistic, we cannot afford DDT most of the time
The reason is simple: execution time. We often see a long list of scenarios in manual testers’ so-called test design spreadsheet. Every row represents one data scenario and expectations.
Let’s do a quick and simple maths calculation. Supposedly on average, the execution time of a test case is 30 seconds (which is not bad for a UI test). 120 tests (i.e. scenarios) will take 1 hour. I have seen over 100 scenarios for just a single user story. In that case, how many user stories can the automated test suite realistically cover?
“If you can manage executing tens of thousands of automated test cases every day and maintain all tests well, you may do spreadsheet-driven DDT. I cannot.” — Zhimin Zhan
Clearly, the data scenarios to be included in automated tests are limited. Please note that there is no judgement to the manual testing process here, I just focus on automated UI testing.
Test Automation is about providing quick feedback (fast regression testing) to the team. Therefore, we cannot apply the manual testing mindset of listing all possible cases to E2E automated testing. This does not mean that we cannot do multi-scenarios either. I will show my approach later.
Diminishing returns on more data scenarios
Let me illustrate an example: business sign-up for WhenWise (a service booking app I created).
A manual tester will probably come up with at least 6 scenarios, one for each business type. However, after the first one or two scenarios, the actual value of other scenarios in the context of test automation is very little.
The purpose of UI test automation is to verify business features from a user’s point of view. Technically speaking, the execution of one scenario will verify the following:
Route configuration
Infrastructure/Integration
MVC (Model, Controller, View)
Database
JavaScript and CSS in the view (browser rendering).
Extra data scenarios will only add checks for a few specific lines in MVC, but will cost almost the same execution time as the first. Remember, all companies have a limited testing infrastructure and human resources.
For WhenWise business sign-up test design, I will probably only need to develop the following two automated test cases:
Sign up for a 1-to-1 appointment-type business, such as Driving School
Sign up for a group-lesson-type business, such as Yoga Class
After that, I will have to live with that incomplete feeling. When a bug is raised later for another particular business type that is not covered by automated tests, I will create an automated test to replicate the defect as a part of the bug-fixing process. Then, I will add this new test to the regression suite.
Add more scenarios with randomized input data within one test script
Assume we have two test designs for creating a new member:
A male member
A female member
In a manual tester’s mind, these are two test cases.
However, thanks to flexible scripting, we can merge two cases into one automated test case.
new_member_page.select_gender(["Female", "Male].sample)The above test step will select Female or Male randomly.
Some readers might point out the issue: if a test execution fails, we don’t know what input data were used. Good thinking. We can print out the input data.
random_gender = ["Female", "Male].sample
puts("select gender => " + random_gender)
new_member_page.select_gender(random_gender)We can add more dynamic input data, such as
random_birth_date= Faker::Number.between(from: 18, to: 60).years.ago
puts("Birth date => " + random_birth_date)
member_page.enter_birth_date random_birth_date.strftime('%Y-%m-%d')Ruby, the best language for scripting test automated tests, will make it easy and readable.
Run tests in a CT server and display test output
For tests with randomized test input, running it individually is fine as we can see the system output in the console. How about running it as a part of the test suite?
You shall not run a test suite in a testing tool (such as TestWise or Visual Studio Code) or from a command line, because it is hard to isolate test output and the execution history will be lost. More importantly, running automated UI tests sequentially on one machine simply takes a very long time. The chance of getting a green build (passing all tests) for executing a 50+ E2E test suite this way is nearly 0%.
Instead, you shall run the suite in a Continuous Testing server (please note, not a CI server such as Jenkins) with parallel test execution in multiple build agents.
Below is a screenshot of a build in a BuildWise CT server. Click the ‘Output’ link to show the test output.
If a test execution fails and the app has a bug related to gender, we could figure that out based on the test output on the BuildWise CT server.
How to cover all data combinations with randomized input data?
You don’t need to worry about this if the tests are run frequently in the CT server. Remind you that we could not get full coverage anyway. To cover more in automated tests, the company needs to invest more resources (hardware and human).
The practical approach is to trigger builds in the CT server as often as possible, i.e., whenever there are any changes to code, test scripts or infrastructure. Along the way, the randomized input data will cover more input combinations.
Below is a test execution summary of the WhenWise test suite on the BuildWise CT server. Within 4 hours, there are 337,311 test executions.
Randomize multiple input data but verify the expectation from a spreadsheet
The approach below, in my opinion, is as close as possible for practical DDT.
Test Input Data: randomize
random_gender = ["Female", "Male].sample
puts("select gender => " + random_gender)
member_page.select_gender(random_gender)
random_birth_date= Faker::Number.between(from: 18, to: 60).years.ago
puts("Birth date => " + random_birth_date)
member_page.enter_birth_date random_birth_date.strftime('%Y-%m-%d')
# ...
member_page.click_submit()Verification: Extract from a spreadsheet
The team, mostly BAs, prepared a spreadsheet (Excel) that contains the input data columns and expected result columns.
$spreadsheet_file = "membership-fee.xls"
# call a function that parse the Excel file, and look up
# with the input data.
# fill the form with random data
membership_fee = lookup_membership_fee(random_gender, random_birth_date)
expect(driver.find_element(:id, "fee").text).to eq(membership_fee)Please note, this approach is different from the traditional DDT which will have 100 loops for 100 rows in Excel. Here, one test execution will run with only one random combination of input data, i.e. much quicker. Of course, you run this test (along with others) often in the CT server to cover more scenarios.
Of course, there will be some programming effort required, but well worth it compared to the conventional DDT.
A suggestion for must-done DDT
Sometimes automated testers have to do DDT under a manager’s pressure. In this case, I suggest
Develop one or two scenarios as automated UI tests.
to verify the user interface.Do all scenarios as API tests, if possible.
this will be much faster and more reliable too.
Some might ask: “Do you mean never do DDT via UI?”. No, do if you can. If it is manageable, maintainable and fast, surely you can do it.
“Unreliable and unmaintainable automated tests are worse than no tests.” — Zhimin Zhan
From my memory, I only did traditional DDT once. It was a pricing calculation page with a number of fields. I wrote a Selenium test that loaded input data from a CSV file (for more, check out Data-Driven Testing Clarified), and verified the result against the one in CSV. I did that way because it was a very simple page, with no JavaScript (execution of one scenario only takes about 3 seconds).
Related reading:
My eBooks
“Practical Web Test Automation with Selenium WebDriver”
self-learn real test automation quickly
Test Automation Camel, a metaphor that explains why most test automation attempts failed?