Data-Driven Testing Clarified

The drawbacks of DDT outweigh its benefits for most testing needs.

This article is one of the “IT Terminology Clarified” series.

Data-Driven Testing (DDT) is one type of functional test automation that loads test data stored in a table or spreadsheet format. Over a decade ago, DDT was often a highlighted feature in commercial test automation tools. Though it is still the case for some, I see it less now. Why? DDT looks good in the demonstration but has little value in practice. In this article, I will share my experience with DDT.

The core concept of DDT is that the test data is separated from test scripts.

Pros:

• Can use the same test script for the different data sets

• Non-technical staff, such as business analysts, may get involved in managing test data.

Cons:

• Test scripts (with test data) are easily outdated, as scripts and data are tightly coupled, but they may be changed independently.

• More difficult to debug

• The introduction of GUI tools often makes things more complex

• DDT is often unnecessary
  Over my 15 years in software testing, there is only one case I could remember where DDT was useful.

Overall, I recommend avoiding DDT. I will dive in more about DDT and some of its shortcomings.

Table of Contents:
· DDT in Commercial Tools
· How to do Data-Driven Testing with raw Selenium WebDriver?
· A Sample DDT (Selenium) execution
· DDT is difficult to maintain or debug
· DDT Reporting

DDT in Commercial Tools

The below screenshot is taken from SmartBear’s TestComplete documentation.

Image Source: https://smartbear.com/learn/automated-testing/introduction-to-data-driven-testing/

From the highlighted text, such as “Data-Driven Loop”, “Variables.DBTable (“Credit Card”)…” and “OrderForm”, we decipher that the GUI utility tries to create a loop of filling data (from a DB table) into a form on the application.

Image Source: https://smartbear.com/learn/automated-testing/introduction-to-data-driven-testing/

The above looks complex to me. As you can see, this kind of DDT is vendor-specific, which I don’t like.

How to do Data-Driven Testing with raw Selenium WebDriver?

You can do DDT with a free and open-source framework such as Selenium WebDriver. In fact, I think it is easier this way.

1. Prepare Test Data

Typically the data source file is in Excel or CSV format.

2. Read the Test Data file

It is a good practice to reference the test data file with a relative path (to the test script).

Excel:

require 'spreadsheet'
excel_file = File.join(File.dirname(__FILE__), "..", "testdata", "users.xls")

CSV:

require 'csv'
csv_file = File.join(File.dirname(__FILE__), "..", "testdata", "users.csv")

3. Parse the test data into a list of data sets

Most programming languages can handle parsing CSV or Excel quite easily. Process data row by row. Each row contains a set of column data, such as ["Valid Login", "agileway", "testwise", "Welcome AgileWay"] in our example.

Excel:

excel_book = Spreadsheet.open excel_file
   sheet1 = excel_book.worksheet(0)
   sheet1.each_with_index do |row, idx|
     next if idx == 0 # ignore first row
     login, password, expected_text = row[1], row[2], row[3]
     # ...
   end
end

CSV:

CSV.foreach(csv_file) do |row|
   # get user login details row by row
   login, password, expected_text = row[1], row[2], row[3]
   next if login == "LOGIN" # ignore first row   # ...
end

4. Write test steps using dynamic data

For each data set, write the test steps using the data, as in the example below:

driver.navigate.to("http://travel.agileway.net")
driver.find_element(:name, "username").send_keys(login)
driver.find_element(:name, "password").send_keys(password)
driver.find_element(:name, "username").submit
expect(driver.find_element(:tag_name => "body").text).to include(expected_text)

5. Make sure the clean state after each data set

Before a loop ends, it is important to ensure the application is in a state that can continue the next loop. For example, for our “User sign-in ” DDT, some cases sign in OK while some fail. Either way, we need to drive the application to the login page.

The code below tries to sign off. If it cannot do so, it won’t throw an error.

# if logged in OK, try log out, so next one can continue
fail_safe{ driver.find_element(:link_text, "Sign off").click }

A Sample DDT (Selenium) execution

DDT is difficult to maintain or debug

Because the data is separated from the test script and these two are tightly coupled at the same time, therefore, DDT tests are difficult to maintain.

Furthermore, DDT tests are hard to debug. For example, I updated EXPECTED_TEXT in the second data set (row in CSV). The test failed as expected. However, it is not easy to identify the cause.

From the error stack trace, we can see that line #94 failed. We know the test step failed, but we don’t know what data it failed on. We can print out data to help to debug. But that’s extra effort, isn’t it? This might seem like a minor effort. However, for a large test suite, it can be a big deal.

DDT Reporting

One selling point of DDT in commercial tools is to generate a nice-looking execution report. I admit that I was influenced by that too, I implemented this in TestWise 4.

Display test results for Data Scenario in TestWise 4, removed in TestWise 6

However, I found DDT very hard to maintain, so I deprecated this DDT report feature to discourage people from using DDT.