My Innovative Solution to Test Automation: Run the same test against a different server with a click of a button

Very helpful on bug fixing.

This is included in the “My Innovative Solution to Test Automation and Continuous Testing” series.

“It worked on my machine” is one of the most popular excuses used by programmers. Programmers like to say that because there are usually several server environments: the issue reported by the tester could be a server infrastructure issue, not code, as it is fine on my local machine.

Table of Contents:
· A typical set of server environments
· The need for running the same test against different servers
· Automated Test Scripts (Selenium) that support multi-servers
  ∘ Run the same test against different servers from the command line
· Set up server environments
· Change the target server environment

A typical set of server environments

Somehow, nearly all software projects agreed on server environments' naming and purposes, even those not taught at universities.

• Local or Development
  on the developer’s local machine.

• Dev
  a trial server for developers freely deploys immature builds.

• Test ( or System Test, or Integration)
  the main server for performing functional testing.

• Staging
  A production-alike environment.

• Production

Besides the software build, there are other differences among those servers, such as configuration, dependent third-party services, database servers and machine specifications. A good software company (such as Google’s engineering productivity team) will keep the differences minimal in terms of functional testing.

The need for running the same test against different servers

After I mastered test automation (End-to-End via UI), the need for quickly verifying a user story against two servers (typically, Local and Test) became obvious. One direct reason is to confirm the defect that the tester found on the Test server. (At that time, I worked as a programmer. I still am, but only coding my own apps). That means, I need to replicate the issue on my local server.

To eliminate the doubts completely, I set two goals:

• No change of test scripts/code

• Can be done simply and quickly

I achieved that with test execution from the command line and in TestWise IDE (2007), and I have been using it frequently ever since.

Automated Test Scripts (Selenium) that support multi-servers

Like everything in E2E test automation and continuous testing, I like the test scripts are 100% open and independent, not dependent on any tools including my TestWise and BuildWise.

In a typical plain Selenium test, we started with

driver = Selenium::WebDriver.for(:chrome)
driver.get("https://whenwise.agileway.net")
driver.find_element(:link, "Sign in").click
# ...
driver.get("https://whenwise.agileway.net/sign-up") # go a page

That is no good, as the actual host (or base_url, in this case,

https://whenwise.agileway.net

was hard-coded in multiple places). I changed the design, a very simple change:

driver.get(site_url)
#...
visit("/sign-up")

The site_url and visit functions are defined in the share test helper:

$BASE_URL = "https://whenwise.agileway.net"  def site_url(default = $BASE_URL)
    ENV["BASE_URL"] || default
  end  # go to path based on current base url
  def visit(path)
    driver.get(site_url + path)
  end

Run the same test against different servers from the command line

# on Linux or macOS
$ export BASE_URL=https://ci1-whenwise.agileway.net
$ rspec spec/login_spec.rb

on Windows:

> export BASE_URL=https://localhost:8080
> rspec spec\login_spec.rb

The above works fine and is the standard approach to customize test execution in CI/CD. But for usual work use within the team (if test automation is implemented well, all team members would run automated tests on their local machines), I want a more efficient way. So, I went on and implement this feature in TestWise IDE.

Set up server environments

Like most IDEs, TestWise has a concept of ‘Project’ to wrap the files and settings. In the “Project Preferences” dialog, you can specify up to 10 servers (name and base URL).

As a convention, I will use the format of http://mysite.com , so that in my test scripts, I would refer to other pages like below:

visit("/forgot-password")

Change the target server environment

To switch the target server in TestWise, just simply click the “Env:” combo box and select a server name.

Here are some examples.

1. Development Server

Notice the bottom URL changes.

2. Test Server (CI-1)

3. Production

We must not run automated tests against the production server, unless for very special reasons (specific tests only). Here, I just did it for the demo.

Please notice that the target server address’ colour (red) in TestWise’s bottom status bar.

When a tester reports a defect detected by executing an automated test script, I would immediately perform the following, preferably with the tester watching:

1. git pull to get the latest script

2. Open the ‘failed’ test script in TestWise

3. Run the test against the test environment (where the defect was detected)

4. Change the environment to Local in TestWise (shown above)

5. Rerun the same test.

For both test execution, I would choose “Keep the Browser Open after Executing an Individual Test” execution style. After a test execution completes, the tester and I could inspect the web page for comparison.

---

Related reading:

• My eBooks

  • “Practical Web Test Automation with Selenium WebDriver”

  • “Practical Continuous Testing: make Agile/DevOps real”