Set Up a Continuous Testing Server to Run Selenium Tests in Minutes

Do it quickly using 100% free and open-source software, no previous CI/CT experience is required

While Continuous Testing (CT, in short) is not a new term, it remains mysterious to many people, as they have never seen it working in software projects. In this article, I will show you how to set up a CT server from scratch to run a set of Selenium WebDriver tests on your machine in minutes. No previous CI/CT experience is required.

What are the differences between CI and CT servers? The short answer: CI server is suitble for managing executions of unit tests; CT server is suitable for managing executions of functional tests.

Every software you will use is 100% free, in both freedom and cost.

Objectives

• Install and set up a CT server (BuildWise)

• Create a CT project for a set of existing test scripts

• Trigger a test execution of all test scripts with a click of a button

• See test execution results on CT Server

Estimated time: 10–30 minutes. The majority of the time will be spent on installing prerequisite software such as Ruby and its libraries. If you have already had them on your machine, it will be much quicker.

Setting up a continuous integration/testing server from scratch might sound intimidating for some. Don’t worry, I am quite confident you can get it done after following the instructions in this article. Also, you can find some helpful screencast videos here.

The environment for this exercise is the Windows platform. For macOS/Linux users, the same steps can be done in a very similar process.

You don’t need to worry about the test script syntax yet (it is RSpec). For now, just focus on our main objective here: run a suite of functional tests (in this case: Selenium WebDriver tests for a web application) in a CT server, from scratch.

Prerequisite

• Git
  Please read “10 minutes’ Guide to Git Version Control for Testers” if you are unfamiliar with Git.

• Chrome browser and its matching ChromeDriver

• Ruby
  Download RubyInstaller for Windows with Devkit, such as ‘Ruby+Devkit 2.6.6–2 (x64)’. The DevKit is required for compiling certain libraries (called Gem in Ruby).
  Run the installer after it completes (don’t forget the DevKit setup part). Here is the detailed guide. You can also find a tutorial video (45s) here.
  Run the command below in a new command window:

> gem install bundler --no-document

If you encounter some issues on starting up the server (Windows only), try run the commands below:

> gem install sqlite3 --platform=ruby
> gem install ffi --platform=ruby

These two libraries may need to be compiled (using DevKit) to work on your machine.

• Browser drivers (such as ChromeDriver for Chrome)
  To run Selenium WebDriver tests against the Chrome browser, besides the Chrome browser itself, you need to install ChromeDriver: download and put it into a folder in your PATH, for example, C:\Ruby26-x64\bin.
  To verify the installation, open a command window (terminal for Unix/Mac), execute the command chromedriver --version. You shall see output like the below:
  ChromeDriver 89.0.4329.23

Install BuildWise CT Server

The Continuous Testing server we will use is BuildWise, a free and open-source CT server created by me (BuildWise won the 2nd prize of the Ruby Award in 2018). You may use it freely for the job, your clients, and your own projects.

Please note that the techniques I presented here are generally applicable to other CT servers. In fact, some features I firstly implemented for CruiseControl (the grand-daddy of CI servers), as plugins, back in 2007.

1. Install BuildWise Server

Download the BuildWise server (in zip format, e.g., buildwise-1.8.9.zip) from TestWisely, unzip to a folder. For example, c:\agileway\buildwise-1.8.9 .

Open a command window (or terminal on macOS/Linux).

> cd buildwise-1.8.9 

Run the gem-install.bat (macOS/Linux users: gem-install.sh). It will take a few minutes, I suggest you ‘Prepare a sample test project’ (see below) while waiting for this step to complete.

2. Startup BuildWise Server

Run startup-demo.bat (or start-demo.sh on macOS/Linux) under the folder to start the BuildWise server. If you see the text below, the server is started successfully.

* Listening on tcp://0.0.0.0:3618 
Use Ctrl-C to stop

Open http://localhost:3618 in your browser.

Here I started BuildWise in demo mode, the only difference from the production mode is that: Demo uses a single-file-based Sqlite database, while Production uses a MySQL database. The database configuration is in config/database.yml

3. Login

The default login/password: admin/buildwise.

Create a Build Project

BuildWise uses the concept of ‘Project’ to confine the settings and build activities on how you execute tests. A quick way to create a project in BuildWise is to provide a name (for displaying), an identifier, and a local working folder that contains test scripts. To make it easy for beginners, BuildWise includes a set of sample project configurations (with code hosted on Github) for different test frameworks.

1. Prepare a sample test project

I created some sample Selenium WebDriver projects on Github. Clone it to your machine using Git.

> cd c:\work 
> git clone https://github.com/testwisely/agiletravel-ui-tests.git

You will get a set of Selenium test projects under c:\work\agiletravel-ui-tests. Run the commands below to install the libraries to run the test scripts in this project.

> cd c:\work\agiletravel-ui-tests\selenium-webdriver-rspec 
> install-lib.cmd

Run a sample test script to verify (after gem-install.bat completes).

> rspec spec\01_login_spec.rb

You shall see test execution in a new Chrome browser window. The output will be like the below:

..Finished in 9.37 seconds (files took 0.4117 seconds to load) 
2 examples, 0 failures

2. Click “New Project” on BuildWise

3. Click the “Fill demo project” dropdown (on the top right) and select “RSpec”. BuildWise will populate settings for a demo project.

4. Change “Working Folder” to the location of your prepared test project, if necessary, to c:/work/agileway-ui-tests

5. Click the “Create” button

This will create the project ‘AgileTravel Quick Build RSpec’ in BuildWise.

Trigger test execution

To start a build (for our purpose, ‘Build’ means executing a suite of automated functional tests), click the ‘Build Now’ button.

The colour of the lava lamp image (of the project) is now changed to orange, indicating a build is underway.

Soon you will see a browser window launched, and your test is executed in it.

Feedback while test execution is in progress

If the build page is not shown already, click a build label (such as ‘1:building’) to show details of test execution in this build:

You will see the test script results as soon as it finishes the execution, no need to wait for the whole suite to complete. The above screenshot was taken when the first test script (containing 2 test cases) finished execution.

If there are test failures or errors, they will be displayed on BuildWise’s web interface.

From the error or failure description, experienced test engineers have a good chance to guess the cause right. If not, we can navigate to the test case where the error occurred, with the highlighted test file name and line number.

Please note that the tests are executed on the build server, not on your machine (unless you are running the BuildWise server locally). You may open the test project in a testing IDE (such as TestWise) on your computer, run a test-of-interest, and don’t have to wait for the CT build to complete.

Build finished

When a test execution completes, you will get the full test results shown on BuildWise. One out of six automated test cases failed in this build (this test failure was intentional).

Click the camera icon to view a screenshot of the browser when the test failure occurred.

Visit the BuildWise server's home page, and a red lava lamp image is shown next to the project.

First failed build

A red lamp! You might naturally think: how do I fix the build? We will do an exercise to fix this broken build later. For now, just assume the next user commit will fix it.

Trigger another build on the CT server. The commit message and changed files are shown on the new build.

If all tests pass, you will be rewarded with a green lava lamp on the dashboard page.

Your first green build

Please note this sever-setup is not a thrown-away demo. Just configure it with a MySQL database ( config/database.yml) and run the BuildWise server in production mode ( start-production.bat), you are set for serious Continous Testing like this one: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”.

This article is an excerpt (Chapter 2) of my book “Practical Continuous Testing”. You will find more CT topics/techniques in this book, such as:

• parallel test execution

• dynamic (and intelligent) execution ordering

• auto-retry

• manual-retry

• parallel testing lab

• set up executing tests in other frameworks such as Mocha, PyTest, and Cucumber

• …