Case Study: Continuous Performance Testing

Conduct performance testing frequently, many times a day. How? Make it easy by executing them in a CT server.

In a previous article, Courtney showed performance testing using Mechanize (Ruby) library. To make performance testing useful, we must run the performance tests (using thread-based, e.g. Mechanize, or browser-based, e.g. Selenium) in a CT Server. By doing that, we can easily compare the timings (from history), which is essential for performance testing.

Here is how I do performance testing:

1. Click the “Build Now” button on a project in a BuildWise CT server.
   BuildWise will fetch the latest performance test scripts (from Git) and run them.

2. Check the results in a minute or two
   performance testing execution is usually very fast

I call this “Continuous Performance Testing” ( in my book, Practical Performance and Load Testing).

Below is a run of 10 Mechanize performance tests for my WhenWise app, in a CT server.

A run of several Mechanize performance tests in a BuildWise CT Server

This test suite is for real. For example, with the help of this performance testing, I improved the login process (for the testing environment) ~300% faster.

As per the above graph (BuildWise individual operation history view), there are three segments (on average):

• 0.55 seconds

• 0.25 seconds

• 0.18 seconds (the final one)

For every performance enhancement change, I need to do a full automated end-to-end (via UI) regression testing first (i.e. functional testing), to ensure no regression errors were introduced. More often than not, I did introduce a few defects in the process of adding performance enhancement code. Thankfully, I have a good automated regression (functional) testing process.

A recent run of the WhenWise regression suite, consisting of 559 raw Selenium tests, running in a BuildWise CT Server

I don’t plan performance testing ahead, I do as I feel like, as it is so easy to run. Whenever I have an idea for performance improvement, I just do it. If it didn’t work out (cause functional defects or slow down), revert back.

“‘if it ain’t broke, don’t fix it’ is often used to prevent performance-enhancement work. It is kind of true. But not for ones with a good Automated Regression Testing process. I refine as I go, that’s why all my apps run quite fast on small instances on Cloud.“ — Zhimin Zhan

The Benefits of Continuous Performance Testing

While everyone likes better performance (of the app), with a good Continuous Performance Testing process, your app’s performance will improve. The prerequisite: your team is doing real agile, i.e., automated end-to-end (via UI) test coverage for most of the user stories. (note: few can achieve that)

The benefits of Continuous Performance Testing are obvious. I will illustrate this with my WhenWise app.

1. Faster app response => productivity boost