My Innovative Solution to Load Testing: Run Selenium Tests (in real browsers) in a CT Server with Parallel Execution for better Load Testing

I started with this new load-testing approach for my app first. In 2015, In 2018, In 2018, Business Wire referred to real-browser-load-testing as “Groundbreaking Technology”.

This is included in the “My Innovative Solution to Test Automation and Continuous Testing” series.

Traditional Load Testing Tools, such as Load Runner and JMeter, generate load using thread-based Virtual Users (VU) to hit the target server.

I created LoadWise (2010–2013) for load-testing web apps. It got some commercial interests, including the US Department of Homeland Security. However, the approach does not work well for AJAX, the trend of modern websites. Though it was still commercially feasible at that time, I shelved it.

For effective and simple testing AJAX in Selenium, check out this article, “Test AJAX Properly and Efficiently with Selenium WebDriver, and Avoid ‘Automated Waiting’”

My LoadWise Tool (2010–2013)

Since then, while my focus has been mainly on Functional Testing and Continuous Testing, I still thought about load testing now and then. After developing my international award-winning BuildWise CT server, I believed browser-based load testing was feasible with a CT Server and would become the mainstream in the future, based on the following reasons:

• Traditional protocol-based load testing tools are unable to test modern sites, at least not well.

• Browser, such as Chrome, is reliable and fast.

• Good test automation framework Selenium WebDriver is reliable and fast!
  In this quick benchmark test, completing 24 test steps only took 1.1 seconds on MacMini M1.

• The hardware (CPU, Memory, and Hard drive) is getting faster.
  The upcoming Apple M3 CPU, based on TSMC 3nm manufacturing process, is widely expected to have another big boost in performance.

• It is easy and cheap to create a large parallel testing lab on Cloud.

• Functional load test scripts are much easier to develop, debug and maintain.

I mentioned it to a few colleagues and friends, they thought it “was impossible”. I was ahead of my time, again.

Some software engineers would think “using a functional testing framework for load testing is insane”. That is because they haven’t witnessed a successful end-to-end (functional via UI) test automation. A criterion for successful test automation (therefore, Agile ad DevOps) is to run a suite of comprehensive end-to-end (via UI) tests daily as regression testing. This means, running them concurrently in a parallel testing lab.

This is easy to work out by math. In a real agile project, every user story must be covered by at least one automated end-to-end test. “Done, Done”, right? Suppose your project has 250 user stories, a ratio of 1.5 tests-to-user-story, and an average 30 seconds execution time for one test case.

A run of regression testing = 250 * 1.5 * 30 / 60 = 187 mins = 3 hours

A three-hour regression testing is too long for a real agile project. Please note, this is the best scenario. In reality, there will be test failures in most cases, then the team needs to analyze and trigger a few more rounds of regression testing.

The solution: run all tests in a parallel testing lab, to reduce the overall execution time, and increase the reliability of test execution (auto-rerun).

Below is a picture of a small section of Facebook’s test lab.

A small section of Facebook Test lab. Image credit: from the slides in the video presentation

That’s why Facebook could achieve:

Please note, Facebook does run UI Tests, below is its Testing Pyramid, which specifically lists WebDriver for the UI Testing tier.

Image Credit: “Continuous Integration at Facebook”

With cloud-based infrastructure (such as AWS and Azure), creating a large parallel testing lab is not hard, and is quite affordable too.