Benchmark Selenium Test Executions on Machines with Different OS and CPU

Choose cost-effective build machines for executing automated tests

Execution speed is an important factor in Automated Functional UI Testing, especially for a large regression suite (e.g. 200+ user story level tests, Level 3 of AgileWay CT Grading). While we can run the tests in a Continuous Testing server that supports real (not in a test automation framework, which is bad) parallel test execution (like BuildWise), it is still wise to choose a fast, reliable, and affordable build machine for executing automated tests.

In this article, I will benchmark test executions of the same Selenium WebDriver test in the following build machines (in my testing lab):

• MacMini 2012 — i7

• iMac 2015 — i5

• MacMini 2020 — M1 (the base model)

• Windows 10 — in a VM inside iMac 2015

• Linux Mint — in a VM inside iMac 2015

Test Case: a Selenium WebDriver test in RSpec. This is an end-to-end test for WhenWise covering a wide range of web elements in a typical modern web app.

Execution: in TestWise IDE on macOS, Windows, and Linux.

Raw Benchmark Results

• MacMini 2012 — i7

• iMac 2015 — i5

• MacMini 2020 — M1

• Windows 10 — in a VM inside iMac 2015

• Linux Mint— in a VM inside iMac 2015

Comparison

Analysis

• Test executions on virtual machines will be slower than on physical machines. It is not really meaningful to compare the results of these two categories (Windows & Linux above).

• The MacMini M1, to my surprise, is not a clear winner. Maybe it is due to two reasons: Chrome and ChromeDriver are not fully optimized for Apple Silicon CPU yet and TestWise macOS edition is not a native Apple Silicon app (the impact of IDE on test execution shall be minor, though).

Findings

1. The CPU does not matter much

According to cpubenchmark.net, the CPU Marks for MacMini 2012, iMac 2015, and MacMini M1 are 5078, 5658, and 15018, respectively. However, very minor differences were exhibited in the results.

2. The HardDisk speed does not matter much

The NVMe SSD hard drive in MacMini M1 is about 6 times faster than the SSD hard drives in iMac and MacMini 2012. However, the test results on three physical Mac computers are quite similar.

3. Linux is faster than Windows

Windows VM is allocated with 2 CPUs cores and 4.5GB Ram; Linux VM is allocated with 1CPU core and 2GB Ram. Yet, the test execution in Linux is about 13% faster than it in Windows.

The starting browser time in Windows was noticeably slower compared to macOS and Linux.

Recommendation

Based on the benchmark results, I have the following recommendations to keep in mind for build machines (for executing automated tests in Chrome).

• Cheap
  No need to buy an expensive computer as a build machine for executing automated UI tests.

• Small
  So you can fit more in your test lab.

• Quiet
  It may not sound necessary. However, when your test lab has 5+ machines, you will appreciate the quietness. Also, a quieter machine generally means consuming less power, i.e., cost-saving and environment-friendly.

• Free OS
  Windows 10 is the only consumer OS that is not free. Besides the cost, test execution on Windows is slow. Furthermore, Windows 10 updates can be quite annoying to build agent machines.

In terms of specific machines, my preference is MacMini M1. It is so quiet and well priced (starting from US$699). FaceBook’s build farm consists of aisles and aisles of Mac Minis. Check it out, and prepare to be impressed!

Linux on a Mini PC, powered by AMZ Ryzen or Intel CPU (such as Intel NUC), is good too. I don’t have a comparable physical machine running on Linux, but I think Selenium test execution on Linux will be slightly faster than it is on macOS.

FAQ

Zhimin: I will write a separate article on setting up a testing lab.

1. How many build machines should I plan for?
   I don’t recommend spending a lot of time on big plans, rather, get the test lab up and running on the first day. Do as needed, 3-machines is a good start.

2. Will a test lab cost a lot?
   No, not all. I have built one for the client for under $1000, under one day. See a sample cost of a Win build machine below. Besides the machine cost, there are one CT server and multiple Agent licenses. If you chose BuildWise like me, the server is free and open-source; the agent can run in free mode (with a time limit, and the agent license is only $30/month).

3. How about cloud-based testing labs?
   I am aware of some, such as sauce labs. I only heard negative feedback, besides all sorts of problems, I don’t like its complicated and expensive pricing, with very limited time. Trust me, a working-well CT process needs many machine hours per day. AgileWay (my company) is about to release our Cloud-based testing lab: TestWisely, which only costs a fraction (like < 5% or less) of other competitors. While TestWisely is still feature-limited, it is usable, and you can sign up now. With your feedback, we can make TestWisely better.

4. My project requires testing on all three platforms: Windows, Linux and macOS. Shall we purchase machines one-third for each OS?
   No. Just one or two Windows OS will do. Because in a proper Continuous Testing process, automated tests will be run frequently as regression testing. Over time, most of the tests will be run in each OS. Therefore, focusing on using cheap/reliable/fast build machines, such as Mac Mini M1 (and M2 is coming).

5. Windows OS license is a pain and expensive (from US$139). Any specific advice on Windows build machines?
   Normally, you just need the basic Windows, i.e., Home Edition. If you have some spare Windows licenses, using VMWare is fine. The benefits of using VMs: easier to replicate. Otherwise, I suggest you purchase a refurbished machine like this one (there are plenty of similar deals):

Based on my benchmark, the above spec is fine as a build agent machine. It is cheaper than a Windows 10 Home license.

Related reading:

• eBook: “Practical Continuous Testing: make Agile/DevOps real”