Correct a Common Misconception: “Setting up Selenium is Complex and Time-Consuming” with a benchmark test.

Selenium is much easier and quicker to set up, if done properly.

This article is one of the “IT Terminology Clarified” series.

In the article “Love-Hate Relationship with Selenium Automation”, the author wrote: “Setting up and maintaining test environments can be complex and time-consuming.” Over my 12 years of consulting/coaching in test automation, I heard this wrong statement a few times. When possible, I corrected this wrong statement by hands-on setting Selenium up, from scratch, in minutes. Some honest people would say, “It is far easier than anything I have seen”. This is true. Selenium WebDriver beats any other test automation framework, Cypress or Playwright, in every aspect (completely free, easier to learn, faster, more reliable, better syntax, with better support, …).

In this article, I will shock JS testers (the current favourite is Playwright, Cypress seemed being dumped, like previous failed JS ones, such as Phantom.js and Protractor.js) by comparing setting up test automation from scratch: Selenium vs Playwright.

Table of Contents:
· The Benchmark Setup
· Raw Selenium WebDriver
· Playwright
· Verdict: Selenium Wins
· FAQ

The Benchmark Setup

A Windows 10 system with just Chrome installed. It is a relatively slow machine (VM in 2015 iMac). The absolute timing might be different (slower) from yours.

The Goal:

• A tester can start to create a new test against his/her specified website and run it in Chrome.

• Write one simple test statement (to verify it is working)
  Click the “SIGN IN” link

• Run the test from the command line, i.e. total independent

• Timing excludes variable factors, such as downloading the installer and text-typing for installation commands.

The requirement:

• The whole setup shall be fully repeatable by others on any other Windows machine

• The similar setup (not exactly steps) shall work for macOS and Linux, i.e., platform-independent

• No special customization (e.g. change machine registry)

• Free (mostly mean freedom, as GNU stated), others can perform the same without worrying about licensing

Raw Selenium WebDriver

The setup time (from scratch): 90seconds in total.

Some readers probably won’t believe. Again, this benchmark is on a VM of 8 years old machine, it shall be shorter when you do it. Also, this is the first step I asked the attendants (many have no test automation experience at all) of my one-day Web Test Automation training to do. Because it is so simple, everyone can follow. I know some might still have doubts, so I included the whole setup screencast video.

1. Install TestWise Ruby Edition, 52 — 6 +10 = 56 seconds.

The above (52-second, including the 6-second downloading) video includes downloading, installing and launching TestWise.

I cut the 30s of Chrome scanning the downloaded file (for some reason, this time was quite long) to reduce video size and better viewing. Anyway, downloading is excluded from our timing, focusing on set-up.

The extra 10 seconds is added for the TestWise splash screen (for unlicensed users). Don’t worry about the licencing here, you can use TestWise in free mode forever, with no need to pay a cent. By the way, this is about the setup of Selenium Test Automation to use. I include the 10s delay anyway.

For internal/on-premise training, I generally get the Admin to put the TestWise Ruby installer (~25MB) on a network drive (passing all company policy checks first). Why? The first minute is important, as the attention span of the new generation is short. I want that they set up and run a test against their AUT (a test project I created earlier and also on the shared network drive) in under 1 minute.

Now you see how quick the installation is. We are talking about seconds here (the machine I used is a VM in 2015 iMac).

2. Create and run the first test, 34 seconds

It could be any website, for this exercise,

https://whenwise.agileway.net

, a quite modern site (using Materialize design).

So in total, 90 seconds.

Some will say, “wait, you haven’t done Goal #3, running a test from the command line”. The set-up is done, just to run the test, which I will do next.

3. Run this test from the command line

TestWise Ruby Edition bundles a standard Ruby Windows distribution (packaged from the standard Ruby Installer for Windows). To run the test from the command line, add C:\agielway\TestWise6\interpreters\ruby\bin to your PATH or execute the command there.

The output:

The video:

Playwright

For Playright, I decided not to provide the videos, as it definitely will take longer. My daughter Courtney was regarded as the best Playwright tester at a large telecom company during her 3–month internship. I managed to get her to perform the Playwright installation step by step, with timing.

Ironically, Courtney hasn't heard of Playwright before the internship. She did the work in Selenium Ruby first, later, learned and scripted Playwright as requested by the senior test engineer (a fake one, most of whose tests did not run as expected). Courtney’s tests (both Selenium and Playwright) ran daily on her BuildWise CT server. Her productivity shocked the teams. Why? She did Selenium Ruby in TestWise first, then converted to Playwright. For more about this story, check out article, An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’

By the way, she was unwilling to perform this task (timing the playwright setup), as she perfer not using Playwright or does not like JavaScript for test automation. Courtney currently is working as a software development engineer at a FAANG, she does coding, not test automation, at work. Like me, she thinks Selenium Ruby is much easier, more intuitive and sometimes fun. I avoid JS automation frameworks as possible, I am her father, so she had to do that, sorry.

1. Install Node.js, 34 seconds.

After downloading the Node.js msi installer file, launch it. Clicking through/waiting for the installer to complete took 34 seconds.

2. Install Playwright, 58 seconds

Vanilla Playwright installation took 25 seconds by running:

 npm init playwright@latest

You need to answer some questions.