Playwright vs Selenium Speed Comparison

Selenium WebDriver is slightly faster than Playwright.

In the previous article, I compared the test syntax of Playwright and Selenium WebDriver. One feedback I have heard is that “Selenium is slower”, I disagree. Both are fast in terms of test execution from my experience.

Today, I did a benchmark to verify the timing difference.

Test Setup

1. Test Machine: iMac (2015 model, macOS 12.6.2)
   I chose a relatively slow machine purposefully.

2. Target website: https://travel.agileway.net
   This a simple test site created by my father for training purposes.

3. Execution method
   I developed both versions in TestWise IDE. To remove any factors, I timed and ran them both from the command line.

4. Framework versions
   - Selenium WebDriver: 4.7.1 (Ruby 3.0.2)
   - Playwright 1.18.1 (Node v18.10)

5. Browser versions
   - Chrome: 108.0.5359.124 (Official Build)
   - Playwright’s Chromium: 99.0.4812.0 (Developer Build)

6. Test Scenarios
   24 test steps covers:
   - Launching and Closing the Chrome/Chromium browser
   - Various control types, such as Link, Text Field, Checkbox, Radio, Select List, and Button
   - Different locators, such as ID, Name, Tag, Attribute, CSS and XPath.
   - Assertion

Zhimin: Some framework/tool vendors release dubious benchmark test results, with specially optimized web pages. That is fraud. The site used for this benchmark testing just contains several simple generic web pages, good for testing raw execution speed.

Test Script

TestWise supports Mocha too, i.e. JavaScript frameworks, such as Playwright. But using Ruby is much more fun!

• Raw Selenium WebDriver (Ruby) in RSpec syntax framework

  it "End-to-End Selenium Raw" do
    driver.find_element(:id, "username").send_keys("agileway")
    driver.find_element(:id, "password").send_keys("testwise")
    driver.find_element(:id, "remember_me").click
    driver.find_element(:id, "username").submit
    expect(driver.find_element(:tag_name, 'body').text).to include("Signed in")

    driver.find_element(:xpath, "//input[@name='tripType' and @value='oneway']").click
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "fromPort")).select_by(:text, "Sydney")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "toPort")).select_by(:text, "New York")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:id, "departDay")).select_by(:text, "02")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:id, "departMonth")).select_by(:text, "May 2016")
    driver.find_element(:xpath, "//input[@value='Continue']").click

    # now on passenger page
    driver.find_element(:name, "passengerFirstName").send_keys("Bob")
    driver.find_element(:name, "passengerLastName").send_keys("Tester")
    driver.find_element(:name, "passengerLastName").submit

    # one payment page
    driver.find_element(:xpath, "//input[@name='card_type' and @value='master']").click
    driver.find_element(:name, "holder_name").send_keys("Bob the Tester")
    driver.find_element(:name, "card_number").send_keys("4242424242424242")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "expiry_month")).select_by(:text, "04")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "expiry_year")).select_by(:text, "2016")
    driver.find_element(:xpath, "//input[@value='Pay now']").click
  end

• Raw Playwright in Mocha test syntax framework

  it('End to End Playwright', async function() {
    this.timeout(5000)
    await driver.locator('text=Login').click()
    await driver.fill("#username", "agileway")
    await driver.fill("#password", "testwise")
    await driver.click("#remember_me")
    await driver.click("input:has-text('Sign in')")
    await driver.textContent("body").then(function(body_text) {
      //console.log(body_text)
      assert(body_text.contains("Signed in"))
    });
    
    const trip_radio = await driver.$$("input[name=tripType]");
    await trip_radio[1].check();
    await driver.selectOption("select[name='fromPort']", "New York");
    await driver.selectOption("select[name='toPort']", "Sydney");
    await driver.selectOption("select[name='departDay']", "02");
    await driver.selectOption("#departMonth", "052021");
    await driver.click("input:has-text('Continue')");
    
    await driver.fill("input[name='passengerFirstName']", "Bob");    
    await driver.fill("input[name='passengerLastName']", "Tester");
    await driver.click("input:has-text('Next')");

    const card_type_radio = await driver.$$("input[name='card_type']");
    await card_type_radio[1].check();
    await driver.fill("input[name='holder_name']", "Bob the Tester");
    await driver.fill("input[name='card_number']", "4242424242424242");
    await driver.selectOption("select[name='expiry_month']", "04");
    await driver.selectOption("select[name='expiry_year']", "2016");
    await driver.click("input:has-text('Pay now')");
  });

Benchmark Results (raw)

Zhimin: Please note, all the timings are in normal execution mode, not headless (which will be a further ~8% faster, according to my benchmark test).

1. Raw Selenium WebDriver, 2.21

Raw Playwright: The best of 3 runs, the other two: 2.23, 2.50

2. Playwright, 2.34

Raw Playwright: The best of 3 runs, the other two: 2.43, 2.44

Zhimin: besides the comparison (see below). I will point out that both are quite fast driving the web pages in a browser: completing 24 steps (including launching and quit the browser) ~2 seconds.

Timing of the same test on Mac Mini M1 (2020 model), is ~100% faster than my old iMac 2015 model.

I am not sure I want test execution any more faster, otherwise, I won’t be able to see the app. I detected many defects by watching test execution.

By the way, because of Selenium test execution is so fast on new hardware, it is possible to perform load testing using Selenium. Apple’s M3 chip (TSMC’s 3nm process) is expected to be a further big improvement. If you are interested, check out my book: Practical Performance and Load Testing.

My point here is: there is no value to debate the test execution speed if reaching the top limit of useful range (Selenium and Playwright both did) in functional testing perspective. Still, we want to shorten the overal execution time, the only practical way is via parallel execution in a proper way (many JS frameworks got that wrong, it shall never be the automation’s framework’s responsiblity, its CT server’s). Check out my article, My Innovative Solution to Continuous Testing: Parallel Automated End-to-End Test Execution.

Refactored Versions

After getting the test steps right, as a habit, I immediately refactor based on the Maintainable Automated Test Design, using Page Object Models, a well-known design pattern in test automation. Because the raw linear test steps are hard to read and maintain.

• Selenium WebDriver using POM

  it "End-to-End Selenium POM" do
    login_page = LoginPage.new(driver)
    login_page.enter_username("agileway")
    login_page.enter_password("testwise")
    login_page.check_remember_me
    login_page.click_sign_in
    expect(driver.find_element(:tag_name, 'body').text).to include("Signed in")

    flight_page = FlightPage.new(driver)
    flight_page.select_trip_type("oneway")
    flight_page.select_depart_from("Sydney")
    flight_page.select_arrive_at("New York")
    flight_page.select_depart_day("02")
    flight_page.select_depart_month("May 2016")
    flight_page.click_continue

    passenger_page = PassengerPage.new(driver)
    passenger_page.enter_first_name("Bob")
    passenger_page.enter_last_name("Tester")
    passenger_page.click_next

    payment_page = PaymentPage.new(driver)
    payment_page.select_card_type("master")
    payment_page.enter_holder_name("Bob the Tester")
    payment_page.enter_card_number("4242424242424242")
    payment_page.enter_expiry_month("04")
    payment_page.enter_expiry_year("2016")
    payment_page.click_pay_now
  end

• Playwright using POM

it('E2E Playwright POM', async function() {
    this.timeout(5000);
    
    let login_page = new LoginPage(driver);
    await login_page.enterUsername("agileway");
    await login_page.enterPassword("testwise");
    await login_page.checkRememberMe();
    await login_page.clickSignIn();
    await driver.textContent("body").then(function(body_text) {
      assert(body_text.contains("Signed in"));
    });
    
    let flight_page = new FlightPage(driver);
    await flight_page.selectTripType("oneway");
    await flight_page.selectDepartFrom("New York");
    await flight_page.selectArriveAt("Sydney");
    await flight_page.selectDepartDay("02");
    await flight_page.selectDepartMonth("052021");
    await flight_page.clickContinue();

    let passenger_page = new PassengerPage(driver);
    await passenger_page.enterFirstName("Bob");
    await passenger_page.enterLastName("Tester");
    await passenger_page.clickNext();
    
    let payment_page = new PaymentPage(driver);
    await payment_page.selectCardType("master");
    await payment_page.enterHolderName("Bob the Tester");
    await payment_page.enterCardNumber("4242424242424242");
    await payment_page.enterExpiryMonth("04");
    await payment_page.enterExpiryYear("2016");
    await payment_page.clickPayNow();
});

Astute readers will conclude that putting the differences in language conventions aside, these two versions are quite similar. Yes, that’s the beauty of Page Object Models.

Benchmark Results (POM)