Correcting Wrong “Playwright’s Advantage over Selenium” Part 4: “Playwright has a native test runner”

It is more a Con, not a Pro.

Continue to correct the fourth wrong claim in this YouTube video, “Playwright vs Selenium: What Advantages Make Playwright the Winner in Automation Testing Battle 🏆”.

This article series:

• “Playwright is Modern and Faster than Selenium” 😒

• “Playwright has Parallel Execution Support” 👎🏽

• “Playwright has Native Auto-Waiting Mechanism” 👎🏽

• “Playwright has a native test runner” 👎🏽

• “Playwright has native HTML Reporters” 👎🏽

• “Playwright Features Can be Configured in one Configuration” 👎🏽

• “Playwright supports a range of Testing Types, e.g. API Testing, Component Testing, …” 👎🏽

• “Playwright UI Mode, CodeGen, Debugging support.” 👎🏽

• “Playwright ARIA locator support” 👎🏽

• “Playwright has frequent releases” 👎🏽

• Wrap Up

Table of Contents:
· Claim 4: “Playwright has native playwright tests”
· I don’t like “npx playwright test” runner
· There are good reasons why most automation frameworks do not provide their runners.
∘ Why should Automation and Test Syntax Frameworks remain loosely coupled?
· The problems with the Playwright Runner.

Claim 4: “Playwright has native playwright tests”

Selenium WebDriver is an automation framework, and so are Protractor and Playwright. By convention, a web automation framework drives the app in the browser, and should not have its test runner. The test syntax, runner and report are the responsibility of Test Syntax Framworks, such as RSpec, JUnit, PyTest, Mocha and Cucumber.

Test Automation Framework = Automation Framework + Test Syntax Framework

That’s why we hear “Watir + RSpec”, “Protractor + Mocha”, “Selenium + RSpec”, “Appium + RSpec”, or “Selenium + Cucumber”. As a comparison, back in the old days, commercial automation tools such as QTP intentionally blurred the boundary between automation and test syntax frameworks.

For readers who are not familiar with RSpec, the first BDD framework (first release in 2007). BTW, BDD framework ≠ Gherkin. RSpec v3.8 has 210 million downloads. Mocha and Pytest are considered clones of RSpec; they work in a similar way. I think RSpec is still the best test syntax framework for E2E test automation.

While combining automation and test syntax framework (including test runner) is not strictly forbidden, it is more a ‘Con’ than a ‘Pro’ (I will explain shortly).

In Playwright's case, it is more tricky. Because of this runner, we have two packages playwright and @playwright/test (a package contains a test syntax framework with runner). A package name containing ‘@’ symbol! Don’t you think this adds extra complexity unnecessarily?

For example, the description for those two are exactly the same.

I don’t like “npx playwright test” runner

Frankly, I don’t like the “npx playwright test” runner syntax.

Mocha is the widely used test syntax framework in the JS community. You can certainly achieve E2E test automation goals with Playwright + Mocha.

Some Playwright fans may can’t wait to argue, “There are benefits of Playwright test runner, such as … ”, which I disagree with; See my explanation later. As a test automation engineer, I verify the results. I haven’t yet seen one successful Playwright E2E test automation with 50+ user-story level UI tests running valid as daily regression testing. But I do know raw Selenium WebDriver + RSpec works very well, like the one below.

Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for Large/Complex Apps.

There are good reasons why most automation frameworks do not provide their runners.

Before Playwright, the convention was that an open-source automation framework is loosely coupled with test frameworks, such as Selenium (with multiple test frameworks, e.g. RSpec, Junit, Pytest, Mocha, …) and even Cypress (default Mocha). In other words, automation framework creators don’t care how users run it, leaving the choice to the users (to choose a test syntax framework).

Why should Automation and Test Syntax Frameworks remain loosely coupled?