Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
JavaScript is OK for developing apps, but not for functional test automation, where its audience is beyond programmers. Ruby is a far better choice.
A repost of my past article published on Medium in 2021
I know many JavaScript programmers/testers won’t like the title of this article. However, this is the truth from my over 15 years of observation while working in test automation. Some would argue that JS-based test automation frameworks are gaining popularity. Yes, it might be true. However, when you take the fact that most test automation failed, this recent popularity of JS-based test automation frameworks does not mean much at all.
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly. ”
— Alan Page (author of ‘How We Test Software at Microsoft’ book) in an interview ‘Testing Lessons Learned at Microsoft with Alan Page’ (2015–03–17)
I have worked/consulted on projects that use most of these so-called new JS test automation frameworks. My conclusion is still the same: Avoid using JavaScript for UI test automation.
“The audience of Automated End-to-End Testing via UI is the whole team, not just the programmers” — Zhimin Zhan
First of all, automated functional testing aims to enable fast regression testing to provide quick feedback to the team. Professional test automation engineers must keep this in mind. An individual's preferences on a specific framework or language do not really matter.
In this article, I will list my reasons to avoid JavaScript in the following sections:
1. Is your test automation with JavaScript actually useful? I doubt it.
The purpose of test automation is not fancy demos. It is to speed up software development by detecting regression errors early and enable the team to push software updates to production daily. It makes sense to keep this in mind when comparing frameworks/languages for test automation.
Judging by the objective AgileWay CT Grading, every test automation attempt using JavaScript that I knew of failed at Level 0 (i.e. total waste of time). Have you ever worked on a successful Agile project in which automated end-to-end regression tests are written in JavaScript? Some probably would say “Yes”. My next questions are:
What is the size of the regression test suite? 50, 100, or 200 (as user story UI tests)?
How many times do you run all these tests a day? Twice or 3 times?
Does the team have the confidence to push a new build to production after passing automated regression testing?
The above questions are general, objective and irrelevant to your test automation framework or language. The purpose of automated functional testing is to enable the project to release early and release often with high confidence, not as an ad-hoc activity to claim “we are doing Agile/DevOps’.
Test Automation is about being objective. Many have claimed they have successful test automation in their favourite language. It is a reality, most (99.75%, according to Microsoft Guru Alan Page, see the quote earlier. Please note, the percentage Alan used is based on Microsoft engineers, which shall be considerably higher than the average) lied or did not even know what successful test automation is.
“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, software legend, in a podcast (2018)
Over 15 years of working and consulting in test automation and consulting, I found every team that attempted to use JavaScript failed badly. Before I showed them the real success (automated regression testing enabled daily production release), those engineers were embarrassed that they thought their previous attempts were OK.
A good example is PhantomJS. Between 2010–2015, Headless browser testing with Phantom JS was hyped highly by test architects/engineers. I tried and was puzzled on how it could possibly work? I wrote down my suggestion to avoid headless testing with PhantomJS in my book “Selenium WebDriver Recipes in Ruby”. In 2017, PhantomJS was deprecated. According to its creator, “(Headless) Chrome is faster and more stable than PhantomJS. And it doesn’t eat memory like crazy.” In other words, PhantomJS was just an immature toy, some JavaScript and Java programmers jumped on with a few hello-world tests and claimed success. The dozens of test automation engineers/architects (whom I met) claimed successful test automation with PhantomJS, were just liars. By the way, none of them could show me a solid test report over a month. Frankly, even now, when someone claims experience of “test automation in JavaScript”, it reminds me of widespread PhantomJS lies. Of course, it might be possible with successful test automation using JavaScript, unfortunately, I haven’t yet witnessed a real one.
I have a good track record of implementing successful test automation, and I can prove it. I have achieved Level 4 on AgileWay CT Grading for two projects, and many Level 3s on clients’ projects.
Back to JavaScript language in test automation. I have been using JavaScript since 1997, from Prototype, JQuery to Node.js. Programmers tend to be fixated/biased on a preferred language. I am probably more objective than most of them as my Selenium Recipes Book series covers all five official Selenium languages.
2. Judge against Criteria listed in ‘Agile Testing’ book
JavaScript scores poorly.
While a language preference is subjective, test automation is objective. Let’s evaluate JavaScript against the criteria for a good automated testing language in the classic “Agile Testing” book.
The choice of the two authors is Ruby. How does JavaScript score?
purely object-oriented
No. JavaScript is not a true Object-Oriented Language (read this article: ‘Is JavaScript a (true) OOP language?’ ). Some call JavaScript a prototype-based programming language.
a dynamic, interpreted language
Yes.
syntax: elegant & expressive
No, from a non-JS programmer's perspective. JavaScript is far behind Ruby on these. Below are some Ruby test script examples I have used often:
home_page.check_remmeber_me if site_url.include?("test")
[1, 2, 3, 4, 5].select(&:even?) #=> [2, 4]
[2, 2, nil, 1, 3].compact.uniq.sort #=> [1,2,3]
['female', 'male', 'unknown'].sample #=> a radom value in list
[].empty? #=> true
3.days.ago # after loading activesupport gemPowerful
Yes.
Suitable for creating Domain Specific Language (DSL)
Not as good as Ruby. As you might have known, Watir and Cucumber were created in Ruby.
If I may add two more generic attributes:
Concise
JavaScript is behind Ruby and Python.
Speed
JavaScript wins. However, for automated functional test execution, the raw language speed does not weigh much. (see my article: Performance Comparison: Selenium Ruby, Python, and Node.js)
All in all, JavaScript scores poorly (compared to Ruby) based on the criteria in the “Agile Testing” book.
3. Human Factors
Technically, any of the five Selenium WebDriver official languages (Ruby, Python, Java, JavaScript, and C#) will work. As the audience of functional test automation is beyond the developers, human factors matter.
The audiences of end-to-end automated tests are not just the developers. They include all the team members. This will change the perspective totally, won’t it?
1. Not intuitive and complex for non-JS-developers
For all End-2-End test automation attempts using JavaScript (from what I have seen, 100% failed, by the way), the only people who use them are JS developers. This is clearly wrong! The audience of end-to-end test automation is all team members, including manual testers, business analysts, and even customers. (DevOps, right?)
Let’s have a look at a simple Playwright test (on the Playwright doc).
(async () => {
const browser = await firefox.launch();
const context = await browser.newContext();
const page = await context.newPage();
await page.goto('https://www.example.com/');
const dimensions = await page.evaluate(() => {
return {
width: document.documentElement.clientWidth,
height: document.documentElement.clientHeight,
deviceScaleFactor: window.devicePixelRatio
}
});
console.log(dimensions);
await browser.close();
})();I understand the above for I coded JS professionally before, and I am the author of the Selenium Recipes in JS book and a Selenium JS course on educative. After working with many manual testers for years, I know many Non-JS-developers would dislike it. Especially after I created a few of my own apps, I see Test Automation and Continuous Testing from perspectives in different roles: owner, manager, business analyst and customer.
Below is some syntax that JS developers take for granted, but others will find it not intuitive and complex.
(async() => {what is
const?what is
await?console.log})()
Please note, this is just a simple HelloWorld-level automated test. There are far more confusing Node.js features (see below).
2. Some Node.js features are bad for test automation
I have covered some of these in another article: “Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)”
Confusing JavaScript Promise
Unnecessary NPM package.json (from a testing perspective)
Large node_modules folder
not every other language has annoying things like node_modules.
In particular, JavaScript Promise is bad and confusing. Again, we are talking about the audience of automated functional tests here, not only programmers.
As for Ruby, there are no such issues at all.
3. Developer-focused, but developers write bad automated UI tests commonly
Programmers, without proper training, are generally quite bad at testing (see Alan Page’s comment, above). Unfortunately, many programmers overestimate their test automation skills.
“Even if you have a test group, testing is harder than developing and this is what they don’t know. That if you want to have good testing you need to put your best people in testing. ” — Gerald Weinberg, a software legend in an interview ‘Why Testing is Harder than Developing’ (2016–05–05)
The rise of JavaScript-based test frameworks came after the popularity of JS-based application frameworks (in particular, AngularJS) in recent years. Commonly, a JS test automation framework was introduced by a tech lead with a JavaScript programming background. This is actually a problem, a big problem, which will lead to a test automation failure.
Update: Protractor, a popular test automation framework from the AngularJS team, is being deprecated. Actually, AngularJS itself is being deprecated as well. See this article on my story to convince a company to switch from Protractor to Selenium Ruby.
Why? Automated Functional Testing is very different from programming, obviously. Many mediocre programmers think automated functional testing is easy, and they couldn’t be more wrong. I will write another article to explain why programmers tend to write bad automated functional tests. Here I will quote some experts at Google.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold” — Patrick Copeland, Google’s VP in an interview “Testing the Limits with Patrick Copeland” (2010–02–03)
4. May affect testers’ independence
As we all know, the testers’ job is to verify the programmer’s work. Now, with a JS-based test framework (and its toolset), programmers intentionally or unintentionally affect the tester’s independence, which I have seen numerous times. The reasons are:
Testers are forced to use a tool that the developer uses, such as Visual Studio Code
Testers need to ask programmers’ help with JS syntax and tool use
5. Easier to come up with different syntax => more confusion
JavaScript programmers might claim that JS is flexible and can offer different syntax varieties. But in the context of scripting automated tests, the varieties are bad.
The below points in this comparision article on IronHack are correct in the context of test automation.
“Ruby’s syntax is easier than JavaScript’s. It is short, easy to visualize, and often has only one way of achieving a result.
On the other hand, JavaScript has a lot of curly braces and other “weird” punctuation which might seem a bit off-putting for beginners. It also has multiple ways of achieving a result. It’s true it can sometimes be harder when you have a choice in how to achieve something: just like in cooking where it’s easier to follow a rigorous recipe than to actually cook by choosing the ingredients yourself.”
Let’s look at the below equivalent examples in JS.
var webdriver = require('selenium-webdriver')
var expect = require("chai").expect;
const timeOut = 10000;
describe("Mocha, Chai with raw Selenium", function() {
it("Verify Title", async function() {
this.timeout(timeOut);
let driver = await new webdriver.Builder().forBrowser('chrome').build();
await driver.get('https://whenwise.agileway.net');
driver.getTitle().then(title => expect(title).to.equal("WhenWise - Booking Made Easy") );
driver.quit();
});
});—
var webdriver = require('selenium-webdriver');
var test = require('selenium-webdriver/testing');
var assert = require('assert');
var driver;
const timeOut = 10000;
test.describe('Mocha Seleenium Testing', function() {
test.it('Verify Title', function() {
this.timeout(timeOut);
driver = new webdriver.Builder().forBrowser('chrome').build();
driver.get("https://whenwise.agileway.net")
driver.getTitle().then(function(the_title) {
assert.equal("WhenWise - Booking Made Easy", the_title);
});
driver.quit();
});
});Update (2021–06):
selenium-webdriver/testingseems deprecated in the Selenium 4, so the above test need to change. This again proves my point.
You can see there are differences
async,awaitvs not using themtitle => expert(title) ...vsfunction(title) { ...
As a comparison, the equivalent Ruby version is more clear and more concise.
require "rspec"
require "selenium-webdriver"describe "RSpec Selenium" do
it "Verify Title" do
driver = Selenium::WebDriver.for(:chrome)
driver.get("https://whenwise.agileway.net")
expect(driver.title).to eq("WhenWise - Booking Made Easy")
driver.quit
end
endNote: I have been using the same test syntax for numerous projects, since 2011.
Another example: even for a function definition, there are several ways. Below are two ways to define a function selectTripType .
function selectTripType(trip_type) {
// ...
}—
class FlightPage extends AbstractPage {
constructor(driver) {
super(driver);
} selectTripType(trip_type) {
// ...
}
}Have I mentioned TypeScript?!
6. The history of JavaScript in test automation is awful
In short words, Selenium v1 used JavaScript commands to drive the browser via Selenium Core (for more, please read this article). In 2009, Selenium merged with WebDriver to a new project Selenium WebDriver, in better architecture. We should thank the programmers behind Selenium v1 who led to the great Selenium WebDriver somehow. It is worth noting that Selenium WebDriver is completely different from v1. From a different perspective, the short-lived Selenium v1 somewhat showed the limitation of JavaScript. (Selenium WebDriver is 12 years old, still going strong, with the support of all major browser vendors)
While some would argue Node.js is a game-changer for JavaScript. It is in the context of programming apps. 20 years ago, programmers only used JavaScript on web pages; At that time, JavaScript was often criticized or laughed at by Java programmers as not a proper language. Now, we can program apps with Node.js. However, JavaScript still is not convincing in the context of test automation.
“Because the mainstream story in web development of the past decade or so has been one of JavaScript all the things! Let’s use it on the server! Let’s use it on the client! Let’s have it generate all the HTML dynamically! And, really, it’s pretty amazing that you really can do all that. JavaScript has come an incredibly long way since the dark ages of Internet Explorer’s stagnant monopoly.
But just because you can, doesn’t mean you should.
The price for pursuing JavaScript for everything has been a monstrosity of modern complexity. Yes, it’s far more powerful than it ever was. But it’s also far more convoluted and time-consuming than is anywhere close to reasonable for the vast majority of web applications.”
— an article by DHH, a software legend, creator of Ruby on Rails, Founder & CTO at Basecamp & HEY, NYT best-selling author.
In April 2021, the hugely popular JS automation framework Protractor is being deprecated. In this article, I explained the reasons why I avoided Protractor a few ago when Protractor was at its prime time. Also, Story: A Former Mentee Failed Test Automation with Protractor is a good read.
When scripting an automated test, a test engineer will mostly work with standard JavaScript syntax. In other words, the previous problems with JavaScript in test automation still exist. Some will argue that with new frameworks such as Cypress or TestCafe. Wait…, we are talking about the language here, not the frameworks. (See my article: Why Do Most UI Test Automation Fail? Part 1: Wrong Automation Framework)
7. Framework Juggling
There are too many JavaScript-based test automation frameworks, such as Selenium WebDriver (JS binding), WebDriverIO, Cypress, TestCafe, Puppeteer, Playwright, Jest, …, etc. This makes test automation a wonderland for fake test automation engineers.
I have seen a few fake test automation engineers doing fake test automation as below:
Try a new JavaScript test automation framework, e.g. Cypress. Looks good! 😁
Develop a few automated tests; Start to promote the new framework within the team, and set up CI/CD… 😑
Struggle to maintain existing test suite😓
Many tests failed in CI/CD every day 😫
Blame the framework/tool, go to Step 1 (or abort) 😭
It is absurd, isn’t it? Considering these:
The challenges of test automation have never changed
The target technologies (web) have largely unchanged
If one test automation engineer successfully implements the test automation for once, the knowledge and practices can be directly applicable to other projects. In other words, he/she shall be able to implement real tests on the first day.
Of course, framework juggling can occur in other languages as well, but more likely with JavaScript.
I have been working on hands-on test automation for over 15 years, mainly using two automation frameworks: Watir (2006–2010) and Selenium WebDriver (2011–present). Both are in the wonderful Ruby language.
How good is Ruby for a beginner? Below is the text that a manual tester sent to me on the evening after she learned to write a few automated tests that afternoon for the first time.
Some may wonder how is it possible? Oh well, mastering test automation is hard and will take time, but learning to use it (for work) can be quick and fun, if under proper guidance. Seeing is believing, check out this article.