Why Do Most UI Test Automation Fail? (Part 3: Wrong Scripting Language)

Some languages are not suitable for UI test automation, such as Java, C#, and JavaScript. Use Ruby instead.

A scripting language in the context of test automation is a language syntax used in automated test scripts. Commonly, the automated test scripts are in the following three language syntaxes:

• A Compiled Programming Language, such as Java and C#

• A Scripting Language, such as Ruby and Python

• Text-based syntaxes (top tier), such as Gherkins (Cucumber) and FitNesse

Right Choice: Scripting Language. However, 90+% would choose wrongly

The correct choice, of course, is Scripting Language. We call test scripts for a good reason, i.e. test scripts shall be written in a scripting language, such as Ruby and Python. However, most software projects make wrong choices. Before I get into the reasons, let’s consolidate the correct choice first.

1. Ruby, Experts’ choice

The two authors of the classic “Agile Testing” book listed objective criteria for choosing Ruby. Please use them (highlighted in green) when judging whether a language is suitable for functional automated tests.

As a Test Automation and CT engineer with over 15 years of hands-on experience (and an experienced programmer in Java, JavaScript, and C#), I totally concur that Ruby is the best test scripting language, period. This has also been confirmed by many manual testers/business analysts whom I mentored.

2. Ruby has a long history as a tester-friendly language

Below are some Ruby test statements:

['male', 'female', 'other'].sample # <= get a random one from a list
page_text.scan(/Wise/).count       # <= find occurrences
driver.close unless driver.title.include?('Sandbox')

Easy to understand, isn’t it? Ruby has been used for testing for many years.

“Everyday Scripting with Ruby — For Teams, Testers and You” by Brian Marick, 2007

3. Ruby has a mature set of libraries for testing purposes

For example, Nokogiri is great at parsing HTML/XML. One fellow Java programmer fell in love with Ruby after seeing how HTML parsing was done in my automated tests.

Test data generation is an essential part of test automation. The Faker library makes it easy.

Faker::Name.last_name                   #=> e.g. "Ernser"
Faker::Number.between(from: 1, to: 10)  #=> e.g. 6

ActiveSupport is a library from the famous Ruby on Rails framework. It adds extensions to achieve the following:

3.days.ago
Date.today.advance(weeks: 5)

Some manual testers who attended my training were deeply impressed by the above syntax.

Wrong Choice: Compiled Language, e.g. Java, C#

Many software teams use compiled languages (e.g. Java or C#) for scripting functional tests, which is wrong. I am not saying it is impossible to implement high-quality automated functional testing in Java or C#, but it is going to be a lot harder compared to a good scripting language like Ruby. (I have programmed Java professionally for over 10 years, and published selenium books in both Java and C#).

The common reasons for this mistake make no sense.

• “Our application was developed In Java (or C#)”
  Functional test scripts don’t need to be in the same language that the target application uses. Test execution interacts with the application, and its implementation language has nothing to do with functional testing (e.g. BlackBox testing). Between 2008 and 2012, Watir was widely used in large companies (including Facebook) to test web apps written in all sorts of programming languages.

• “We are a Java shop, we don’t want to introduce another language”
  From my experience, engineers who say this are often the ones who fail test automation (after significant investment from the company). There are really two options: either ‘try another approach, including a new framework in a new scripting language’ or ‘find a better way to implement in Java, learning from a good solution’. Over the years, I have figured out that this was just an excuse. When I presented the undeniably successful test automation solution (set up the core test on the first day and run regression tests daily), they used this excuse to do nothing.

• “Scripting language is slow, Java is faster than Ruby”
  Java is a lot faster than Ruby, this does NOT translate to Java being better as a functional testing language than Ruby, not at all. By the same token, we cannot simply say C++ is better than Java/C#/JavaScript because C++ is a much faster language.
  In fact, the raw language does not matter much in the context of functional test execution, as the majority of the time was spent on the application-under-test itself by the native driver (e.g. chromedriver). See this article: “Performance Comparison: Selenium Ruby, Python, and JavaScript (Node.js)”.

A common mistake from the management is asking the tech lead’s opinion on test automation. This puts the tech lead in a position where they have to pretend they know something about Test Automation or Continuous Testing (which is often not the case).

Testing is harder than Development

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.” — Gerald Weinberg, software legend

Naturally, these tech leads will choose the (only) programming language they are comfortable with. Lacking the knowledge of Test Automation and Continuous Testing is not those tech leads’ fault. It is the failure of our IT education. I will write a separate article on this. Anyway, on test automation and Continuous Testing, tech leads were usually put into an uncomfortable position, which led to one mistake after another.

Wrong Choice: Text-based syntax such as Gherkin (Cucumber)

In Part 2 of this series: “Wrong Choice of Test Syntax Framework”, I have already covered the drawbacks of using non-scripting language as the top-tier test script. Maybe more and more people have realized the problems with Gherkins when it is used for test automation. My article “Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?” was well received and featured in Start it up, the largest publication on Medium.

Gherkin is NOT the only non-scripting-language syntax that fails UI test automation. There was FitNesse. Some might ask: “What is it?” Exactly, for it is no longer relevant to test automation. FitNesse was a big hype about 14 years ago. Also, I have seen some “new test frameworks” using JSON and XML as the top-tier test script syntax. Unsurprisingly, all these test automation attempts failed.

Wrong Choice: Non-Official WebDriver Language, such as Go

If you agree with my advice in Part 1 of this series: “Wrong Automation Framework”, use raw Selenium WebDriver as the automation framework, then you should only select one of its five official languages: Ruby, Python, Java, C#, and JavaScript. Don’t bother with another fancy/new language, avoid the unnecessary hassles.

Wrong Choice: JavaScript

The application of JavaScript has been changed from adding dynamics to web pages to application development. In my opinion, the syntax of JavaScript has been patched over these years, which makes it not suitable for user-based test automation (not JS-Programmer-based).

Please read my other article: “Why JavaScript Is Not a Suitable Language for Real Web Test Automation?”, which was featured in three major software testing newsletters: Testing Bits, Software Testing Weekly, and Coding Jag (as the cover article ).

Advice

• Use wonderful Ruby language
  Mediocre engineers tend to resist learning new things. If an engineer insists on using his favourite compiled language (Java/C#/TypeScript) to write automated test scripts in a scripting language, is it possible that he can master test automation? I don’t think so, as automated testing is much harder than development. A good mindset to embrace change is mandatory.

Ruby vs Python?

Some readers would immediately ask: Why not Python, the most popular language in recent years? Well, I still vote for Ruby. While I am comfortable with writing Python, I think the indentation rules (good for being concise) can cause frustrations to non-python programmers. Remember, the audience of functional automated tests are the whole team, including manual testers and business analysts.

• Do it in your preferred language well, and prove it in days
  Meetings can rarely help. It is the case in almost all industries. Test automation is objective and practical verification. If you cannot get a core end-2-end test scenario implemented within one day, game over, and do not waste more time on it as the later work will be more challenging, such as:
  – training engineers (including manual testers) to master it
  – maintaining test scripts efficiently to keep up to date with application changes
  – reliable test execution in CT (run many times a day)

Related Articles:

• Series: Why Do Most UI Test Automation Fail? (Technical)
  - Part 1: Wrong choice of automation framework
  - Part 2: Wrong choice of test syntax framework
  - Part 3: Wrong test scripting language
  - Part 4: Wrong choice of test automation tool
  - Part 5: Don’t know how to design easy-to-maintain test scripts
  - Part 6: Lack of Efficiency
  - Part 7: Lake of experience in executing a medium-size regression suite of automated UI tests