Correct two Common Misconceptions: End-to-End Test Automation is “Simple and Easy” or “Complex and Impossible”
There has always been a “Simple and Correct” solution, but few want to take it.
This article is one of the “IT Terminology Clarified” series.
Yesterday, I saw this post, by Pascal BORNET, the author of Intelligent Automation, in my LinkedIn feed.
The heading of this post is, “Are you looking for the sign ‘Simple and Right’?”.
Here I will share my thoughts.
Table of Contents:
· Is End-to-End Test Automation Simple or Complex?
· Many Made Obvious and Wrong Decisions
· My “Simple and Right” Solution
· Simple ≠ Easy
· Summary
The Fact: Most Test Automation Attempts Failed
“Based on my 17 years of experience in this field, I concur with the primary concept depicted in the image, which is that the majority of test automation endeavours tend to fail.” — Zhimin Zhan
Is End-to-End Test Automation Simple or Complex?
The answer depends on whom you ask. For the past 30 years, test automation vendors, such as HP, have marketed the concept of “Record/playback” and “Object Identification Utility”. Despite the fact that these approaches have been demonstrated to be ineffective, some vendors persist in using these methods.
In a typical software team, numerous software engineers and managers may consider “test automation” to be a straightforward task, despite the fact that no one has ever seen a successful implementation of test automation (i.e., the team solely relies on manual testing). As evidence of this, many job advertisements require applicants to have experience in “Creating a test automation framework,” which seems absurd.
So, end-to-end test automation might not be as simple and easy as many thought.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)
Taking into account both the facts and quotes from the aforementioned reputable experts, it appears that accomplishing end-to-end test automation is an insurmountable challenge.
My answer: “It can be simple, but often human errors make it unnecessarily difficult.”
Many Made Obvious and Wrong Decisions
I have seen wrong decisions, often more than one, in failed test automation attempts:
😱 Using a compiled language, e.g. Java/C#, as the scripting language.
We call “test scripts” for a very good reason!
Check out the article, “Automated Test Scripts Shall be in the Syntax of a Scripting Language, Naturally!”😱 Using Gherkin, e.g. Cucumber/SpecFlow, as the test syntax.
This greatly increases maintaining effort, with little or no value.
Check out Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?😱 Using JavaScript as the language for end-to-end test automation.
JavaScript is a complex language (mainly for front-end development) not suitable for test automation (such as annoyingawait,node_modules, …). Every JS tester I met was a fake, with no exceptions.
It comes as no surprise that fake testers cycle through PhantomJS -> WebDriverIO -> Protractor -> TestCafe -> Pupetteer -> Cypress -> Playwright just within a few years’ time, with the same outcome: failure.
Check out “Why JavaScript Is Not a Suitable Language for Real Web Test Automation?”
😱 Using a wrong framework or tool
A classic example is Cypress, a “test automation framework” with a long list of limitations. This is insane! Because Selenium WebDriver is based on a W3C standard and is supported by all browser vendors. Moreover, it has been proven. From the feedback of failed Cypress test automation attempts I rescued, “Raw Selenium WebDriver is much easier (learn, use, maintain …, ) than Cypress”.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
😱 Test Automation Tools and The Style of Scripts are chosen by one or two engineers’ personal preferences.
The audience of end-to-end test automation is the whole team, including business analysts, manual testers and even customers, who don’t need to develop automated tests, but they shall understand the test scripts and are comfortable running them.😱 Running end-to-end tests in a typical CI server, e.g. Jenkins or Bamboo.
CI servers are built for executing unit or integration tests. End-to-end UI tests are much more prone to change and take a much longer time. You should use a proper Continuous Testing server such as Facebook’s Sandcastle or BuildWise.
I can list more, but most readers probably wonder, “enough, we know the failures. Do you have a simple solution?”. Yes, I do.
My “Simple and Right” Solution
I have been implementing end-to-end test automation with a 100% success rate since 2011, using the same formula. The logic is so simple and obvious:
Using raw Selenium WebDriver
Web technologies defined by W3C are largely unchanged, so I use raw Selenium WebDriver (a W3C standard).Using the best, easy-to-learn and proven scripting language, Ruby
The real web test automation (driving the IE browser) started with Watir (Web application testing in Ruby), and the classic Agile Testing book recommended (in 2009).
Keep reading with a 7-day free trial
Subscribe to AgileWay’s Test Automation & Continuous Testing Blog to keep reading this post and get 7 days of free access to the full post archives.