Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver
Don’t ruin your test automation with a silly mistake — “building a new test framework”
Recently I saw alarming text such as “Creating your own Automation Framework” repeatedly appearing in some test automation video courses and blog posts. As a test automation consultant and coach, I believe this has to be one of the most unnecessary and insane mistakes that will kill your test automation attempts. Sadly, this mistake is common.
I would like to give out my recommendation first: just use raw Selenium WebDriver. The reasons are simple:
WebDriver is a W3C standard
WebDriver is the only framework supported by all major browsers: Chrome (Google), Firefox, Safari (Apple), and IE/Edge (Microsoft)
Automated UI tests for web apps are executed in a browser. Without browser vendors’ support, how can a test framework be reliable enough for testing in the long run (as we know, Chrome self-updates about every two months)? We all know the fate of Adobe’s Flash and Microsoft’s Silverlight, both lost to HTML5, a W3C standard.
For people who don’t believe raw Selenium WebDriver will work, please read this article: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”, sample test script in ‘Maintainable Test Design’ is also shown.
Generally speaking, the wrong choices of test frameworks fall into two categories:
1. Vendor’s framework/tools
Testing tools from some big companies, such as HP Quick Test Pro (it has been sold to Micro Focus, who is now a Selenium Sponsor of Selenium WebDriver), IBM’s Rational Functional Tester, and Microsoft Coded UI, have been fading quickly in the test automation market. For example, Microsoft deprecated Coded UI Test and recommend Selenium WebDriver in 2018.
I knew a few people who have been singing for Coded UI Test at meetups or posting some blog articles to say “how great it was”. Oh well, they all turned out to be very wrong for the simple reasons I outlined earlier (which also in my first book “ Practical Web Test Automation”, 2009).
Microsoft acknowledged its testing framework/tool was no good and recommended Selenium WebDriver for web app testing. Isn’t this convincing enough? However, software testing is known for lacking common sense. People often prefer paying money for much worse testing frameworks/tools.
Besides the obvious failures of testing tools/frameworks from traditional big vendors, there are some so-called new vendor test frameworks Ranorex and Cypress. Again, they are bad choices for the same reasons. Every project I know using them failed test automation (my criteria are simple: does the whole test suite get to run several times a day reliably?).
Update: Score your project against AgileWay Continuous Testing Grading
Still not convinced? let me illustrate:
Browser (e.g. Chrome) updates about every two months, and some do self-update.
Test frameworks are code too, i.e. they will have bugs with releases.
Browser vendors such as Google won’t care about the same Ranorex or Cypress tests running in a future version of the browser. Even if Google wants, engineers at Google won’t be able to allocate resources to test against all known versions of third-party test frameworks. They can (and most possibly will, as Google uses Selenium WebDriver too) do for WebDriver, because it is a W3C standard.
The requirement for the reliability of the test framework is very high, naturally. If one minor bug, such as failing to locate an element with an XPath syntax, was introduced, you will not be able to run automated tests reliably. . Even for the early versions of ChromeDriver, there was no shortage of bugs. However, with the reputation of Google, and the existence of a standard and large user community, the bugs with Selenium WebDriver usually get fixed quickly.
For testing web apps, just choose Selenium WebDriver, as Facebook did.
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ” — Katie Coons, a software engineer at Product Stability, in “Continuous Integration at Facebook”
2. WebDriver-based ‘own framework’
Selenium WebDriver is already a very good automation framework, designed by the experts at W3C with numerous revisions. It is so easy to learn, much easier than any other automation tools or frameworks I reviewed over the last 20 years. My daughter started writing raw Selenium tests at the age of 12, and with proper guidance, your child or young siblings could too. So, don’t ruin a top framework by creating a so-called ‘own framework’ on top of it.
For readers who are in doubt, check out my daughter’s article: Set up, Develop Automated UI tests and Run them in a CT server on your First day at work.
A “Selenium-based framework’ is typically created by someone with shallow test automation knowledge, usually whose title includes the word ‘Architect’. For every project I have witnessed, their mistakes (creating a ‘framework’ based on Selenium) cost projects greatly. Surprise, surprise, the two projects I recently consulted with ‘own test framework’ both had a separate bug tracking for ‘own test framework’, how sad! Needless to say, their test automation (and Agile) failed completely. When I came in for a rescue, they were amazed that how easy the raw Selenium WebDriver was! Tests in Selenium ran faster, more stable, easy to learn, easy to read/maintain, easy to run in a CT server, easy to get help, and above all, FREE!
One common design mistake is that they took the failed concept, such as UI Map, of traditional test automation tools (like Coded UI Test) over Selenium WebDriver. As a result, they created a ‘new syntax’ that was supposed to be “easier to use”. Wrong! Before Selenium WebDriver, many claimed a ‘better’ test syntax. They all failed ultimately. A good example is the failed Protractor.
Test automation requires flexibility. Selenium WebDriver wins by utilizing programming languages (better with a scripting language such as Ruby).
Furthermore, automated scripts (UI) are fragile and prone to application changes. This requires ‘test architects’ to have a deep understanding and hands-on knowledge of test script maintenance and continuous testing. By sticking with standard Selenium WebDriver in a scripting language (such as Ruby), people will find that it is much easier to do tasks such as test data generation/manipulation, and environment variables passing to control test execution in a Continuous Testing server, …, etc.
I created a ‘test framework’ too
Some might think: “You say so because you are unable to create one”. I did, based on your definition of a ‘framework’. I created ‘ RWebSpec’ offering an alternative syntax, with a total of 87255 downloads (with quite a number of external users). I created RWebSpec in 2009 with a good reason though. At that time, IE was still the main browser. Watir supports IE well, and Selenium supports Firefox well. RWebspec combines Watir and Selenium, so that I can use one syntax to perform cross-browser testing.
When Selenium WebDriver’s support of IE is satisfactory (and Watir evolves to be just a tier on top of Selenium), I deprecated RWebSpec. Now I just use raw “Selenium WebDriver” with RSpec (sample). For people who have been to my 1-day Selenium training or seen my video, they knew, with the right tool and technique, you could develop/maintain Selenium WebDriver test scripts with very high efficiency.
This article was originally posted on LinkedIn, 2019–01–10