Crazy Web Test Automation: “Freedom Is Slavery”
Choose Selenium WebDriver to embrace freedom
This article is one of the “Be aware of Fake Test Automation/DevOps Engineers” series.
In the all-time classic book “1984”, there are three infamous slogans of the Party:
WAR IS PEACE
FREEDOM IS SLAVERY
IGNORANCE IS STRENGTH
Except for #1, the other two crazy ideas can be commonly found in the software testing world, especially automated web app functional testing. In this article, I will talk about the crazy mindset of “Freedom is Slavery” in test automation.
I have seen many test automation failures, and the №1 technical reason is that projects selected the wrong test automation framework. Please note, I used the ‘framework’, not ‘tool’ here. I will explain this later.
For some projects that were open for alternatives, I rescued them quickly. People there were amazed by my efficiency. At the end of the first day, a typical comment was: “Zhimin did over a month’s work in one day, the first day. How?”. My answer: use the raw Selenium WebDriver which is the most feature-complete, most reliable, most flexible, best supported, future-proof, easiest to learn, most productive, free (in both freedom and pricing), popular and fast test automation framework for web apps.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
“Selenium WebDriver works well, very well. Accept this fact first, especially for people about to try others.” — Zhimin Zhan
Because Selenium-WebDriver is so easy to learn (in doubt? check this article), most manual testers with one-day training could work on the real tests the next day, under my coaching (there is no secret recipe, my book ‘Practical Web Test Automation’ was released over 10 years ago). When these testers started working on Selenium, they were happy 😊. They enjoyed the freedom (plus creativity and productivity) that was not possible in the previous proprietary test automation tool.
People who make mistakes in Test Automation and Continuous Testing are excusable. It is a failure of IT education, as there were virtually no such courses offered at university, and few have access to real test automation and CT training/coaching. I was lucky to have the opportunity to sit with a world-class programmer (all top programmers I met have a strong test automation mindset and practice them daily. In doubt? JUnit was created by Kent Beck, one of the most well-known programmers and the father of Agile) to learn test automation/CT in 2005. That was the turning point of my IT career.
My definition of web test automation: “test engineers write automated test scripts in a test automation framework to test web apps”. Simple and clear. The keywords here:
test scripts
test automation framework
web app
Based on the definition, I will show you why Slogan #2: “Freedom Is Slavery” is common in the field of test automation.
Test Scripts
Automated functional tests shall be written in a scripting language, that’s why we call them test scripts. However, every now and then, there are noises about not using test scripts for test automation, like below.
Scriptless (or codeless) Test Automation is simply mad.😱
The nature of test automation requires very high degrees of flexibility in test scripts, as applications change frequently (and constantly). Only scripting (programming style) can cope with that.
Another common mistake is to use Java/C# as the language for automated test scripts. The reason is simple: Java/C# are compiled languages, and test scripts shall be written in a scripting language such as Ruby.
Test Automation Framework
Testing, in general, is to verify whether a product is fit for its purpose. But firstly, the equipment you use must be accurate and reliable. This is common sense. Accuracy and Reliability must be based on an objective standard. In web test automation, the authority behind all web technologies is W3C. WebDriver is the only W3C standard for “programs to remotely instruct the behaviour of web browsers”.
I know some will argue “the proprietary tool I used is just fine”. There are two mistakes here:
Framework ≠tool
Over the last 25 years, commercial testing tools vendors have done a good (or evil) job of blurring: test framework work= test tool, which is very wrong. For example, PDF format may be considered as a Framework; Adobe Acrobat and Apple Preview are PDF tools. On the contrary, Microsoft Word is a combination of framework & tool. For this reason, when I needed to create a Word document, I had to buy a license from Microsoft. For generating PDFs, I don’t have to. If I want to modify a PDF document, I might need to purchase Adobe Acrobat or another tool, which I have options that I don’t have for Microsoft Word.
It is in commercial tool vendors’ interests to blur the framework and the tool in order to lock the customer in, therefore, for maximum financial gain.
I know this with experience from a different angle that most people don’t see. After my tool TestWise was demonstrated at Agile 2009 AA-FTT in Chicago, a few US investors contacted me with commercial interests in TestWise. A common request was to change TestWise to use a proprietary test syntax (instead of open-source ones such as Selenium WebDriver) and then raise up the price (~5 -10 times). I rejected the proposals.“Fine” is vague
Software Testing, especially automated software testing, is binary, “0” or “1”. Learn to think and speak in a measurable and objective manner, like a real test automation engineer. Creating a hello-world level automated test using a recorder, which can be done by a 9-year-old child, might be considered “Fine” by some. For others, “Fine” means maintaining hundreds of User-Story-Level tests (check out AgileWay Continuous Testing Grading), running them frequently, and enabling the team to push out releases to production on a daily basis.
Hence, set the goal of your test automation, then judge whether your test automation is successful or a complete failure.Scale
For test automation, the size of the test suite matters. Only with a reasonable coverage (represented by a reasonable number of automated end-to-end tests, e.g., 50+), regression testing will be useful. The effort to maintain a suite of 50 tests is not simply 10 times a suite of 5. It is actually more like 100 times. That’s why management bought expensive testing tools after they were impressed by a demonstration (by vendors or a mediocre programmer), but ended up with a complete failure. The simple reason: the team was unable to cope with the challenges of maintaining a suite of tests.
Now judge the test automation framework if any of them are actually used at your organization. My guess is that the project team never receive any real feedback from your test automation team. For real Agile or DevOps teams, the feedback from test automation is many times a day. Please watch this great CI/DevOps Presentation: “Continuous Integration at Facebook”.
Web App
A web app is an application that uses W3C technologies, such as HTML and CSS, and runs in a browser, based on the W3C HTTP protocol (The Importance of W3C Standards). Web technologies are relatively static, compared to other computer technologies.
The execution of automated web tests depends on web technologies. Therefore, the browser’s ongoing support is critical. As we know, end-users tend to upgrade their browsers to the latest version (or browser self-updates) regularly. These big browser-vendor companies, such as Google, Microsoft, Mozilla, and Apple, won’t care whether the next update will affect your tests in a proprietary framework (such as UFT and Cypress) by a small company. They will only verify against the WebDriver protocol because it is a W3C standard.
I have often seen “This test script only runs on Firefox v ” in the ReadMe file in test automation projects. How sad!
Furthermore, good test scripting alone is not enough to make test automation successful. We need to run these tests frequently on a CT server (not CI server such as Jenkins). Otherwise, they will be obsolete soon. Selenium WebDriver tests are easy to integrate with any CT solutions. On the other hand, with a proprietary test framework, your choices might be limited.
The solution to all these unnecessary problems is to use Selenium WebDriver, to enjoy freedom.
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ” — Continuous Integration at Facebook presentation
In 2018, Microsoft deprecated Coded UI Test and recommended Selenium WebDriver.
The largest software company dumped its own test automation tool to recommend Selenium WebDriver. Is this still not convincing for you? Well, you may just like the concept of “FREEDOM IS SLAVERY”.
Some might argue that is Facebook/Google, with one of the world’s top engineering teams. They can do successful test automation with WebDriver, but that does not mean others companies can. Oh well, I will share 3 build reports of my own three apps below, using raw Selenium WebDriver (web app) and Appium (desktop), with 100% FREEDOM.
For more, please read this article: AgileWay Continuous Testing Grading.