Fake End-to-End Test Automation Clarified
After knowing what is not real, motivated professionals can learn and achieve the real.
This article is included in the “IT Terminology Clarified” and “Be aware of Fake Test Automation/DevOps Engineers” series.
This article’s focus is automated end-to-end testing via UI, i.e, testing the entire system as a user would.
For simplicity, I will use Web Test Automation, the most popular type, as the example in this article.
Table of Contents:
· Most Automated Testers are Incompetent or Fake
· What is it like for a software project with real test automation implemented?
· Software Testing Attributes
∘ Software Testing is about Feedback
∘ Software Testing is about being Objective
∘ Software Testing is about being correct, always correct
· Reality
· Real test automation is possible but needs the right attitude first
· Even if you’re lacking skill, you can still do limited Real test automation, and then growMost Automated Testers are Incompetent or Fake
During over 16 years of experience in Web Test Automation (hands-on scripting and coaching), I found most so-called automated testers I met were incompetent or fake.
The incompetent ones are honest if they acknowledge lacking the skills but being assigned to work on test automation.
Here, “incompetent” has no negative meaning. It is the failure of Software development education (in uni) and training.
The fake ones are often troublemakers. They like the way test automation is not useful at all. They are the topic of this article.
Some readers might be upset by my comment, I didn’t mean you, just referring to the automated testers/consultants I have met, and some turned out to do real test automation after my mentoring. Anyway, if you have worked with some automated testers, please assess objectively. Ask simple questions such as :
Do you have automated tests for each user story implemented in the current sprint?
Should be ‘Done Done’ in a typical Agile process.Does your automated regression testing (if you have it) cover all user stories in the previous sprints? This is a must in real Agile/DevOps (Continuous Testing). As we know, regression errors occur often.
Does your team run all Automated End-to-End UI tests every day? Continuous Testing.
Does your team really trust your automated test execution?
If all automated UI tests PASS, deploy the build to production? I do.
For more, check out the Definition of End-to-End Test Automation Success,
This is not belittling most automated testers, GUI Test Automation (real, of course) is simply very hard.
“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.
- Google VP Patrick Copeland, in an interview (2010)“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”
- Gerald Weinberg, in a podcast (2018)
Some might argue the above experts are referring to people, not necessarily mean the Automated UI testing. OK, here are two more quotes.
“Automated testing through the GUI is intuitive, seductive, and almost always wrong!” — Robert Martin, co-author of the Agile Manifesto, “Ruining your Test Automation Strategy”
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)
Based on my observation over the last 16 years in working/consulting in test automation,
“Automated UI Testing in 95+% software projects are not real” — Zhimin Zhan
This is still a conservative figure if compared to Alan Page’s 95% * 95% . Also, be reminded that Alan probably was referring to Microsoft-level test engineers, which shall be much higher than the industry average.
What is it like for a software project with real test automation implemented?
Let’s see some examples.
1. Selenium enables ‘releases twice a day’ at Facebook
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
2. “LinkedIn now releases new web and app features twice per day, compared with once per month before.”
“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs” — “The Software Revolution Behind LinkedIn’s Gushing Profits”
Also, even LinkedIn (in Silicon Valley) had to ‘lure’ someone from Google to implement this.
“Much of LinkedIn’s success can be traced to changes made by Kevin Scott, the senior vice president of engineering and longtime Google veteran lured to LinkedIn in Feb. 2011” — Wired, 2013
Some might dispute: “The companies you listed are the world’s top ones; They are exceptions, out of reach for the 99.999%. What’s your qualification to say many others are fake?”
I am an author of 10+ books on test automation and CT, and also a speaker at international software testing conferences. Well, those could be abstract or “Fake”. So, I will show concrete evidence. I have been achieving daily production releases for all my software products (TestWise, ClinicWise, SiteWise, BuildWise and WhenWise) since 2012. Also, I helped many client projects successfully as well. With my own testing, I can share some stats.
Once I get a green build (passing all tests) on the CT server, I immediately deploy the new build to production.
Also, I don’t do defect tracking at all, just as written in the classic “Agile Testing” book.
Software Testing Attributes
Now, let us examine software testing at its root.
Feedback
Objective
Correct, always correct
Software Testing is about Feedback
Software Testing is not just finding defects, fundamentally, it is all about feedback. FAIL/PASS are both valid feedback from software testing.
Automated Software Testing shall mean “Very Quick Feedback”, right? Now, try to answer this question: “After a programmer’s check-in, how long do they have to wait for testing feedback (whether it introduced regression errors)?”
The answer from Facebook.
The feedback time for my WhenWise App (551 user-story-level automated UI tests in Selenium) was ~45 minutes, as you can see in the previous screenshot.
I could but was unable to achieve 10-min feedback because I don’t have the resources (only 7 build machines) to build a parallel testing lab like Facebook.
My test lab is more like Facebook’s in 2012.
45 minutes for a full regression is still too long. Thanks to BuildWise’s dynamic ordering feature, recently failed test scripts will be run first. So, most of the time, I got needed feedback in a few minutes!
Software Testing is about being Objective
A programmer can test a user story (in fact, they must have tested a few times before marking the user story ‘done’). Why do most software teams still hire dedicated automated/manual testers? One main reason is that we want to verify it objectively. Think about it, “Testing” or “QA” contains the meaning of ‘Objective’.
A good software tester must have an objective mindset.
— Zhimin Zhan
If we agree on that (I am sure most would), Automated Software Testing is then “Absolutely Objective”, right? In manual testing, a manual tester might be flexible to allow a minor spelling error; It is a strict binary world in automated testing, either 0 or 1.
Software Testing is about being Correct, Always Correct
Testing is considered the safety net (last defence line) to prevent shipping bad builds. That’s why testing is also called “Quality Assurance”. Surely, the verification steps need to be correct.
Being correct the first time is easy, but a much more challenging task to keep the verification steps always correct. As we know, the software app under development changes a lot (which is the nature of software development), and some test steps and test data might be out of date quickly. This applies to both manual and automated testing.
In manual testing, testers are often flexible, i.e. executing test steps with updated information. That’s why written manual test steps are often outdated, which is not a big deal as manual testers hold the knowledge in their brains.
In automated testing, strict checking is enforced. This means the execution of the change-affected test steps would fail. A single failed automated test step means a test failure, and a single test failure means a broken build.
Unmaintainable automated test scripts are worse than useless. — Zhimin Zhan
To give a more concrete perspective, below is the Line of Script (LOS) for my WhenWise regression (Selenium) suite.
+------------+---------+---------+---------+--------+
| TEST | LINES | SUITES | CASES | LOC |
| | 26183 | 336 | 561 | 20532 |
+------------+---------+---------+---------+--------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 9883 | 181 | 1625 | 7466 |
+------------+---------+---------+---------+--------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 823 | 5 | 61 | 637 |
+------------+---------+---------+---------+--------+
| TOTAL | 36889 | | | 28635 |
+------------+---------+---------+---------+--------+To get a green Continuous Testing run, each of ~30000 test steps (1 user operation does not exactly equal to 1 step, as there is script reuse) must pass. A test step represents a user operation.
setup_wizard_page = SetupWizardPage.new(driver) #verify on this page
setup_wizard_page.enter_business_phone("07 33558899")
setup_wizard_page.enter_business_display_address("City Library")
setup_wizard_page.click_continue_businessAstute readers will know that the effort for maintaining an automated test suite grows exponentially with an increasing number of test cases. The effort of maintaining a 50-test suite is NOT double of a 25-test suite, more like five times.
“To get 100% accuracy for a handful of three-pointers in a row is not hard for a professional basketball player; the record is 105 by Steph Curry. Please note, Steph did that statically, the same position with no interference. In day-to-day UI test automation, there are many moving parts.“— Zhimin Zhan
Now, you probably agree with the experts’ quotes I showed earlier: GUI Test Automation is very difficult, much harder than coding.
Reality
During my 16+ years working in test automation and continuous testing, in my city, the maximum pass rate of automated End-to-End tests (50+) in a CI server I recalled was merely 48%. That’s very low, as the target rate is 100% valid (PASS or Correctly Failed). Yes, I achieved that every day since 2011 for all projects, with only a few exceptions on major upgrades of underlying frameworks.
If I don’t keep all my automated test scripts valid today, it will be even harder tomorrow. — Zhimin Zhan
In other words, every so-called agile project I visited was 100% faking automated testing (and therefore, fake Agile/DevOps). That’s before I rescued some of them.
Real test automation is possible but needs the right attitude first
After accepting that automated UI testing is a very challenging job, with determination, we can really understand and make real progress towards the goal. Here, I will show common scenarios.
The test script execution is not reliable
Make it reliable, don’t consider the job done once you get one successful execution. Run it many times, and consider more scenarios (the test data, time/day of execution, …). Check out this article “Working Automated Test ≠ Good Reliable Test”, a case study of how my daughter quickly created a working but unreliable automated test for a summer job interview.Some tests always failed in the overnight build
Why not run them multiple times during the day? In fact, I rarely run tests overnight. I sleep well after knowing all my test suites are green.Always false alarms when running in our CI server
You shall not run automated E2E tests in CI servers, e.g. Jenkins or Bamboo, not without customization (inside or add-on). Think of the Basketball 3-pointers metaphor, it is impossible to pass 200 user-story level tests (~10,000 test steps) in a row, as the web apps nowadays are far more dynamic, and a single longer-than-expected JavaScript execution might break the build.
The answer is simple, using a real CT server such as free, open-source and award-winning BuildWise, its auto-retry (not retry in framework, which is a wrong practice) feature can greatly reduce false alarms.
As you can see, if we do care to “listen” to the feedback from problems encountered in executing automated tests, we can work out solutions, i.e. changes. Unfortunately, most are just too stubborn to change. For examples:
Using Java or C#😰 as the scripting language
Wrong: test scripts shall be a scripting language, e.g. Ruby 😊.Using JavaScript as the scripting language
Wrong: JavaScript 😰 is a bad language for real test automation. The decision to use JS is often heavily influenced by programmers, who know (or care) very little about test automation. Solution: Ruby 😊.
(By the way, I do know JS. I authored one book on Test Automation using JavaScript and educative converted it into an interactive online course; Hey, I am an open-minded engineer).Using Cypress 😰
Wrong: it has too many limitations. It is a joke to use a crippled testing framework for verification, especially since the best (in every way) and standard-based Selenium WebDriver 😊 is completely free.Using Cucumber or another Gherkin 😰 based syntax framework
Wrong: It will introduce too much maintenance overhead with little value. Shall use RSpec 😊.Using Jenkins or Bamboo 😰 alike CI servers to run automated UI tests
Wrong: CI servers are built for running unit tests. Execution of automated E2E tests is long and brittle, and shall be run in a CT server such as BuildWise 😊 and Facebook’s Sandcastle.Using programming editors/IDEs such as Visual Studio Code for developing/debugging automated tests
It is feasible but not efficient. It is far better to use a dedicated functional testing IDE such as TestWise 😊.
Even if you lack skill, you can still do limited Real test automation and then grow
After seeing real test automation and my metaphor of 3-pointer shooting, some prideful automated testers probably want to give in: “it is too hard, I don’t want to carry a fake label”. Please don’t.
You can learn
It is actually quite fun to learn test automation and continuous testing. If not for fun, I wouldn't have switched from a Java Tech Lead to an automated tester in 2010. After mastering test automation and CT, I created (and maintained) a handful of highly claimed (one international award-winning) software products.
Check out my books and articles here.Run a small set of automated UI tests well every day
You don’t have to automate every user story, 100% coverage is simply not possible, and neither necessary.
If you do get 20 automated tests (even just simple ones such as “sign-in”, “sign up”, and “access privileges check”) always valid, that is, All PASS at least once every working day, you are better than 90% of automated testers (Level 1 of AgileWay Continuous Testing Grading). More importantly, you are providing test automation value to the team. You can confidently say this to the team: “Manual testers don’t need to test ‘user sign up’, …., as it is covered by automated tests”.
Every morning, the first thing you do is to trigger a run of all regression tests (automated UI) in a CT server. Trust me, even with a very small test suite (e.g 20 tests), you will find many regression errors along with development. Then, your team members would appreciate your work. This positive feedback will encourage you to learn more. Next target: 50 tests, the Level 2 of AgileWay Continuous Testing Grading. Once you reach that, you will be better than 99% of automated testers.
More importantly, you are doing the work (automated testing) real! Every day, you go home with a green build and a sense of accomplishment!
