Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation is Surely Feasible for Large/Complex Apps. Part A: The Story and The Stats
Implementing End-to-End Test Automation, the most valuable type of testing, is most QA Engineers’ responsibility. If others try to discourage you from doing E2E UI automated tests, share this article
A repost of my past article on Medium
(Substack suggested the original article is too long, so I split it into three, all free to read)
Last night (2023-08-09), I saw this in my LinkedIn feed.
The opinion of “avoiding end-to-end tests” is quite common, and I usually disregard it. However, this particular instance caught my attention because it specifically suggested doing just 5 E2E tests instead of 500. I rarely respond to others’ posts, but I couldn’t resist leaving a comment with a link to my article to express my disagreement: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions” (2021).
This led to this article.
A Tester challenged me, “WhenWise might be just a simple demo site”
To my surprise, one tester started to question the complexity of the WhenWise app. He was aware of the WhenWise practice site, so why not just click the link, and log in with the provided username/password? (quite logical for a software tester, right?) He would know in seconds. Instead, he was making a fool of himself. A typical behaviour of fake testers: like to talk, NOT hands-on doing.
1. Is that app a demo website meant for practicing test automation
2. How many real users does your application have?
3. How many features do you (as a developer) release on this application?
3. How complex is your underlying system?
4. What does the system architecture of this entire application look like?If the answer to these questions does not match with a platform like Facebook/AirBnB/LinkedIn/Spotify or any other similar exponentially complex platform out there in the world then you are comparing apples and oranges.
Let’s talk about real world systems. That’s where the real problems w.r.t flakiness happen with e2e tests.
In short, he did not believe that I could maintain a suite of 500+ E2E tests for a complex app. He is wrong! WhenWise is a feature-rich app (which I will prove in a moment), besides WhenWise, I developed and have been maintaining several large E2E test suites for ClinicWise, TestWise (a testing IDE, demonstrated at Agile 2009 Workshop), BuildWise (an international award-winning CT server), SiteWise CMS, .., etc, mostly in my spare time.
Be aware of this kind of person, who tried to bring up deceiving questions (muddling the water) to avoid the tasks at hand, in order to cover his incompetence. There are some obvious errors in his logic, in the context of the post (“5 E2E Tests instead of 500, due to flakiness”).
In real Agile, every user story should be covered by at least one automated End-to-End (via UI) test, otherwise, it is not “Done, Done”, Fake Agile. it is simplely a matter of skills and capability.
According to him, if the target app is feature-rich, just do 5 E2E tests; This is very wrong. Even if the target app is relatively simple, I highly doubt he could manage 20 E2E tests (Level 1 of AgileWay Continuous Testing Grading, Few test automation engineers who can actually achieve Level 2).
Why?
Most web apps face E2E test automation challenges in a similar way. This is obvious, the Web is the same (we automate the operations that a pre-schooler can perform on Chrome). The App’s system architecture is largely irrelevant, that’s why it is “Black-Box Testing”. On consulting, based on what customers show me, I implement real test cases live.
The real test automation challenge lies in maintenance, while the application changes frequently. Maintenance effort goes exponentially with more tests, the flakiness rate will go up, but should always remain extremely low at the test script level (otherwise, how could you a green run?). Flakiness in E2E tests is mostly due to the tester’s capability. I know that because I put a lot of effort into making E2E tests reliable.
Check out my article, Working Automated Test ≠ Good Reliable TestThe number of users in a system will impact load testing mostly, not necessarily on features, it s load testing vs functional testing.
Regardless your app is large/complex or small/simple, a sizeable (definitely not 5) Automated E2E test suite provides great value. After all, it is the foundation of Agile.
The real problem is that few automated testers (see Alan Page’s estimation below) can do proper E2E UI Test Automation, which means running a suite of E2E tests daily as regression testing. In my city, I haven’t yet seen a good implementation with over 50 tests (except for the test automation I led).
“For 95% of all software applications, automating the GUI is a waste of time. For the record, I typed 99% above first, then chickened out. I may change my mind again.” — Alan Page’s Blog (2008), author of How We Test Software at Microsoft
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
- this interview from Microsoft Test Guru Alan Page (2015)
Because there are so many fake ‘automated testers’, they would find all kinds of excuses. A Simple Question from a management perspective will reveal that:
“If a QA Engineer can only do 5 E2E Tests, Why should I hire him?”
Some would argue, “We are doing API testing, Cypress Component Testing or Visual Testing”. Don’t be deceived by those comments. A simple fact.
“Watir, Selenium WebDriver, WebDriverIO, PhantomJS, ProtractorJS, TestCafe, Puppeteer, Cypress and Playright, all of those were created for end-to-end test automation, driving web apps in browser”.
Fake automated testers, the fact hurts, right? As an owner of several software products, I definitely won’t keep these kinds of people around.
As “The World Quality Report” pointed out, product owners care mostly about End-User satisfaction, which translates to E2E Testing via UI in the context of software testing, right? Avoiding it?! That’s being a coward.
I mentored many testers, and I told them on the starting day: DO or LEARN. DO means hands-on working on test automation scripts with quick results (in minutes or hours), as the whole suite will be run daily as regression testing. Work is work, we don’t experiment with different frameworks here, check out my article, “How do I Start Test Automation on Day 1 of Onboarding a Software Project?” There are a considerable number of fake automated testers who felt so uncomfortable with hands-on scripting/maintaining and opted to do manual testing.
In this article, I will show some components of WhenWise and share some of my testing tips along the way.
My purpose here is to boost the confidence of QA Engineers: E2E Test Automation is feasible for complex apps, such as WhenWise. Even TestWise IDE, (this got to be considered large and complex, right?), which has 330 Appium + WinAppDriver E2E tests. For me, comprehensive E2E Test Automation is a must, as I do “Daily Production Releases” for all my apps.
Whether you can do real E2E Test Automation is simply a matter of capability, you can learn. Is it shameful to raise a ‘white flag’ easily and still call yourself ‘QA Engineer’ or “Test Automation Engineer”?
WhenWise E2E Test Suite Stats
Automation Framework: Selenium WebDriver
Test Syntax Framework: RSpec
Scripting Language: Ruby
Test Count (as of today, 2023–08–10): 569
Test Steps: 29,845
Duration: 5 and half years, from 2018–01–24 (the date of first execution in BuildWise CT Server) to present
Do you really maintain and run these E2E all those years? Yes.
Total Test Execution: 359,158
(Update January 2025: These 500+ E2E tests continue to be actively maintained, delivering significant benefits. See the article below, written 14 months later, noting an increase in the test count to 573.)
FAQ
How is it created? Developed in TestWise IDE
How is it run? Executed in the award-winning BuildWise CT Server
How complex is the WhenWise app? See below, I can give a number, though, 193 web pages or modals. How did I know? because E2E tests follow Maintainable Automated Test Design, using page classes. A simple Ruby script gets me the info.
How Long does it take to run the whole suite? ~50 minutes on 5/6 buildwise agents (run in parallel), ~3 hours 40 minutes if one machine.
Do you really run this suite (569 Selenium RSpec tests) as regression testing?
Yes. In fact, I just did two a couple of hours ago. Here is a recent execution report on the BuildWise CT server (note the time stamp).
Do you really push a build to WhenWise production after getting a green run of this E2E test suite?
Of course, I have been doing this for all my apps for over a decade. Yes, just released a new version of WhenWise. Otherwise, How could I manage developing/maintaining all my apps (in my spare time)?
What’s the major benefit of this E2E suite? Huge development productivity boost, WhenWise is just one of several apps I created (solo) and maintained in my spare time. Also, I do “daily production releases” if any code changes on that day. E2E Test Automation enabled all of that, plus a lot more.
Further reading:
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps realWhy Software Testing is Not Effective in Most Software Teams?
Benefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.