How do I OverCome the E2E Test Automation Challenges Posted in DHH’s Article?

My approach to overcoming slow, brittle, and false negatives in E2E Test Automation

DHH, the creator of Ruby on Rails and my idol & hero, published an article titled “System tests have failed”, in which DHH expressed removing most of HEY’s 300 system tests due to “wasted far more time getting system tests to work reliably”. I wrote one article to share my thoughts on it.

Some might wonder, “Even DHH or 37signals had difficulty maintaining 300-odd system tests (automated E2E UI tests). How can you do 500+?”

Before answering that, 300 is a very high achievement already. According to the AgileWay Continuous Testing Grading, it is Level 3+, better than 99.5%.

Whether you believe I can maintain 500+ E2E (UI) tests, I share my approach to overcoming the challenges that DHH mentioned in his article.

“System tests remain as slow, brittle, and full of false negatives as they did a decade ago.”

For the above three challenges in E2E test automation, 37signals engineers have tried hard to resolve them with some degree of success, otherwise, they wouldn’t have been able to reach the 300-test suite. A logical explanation is that their approach and effort were insufficient to handle a large test suite.

1. Slow

End-to-end (E2E) tests run significantly slower than unit and integration tests, often by one or two orders of magnitude. Considering 37signals’ strong financial standing, with annual profits in the double-digit millions, it’s highly likely they used top-tier hardware to execute those tests.

Fast machines help, but not that much in the context of E2E Test execution (driving elements in a browser + server’s processing time). As DHH did not reveal its test execution infrastructure. I will illustrate with an example of mine.

Executing WhenWise’s E2E test suite (570 raw Selenium + RSpec tests) takes ~168 minutes on my Mac Mini M1 (cost A$999).

Suppose money is not a problem, I purchased a top-specified Mac Studio M2 Ultra (for ~A$9000). Executing the suite on this fast machine, assuming 300% faster, would still take 56 minutes.

Over a decade ago, I knew a faster machine was NOT the solution for E2E test execution for a large suite. (see my article, My Continuous Testing Journey). The only practical (and affordable) solution to parallel execution in a Continuous Testing Server.

WhenWise suite, with just 5 build machines (3 of them were slow old machines, 2012 MacMini Intel i3 3210M), I achieved 42 minutes of execution time in the BuildWise CT server.

That’s why Facebook could achieve <10 minute feedback time,

thanks to its massive test lab.

So was Apple.

https://appleinsider.com/articles/20/12/27/old-photos-show-original-iphone-production-line-from-2007

Again, I don’t know how 37signals executes its 300 E2E tests. But, certainly, by choosing a good CT server and with some hardware investment (no problem at all for 37signals, and the test lab setup is fully reusable for other apps), effective parallelization can overcome the ‘slow’ problem.

My Innovative Solution to Continuous Testing: Parallel Automated End-to-End Test Execution

2. Brittle