Expose a Common Lie, “Selenium is Flaky”, Part 2: Clarify ‘automated waiting’

The reason behind ‘flakiness’ is waiting. ‘Automated Testing’ is not that good, there is a much better and simpler solution.

This article is one of the “IT Terminology Clarified” series.

From Part 1, sensible people should agree that Selenium is not flaky, it is the test script issue. In other words, it is to do the tester’s capability.

The so-called “flaky” issue is all about waits in test scripts. AJAX and the use of JavaScript make web pages dynamic. This surely adds some challenges to web test automation because we need to check certain operations whether completed in the test scripts. If an operation, e.g. a payment, takes longer than expected, the next assertion for the receipt number will fail.

By the way, Selenium has two mechanisms to handle the waits: Implicit and Explicit Waits. Again, Selenium is not the issue. But, don’t forget most automated testers are fake (not believing, read this article, One Simple Test Automation Scenario Interview Question that Most Candidates Failed), fake automated testers often actively seek excuses. To them, after failure, a logical way is to switch to another framework, giving them extra fake working time.

To match the needs of fake automated testers, some bad frameworks, such as Cypress, come to the market, claiming “automatically waits for elements to become visible”[link], and similar stuff from TestCafe.

Table of Contents:
· Did you ever witness a single successful Cypress, TestCafe or Puputteer?
· The story of a Principal Software Engineer's fear of me showing Selenium handling dynamic JS/AJAX
· Clarify Waits
∘ Selenium's way of handling waits
∘ So-called “Automated Waiting”
· Why I don’t use “Automated Waiting” then?
∘ A Simpler Alternative

Did you ever witness a single successful Cypress, TestCafe or Puppeteer?

If I started with “automated waiting from Cypress or TestCafe is no good”, many fake automated testers would debate. Over the years, I learned (after rescuing a number of bad TestCafe/Cypress failures back to raw Selenium) that debating on a specific feature with fakers is in vain. Instead, I ask objective questions.

Cypress/TestCafe/Puppeteer/Playwright testers, try to answer this objective question: “Have you ever witnessed test automation success once?” A logical tester will ask, “What is the definition of test automation success?” Check out the objective Definition of End-to-End Test Automation Success.

For readers who don’t bother to open another page, let me lay down the success criteria in simple words:

• 50+ automated end-to-end tests that run daily and remain valid

There are measurable items:

• 50+ tests
  Fifty end-to-end functional tests, by any measure for a web app, is not much, right? Of course, here I mean a good mix of real and useful test cases, not 50 login tests.
  If a project development duration is one year, average 1 test per week. Surely, there is very, very low expectation.

• run daily
  It shall be multiple times a day. Otherwise, a bad code checked-in won’t be detected for days. This will only lead to more regression errors.

• remain valid
  no arguments here, right?

Now, I bet, 99% of automated testers have never witnessed one real test automation success. Frankly, the bar I set (in 2012) was 100 tests (as a minimum for success), but over the years, reality told me that 50 was already too much for most software projects. Check out this real story, An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’.

Please note, as I provided in Part 1, the end-to-end regression suites of my ClinicWise and WhenWise have 611 and 559 raw Selenium tests, respectively.
A engineer will know the difficultly grows exponentially with the number in this case. For example, to achieve 99.99% relaiblity would cost 10+ times of 99.9%.

Frankly, I am not able to achieve 500+ if using Cypress or Playwright, because of all sorts of issues, including being flaky.

To real test autoation engineers, raw Selenium WebDriver with Ruby is way way better that any of those, in every aspect. Because Selenium WebDriver is completely free, few sing for it. I do it here, of course, there is no commerical factors (Selenium is not commeralized)

So, for testers who believed the lie “Selenium is flaky” and embraced Cypress/TestCafe/Puppeteer/Playwright, the so-called “automated waiting” did not help them succeed in real test automation.

The story of a Principal Software Engineer's fear of me showing Selenium handling dynamic JS/AJAX

A few years ago, I was approached to join a financial organization. An interview process was still required (but the outcome has been decided, they wanted my help). M, the principal software engineer, interviewed me.

Later, I learned that M was responsible for a huge embarrassing test automation failure, not long ago. M thought he knew a bit about test automation. He and a few programmers started a solution using Selenium Java and Concordion (the choices of language and test syntax framework were wrong). It turned out to be a disaster, I heard of it the month before at a different company. One former colleague who worked there summarized its failure in one sentence, “the project spent three times of development (coding) effort trying to maintain the test suite and failed”. The company recently did a proof of concept of TestCafe but seemed not satisfactory.

Anyway, I did not know M’s involvement in test automation at the interview time.

Because it was a casual interview, after greetings, M only asked one technical question, “How do you automated test dynamic pages?”

I answered, “It is easy in Selenium WebDriver.”

M said in a doubtful tone, “I mean, one-page website. I heard Selenium can’t test that.”

I said confidently, “Of course, Selenium can. I can show you one right now”, then I reached for my laptop.

M reacted quickly, “No, no, you don’t need to show me”, and then concluded the interview.

I think, from the bottom of M’s heart, he knew Selenium WebDriver can handle AJAX and dynamic web pages (from Facebook’s and Google’s success). But he did not know how, worse, he did not want to find out.

Later, I implemented the 3-month proof of concept (POC) in three days using raw Selenium WebRriver with Ruby. The POC work was to implement a set of automated tests from the failed Java implementation, with no help, I think M wanted me to fail. But he did not expect me, a tester, a much better Java programmer than him. I could read and analyse the bad java test code (five tiers of abstraction, a mess, no wonder it failed so badly) he and his team wrote.

Anyway, after my successful POC (complete in 3 days, rather than the expected three months), he and another tech lead somehow convinced management to insist on implementing it in Java. I rejected and left the company.

Clarify Waits

Let’s talk about the waits, the real reason behind the common so-called ‘flakiness’ in test automation execution.

Selenium's way of handling waits

Firstly, let’s see how Selenium does waits. There are two: Implicit Wait and Explicit Wait. You can safely ignore the Implicit one, I cannot think of a meaningful use case for it.