An Unbelievable True Story: People’s Fixation on an Obviously Failed “Self-Created” Test Automation Framework

While it is common that management sticks with a wrong decision, it is rare to this insane degree!

This silly person hits his own foot, image credit: https://www.sohu.com/a/231853936_642840

This article is one of the Stories series.

Since I switched my day job from Programming to Test Automation 13 years ago, I have worked on many software projects (as a contract test automation engineer). I usually worked on a three or six-month contract, and I care less about the contract rate. A market rate was fine with me (given my expertise, several testing conferences’ brochures listed me as one of the world’s leading test automation experts). I worked as a normal automated tester, regardless of title, as long as I could do hands-on work on automated test scripts. Also, I declined any management roles or work involved with many meetings (such as testing strategy).

While working on these projects, I wanted to

• prove my test automation formula works for all web apps.
  (Yes)

• broad my knowledge to cover tricky scenarios
  (I am particularly interested in the websites that people claim an impossible mission to automate)

I worked like this for about six years and improved my test automation efficiency further. Because of these experiences, I could do test creation live for clients on the first meet. It may sound like bragging, but I am confident that I have been at least 10X more productive than any other senior test automation engineers I have personally met.

Along the way, I have encountered some unbelievable, quite frustrating at the time but interesting stories. I once heard “Life is about stories”, and to experience some unbelievable ones is an asset. Today, I will share one.

Table of Contents:
∘ Background
∘ A Bad Self-Created Automation Framework
∘ The project’s Test Automation was already struggling
∘ I Skipped a trap to get into managing Continuous Testing
∘ The Project Gave In
∘ The Success
∘ Insane: Back to MadX again in Project A
∘ What were the reasons behind that silly decision?

Background:

I joined a medium-sized software consulting company in 2016 as a test automation engineer, for an insurance system upgrade project. Two similar projects were happening at the same time. I was initially assigned to Project A, which was a little delayed, so I started working on Project B.

A Bad Self-Created Automation Framework

A few engineers there had created a new “framework”: MadX, which was just a syntax tier on top of Selenium. They claimed this would make test automation easier and more maintainable. (wrong, of course. Please read my other article, Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver)

I decide not to disclose the framework and the company name. Those working there were nice people. They just didn’t know test automation and CI/CD at all, but somehow they believed they could provide consultance on these two area.

This was the MadX releases page on GitHub.

The last release, never reached Beta.

As you can see, the v2 was never released. The whole thing was dumped eventually.

Ironically, the MadX (they called it a tool as well) page still exists today.

MadX home page.

Of course, the claims are ALL FALSE:

• “make your life easier”
  As a matter of fact, it makes everyone's life much harder.

• “Developers love it”
  Developers hate it (one of the reasons it got abandoned)

• “Tester can actually use it”
  But after using it, they hate it.

The framework works like this (from their official documentation):

1. A sample test script, GoogleTest.grass

invokeUrl = http://www.google.com.au/
google_searchBox.value = {framework} test automation
google_searchButton.clickLink
waitSeconds = 1
verifyText = <em>Test automation</em> for the rest of us.

2. The definitions of the controls, google.grass

#Google Search
searchBox.name=q
searchButton.name=btnG

Basically, “the framework designer” naively believed this syntax was easier to use, and some readers might agree too. I will write a separate article on why this text-based test script syntax is wrong, the failure of FitNesse was proof. For now, accept the fact that this ‘framework’ failed badly.

The project’s Test Automation was already struggling

After seeing the above test script example, I immediately knew it was silly and would fail, if not already. In fact, that was why they hired me to rescue the test automation, but with the wrong assumption: correct test automation in their way.

A clear and objective sign was that the framework had its own defecting tracking register 😱. You can picture what a mess it was.

According to the agile process (the company promoted it highly), a user story needed to be tested automatedly (which was good) using their framework. So, a team of five manual testers (now plus me) needed to work on the “easy” framework.

The testing did not go well in the first two weeks and got much worse day by day.

I managed to work on them, with frustration of course. As always, I did my assigned test cases in Selenium Ruby first. Then I converted them into the MadX using a Ruby script I created.

I Skipped a trap to get into managing Continuous Testing

Even with MadX, I exhibited high productivity (relatively). The project manager casually tried to talk me into managing test execution in CI/CD (which really means CT, running end-to-end tests).

Having worked in this area for years, I knew it was a trap. If I naively said “Yes”, then all the sh*tty stuff would become my problem. However, I didn’t want to reject it bluntly.

I asked: “Are the tests running every day?”

PM: “Sure, every night”.

I asked: “Did they pass?”

PM: “Yes” (lie)

I said in a relieved tone, “OK, Can you show me the daily test execution report in Bamboo (the CI server the company uses)?”

PM: “Sure”.

He opened Bamboo in a browser and explored it for quite a while before he showed it to me. There had been no single build that was successful since the previous month. In fact, the average pass rate recently was a miserable ~30%.

PM grinned, “Not as good as we hoped. But you can make it work, right?”