A Story: One Former Mentee Failed Test Automation with Protractor

The price to pay for not following a proven winning formula.

Besides my own software apps, I have been implementing real test automation & Continuous Testing for a number of client projects by using the exact same formula. The test automation framework used is raw Selenium WebDriver in RSpec.

JavaScript has been the favourite language in recent years and is used in Protractor, TestCafe, Puppeteer, Cypress and Playwright. As a result, many automation testers (though some of them knew Ruby is better, including those whom I mentored) went for Javascript for more job opportunities or give in under the pressure. Here I have an interesting real story to share.

The Story

Many years ago, I took a tech lead role in a small start-up company for 3 months. Besides all the usual hands-on development /testing /DevOps tasks, I also mentored the team.

F was a final-year Uni student when I met him, who had already been there working as a part-time manual tester. I taught him automated testing, Ruby, Git, Continuous Testing, and later, app development. He was very smart (I vaguely remembered he received a Uni Medal for his academic records) and learnt well during my 3-month consultation role there. By the time I left the company, I had set up the regression suite consisting of 148 Selenium RSpec tests that were running fine daily in a BuildWise CT Server with 3 build agents running on three Mac minis. F had been involved in real test automation hands-on under the proper mentoring.

Below is the thank-you email he sent to me after I left.

A few months later, a former colleague met me on the street. He told me the automated regression testing was discarded as the boss wanted the team to just focus on developing new features, ignoring test failures in the CT server. Shortly after, all kinds of quality issues came up. The boss and their partner had to get involved in the manual testing, even on weekends. About a year later, the company went bankrupt.

Several years passed. In October 2019, I received F’s invitation to a get-together lunch, and I went.

During the catchup, F said that he had been working as a developer since graduation. He thanked me again for my mentoring and asked for my advice on implementing test automation in this new job. I replied: “raw Selenium WebDriver in RSpec, exactly the same as what you did before”. He said: “The company’s app is developed in JavaScript”. I smiled and said: “It doesn’t matter. Functional UI testing is irrelevant to what language you use, you shall know that”.

It's a pity. F is really smart. If F had followed the exact formula I coached him on, I believe he could have successfully developed and maintained at least a 50-test End-to-End Regression Suite. This would have greatly helped his career. I think F knew that too (from his email).

However, when he began making changes to the proven formula, the challenge grew significantly larger, and things quickly spiraled out of control.

Two years later (August 2021), F sent me this LinkedIn message. (F’s job title, according to his LinkedIn Profile, is Senior Software Engineer)

Apparently, he ignored my advice. The result turned out as exactly what I had predicted. From his short message, I can see a few problems:

• TypeScript
  This is a wrong choice, of course, not by F. But he dared not challenge that.

• Flaky E2E Tests
  It seems ‘a news’ to him, but it shouldn’t be the case. During the mentoring, I showed him many ways to improve the reliability of test execution, like some in this article: Working Automated Test ≠ Good Reliable Test.

• Running Automated E2E Tests in Jenkins
  This is wrong too. Jenkins is a CI server that is built for executing unit tests. They should use a CT server, which he realized (reflected in his last sentence). Check out this great presentation “Continuous Integration at Facebook”.

• Scheduling E2E test execution every hour
  Wrong again. The mindset behind this bad practice is to assume that test execution is repeatedly reliable. I will cover this in a separate article.

Above all, it was the bad choice of test automation framework: Protractor 😥. It should be raw Selenium WebDriver with RSpec, the winning formula he used years ago. By the way, Protractor was deprecated in 2021, I was not aware of that when I received F’s message.

Clearly, F still wanted to stick with Protractor. After failing with Jenkins (not surprisingly), he wished to use my BuildWise CT server (that he has used before under my mentoring).

My reply to him: “Yes, BuildWise supports RSpec, Cucumber, Pytest and Mocha. If your Protractor tests are in Mocha, they can be run in BuildWise. However, I will still strongly recommend RSpec.” In addition, I answered several other questions and provided some resource links.

F thanked me for that, and that was the last time I heard from him. Obviously, he did not take my advice, more likely, lacking the courage. He changed his job a few months later (from a LinkedIn update).

FAQ