Cypress Catches Up with Auto-Retry, but in a wrong way

Today (2020–08–21), I read an article on Medium: Flaky Tests: Retrying Failed Tests with Cypress. According to the author: ‘ this was one of the top “must-have” features I’ve been waiting for’. A must-have feature, I totally agree. However, why is it only available now in v5? Did the article's author suggest that Cypress testers had been more or less faking test automation all these years?!

Despite what they claimed, not one Cypress tester I have met could answer it when I asked for a history of test executions over 3 months. The teams performed functional testing most manually, resulting in buggy software and slow releases. There might be a good cypress test engineer somewhere who can create, and more importantly, maintain large user-story level UI tests (e.g. 200 test cases, Level 3 of AgileWay Continuous Testing Grading); In addition, the team needs to really trust the automation suite so that they can push the releases to the production, twice a day. However, I haven’t yet met one that can show reliable daily test executions of 50 cypress tests. Therefore, I remain sceptical.

For people who have known me or read my “Practical Continuous Testing” book, they would know that Auto-Retry has been a BuildWise feature from its first release ( internal: 2013, public: 2017, and won a major international award the next year). The idea of auto-retry came to me in 2007, the time when we solved test scripting maintenance with good test design practices (see my book: Practical Web Test Automation) and TestWise IDE. Soon we had to face the test execution reliability problem: It was difficult to get our 30-min test suite execution 100% pass. Then I came up with the Auto-Retry approach and implemented this feature in CruiseControl via a Java Plugin. When I decided to create my own Continuous Testing server, I was certain that Auto-Retry would be a must-have feature in Iteration 0.

It is good to know that other professionals accept and adopt my auto-retry approach. A few years ago, when my team presented our test automation to a fake chief agile coach, she commented: “ You shall not retry test execution!”. The team kept hard not to laugh until this chief agile coach left. After seeing real Agile, real Continuous Testing and real DevOps, the development team realised that auto-retry test execution in a Continuous Testing server was obviously a must.

Why is Cypress still wrong?

1. Test automation framework is not a W3C standard.

Microsoft’s recommendation: “With the WebDriver becoming a W3C standard, we are actively encouraging customers to use Selenium for web apps”. Please note, Microsoft dumped its own Coded UI testing tool and recommended Selenium.

Not just Microsoft, a few other large traditional test-automation tool vendors are now listed as “Selenium Level Sponsors” on Selenium’s home page, including MicroFocus (owner of HP testing tool suite).

I know the above facts will still not convince everyone to switch to Selenium. Some would argue how efficient or how fast they can start with Cypress. Before I get to the efficiency, I would like people to acknowledge that ‘Cypress’ is not a standard. As a result:

• Some testing might not be possible. Please see Alister Scott’s article.

• The technology behind the browser might change. Browser vendors only respect one standard: WebDriver.

• The next release of Cypress might break your test execution. Cypress is like any other software that will introduce bugs. I have experienced some Selenium WebDriver bugs too. Generally, the Selenium team can address them quickly. Again, it is easier to be done if it is based on a standard.

Back to efficiency, you can achieve great efficiency with raw Selenium WebDriver if you have the right tool (while the test scripts are 100% not dependent on the tool). While consulting, I sometimes wrote test scripts with a manual tester (on a projector) in TestWise. The effort used in creating a test script in the life of the test scripts is minor (if you really want a figure, I would say 10%). The challenge is to maintain a large test suite while the application frequently changes.

2. You cannot script Cypress in Ruby!

I firmly believe that Ruby is the best scripting language for automation. Some people will point out that this is only a personal view. Yes, it is. However, I will add some objective references. In the Agile Testing book (2009), Lisa Crispin and Janet Gregory recommended 3 frameworks: Watir (Ruby), Selenium and Canoo WebTest (Java). As we know now, two survived out of three: Selenium WebDriver and Watir. The wide-use Java one was gone. Watir started UI test automation and now becomes a Selenium-Variant (for being with a standard). The beautiful Ruby language played a big role in test automation (that is why Watir is named Web application testing in Ruby. The authors love Ruby, and the creator of Ruby on Rails had the same affection too). Above all, Ruby is fun! Without fun, it is hard to do functional testing well.

3. The features of test execution management shall be separated from the core test framework.

From the vendors’ point of view, the more tight-coupled they can put on the customers, the more financial benefits they get. In 1998, I met Richard Stallman (founder of FSF and GNU and creator of Emacs) at my research centre. I was inspired by him immensely. Since then, I have devoted my tie and efforts to free software (please note: “free” as in “free speech,” not as in “free beer” — Richard Stallman). I turned down commercial offers to add proprietary test syntax in TestWise. Since its creation in 2007, TestWise has been only supporting free and open-source test syntax.

Let’s go back to the test execution. In recent years, I have seen add-ons (e.g. parallel-test) to the test framework. It is wrong. Test script, by itself, shall focus on general test execution reliability and maintainability. The management to test execution, such as parallel and auto-retry, shall be left to the Continuous Testing server.

Why do most commercial vendors not do that? The answer is simple: money. If the testing framework, testing tool and continuous testing server are all independent, there will be competition. There might be individual goodwill engineers who can create a great tool/framework. However, their goodwill will succumb to the company structure pressure (from business partners and investors) if they work for a company. I have seen many testing tools fail, typically in record-n-playback (a common lock-you-down approach). I have been rejecting commercial interests in my testing tools: TestWise and BuildWise. I want to keep them pure for better test execution with 100% freedom. I could have been in a better financial position if I had betrayed my soul. But, hey, I use TestWise and BuildWise every day, which has enabled me to enjoy real agile development (no defect tracking, no JIRA, no burn chart, no retrospective, and production releases in hours if required). Our customers are happy as they get their features implemented in hours (Adding the new features is easy. Not breaking the existing features is the real challenge. That is where Continuous Testing of UI test automation shines). So am I.

4. There are still other ‘must-have’ features missing

Auto-retry is an important but often neglected feature for the Continuous Testing server. There are many other features such as Dynamical ordering, Manual rerun, Delay completion, Test allocation management, failure screenshot, quick-open-in-testing-tool, …., etc. You can find them in my free and open-source BuildWise server.

I know some Cypress testers would argue that Selenium WebDriver is hard to learn while Cypress’ UI and some of its features are good. I will not debate that, as the success of test automation is not judged on that. A more likely outcome will be a big embarrassment after throwing a lot of money into testing automation because people who decide o this do not really understand it. Over the last 2 decades, I have observed that many record-n-playback tools with all sorts of ‘features’ (even with crazy slogans ‘scriptless test automation’) come and go.

What I can tell you is that Selenium WebDriver is the easiest test automation framework to learn (remember, not mastering, like any skill, which will take time and practice), if provided with the right guidance and the right tool. If I list all Selenium WebDriver syntaxes slide by slide, of course, it will look hard. Trust me, there is a much better way to learn Selenium. Every motivated manual tester I have trained was able to work on real test cases the next day. Every sale of my book “ Practical Web Test Automation” proves that. A reader (manual tester) from Europe emailed me to share his excitement: 7 days after he finished reading my book, he received a Test automation engineer job offer to work on Selenium.

Update: See how you can learn the core Selenium in minutes by following my article (with video as well): “Step by Step Showing Why Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework”

Originally published on LinkedIn, August 21, 2020