How to Expose Fake UI Test Automation in Fake Agile companies?
“Agile” claimed by most software companies claimed is fake. Just ask a simple question: “Show me the automated functional testing report in your CI/CT server?”
Over the last decade, many software projects have claimed to do Agile/DevOps, with some degrees of UI Test Automation and Continuous Testing. The reality is, most were faking it. Michael Feathers, a renowned author and agile expert, wrote an insightful post (2009), in which he said: “everyone does” (faking UI test automation). I agree! The situation, from my observation, improved little, “nearly everyone fakes it”.
Some might wonder, how could that be? “Our company has a dedicated DevOps team, dedicated agile coaches”; Or “our project was outsourced to an agile software service company, they even showed us good-looking charts, test reports”, …, etc.
First of all, the chance of finding a real Agile company (doing real UI test automation) is very slim. Don’t just take my word for it. Check out this article on the most recent World Quality Report: 5 Reasons Enterprise Test Automation Is So Challenging. “Based on research Tricentis conducted from 2015 to 2018, …, on Global 2000 enterprises, automated dismal 8% of end-to-end functional tests”. Please note, this is just “have”. It is far from comprehensive, not guaranteed high-pass rates, does not run daily, not if-pass-all-deploy-to-production. It would be safe to say, only < 1% of that 8% can achieve the goal.
Does it really mean that those so-called Agile Consulting are lying? Statistically, yes. But to be fair to them, lacking knowledge of Automated UI Testing and Continuous Integration is common, and it is a failure of our IT education. For example, my daughter is currently studying Computer Science at the University of Queensland, where the “Penetration Testing” course is offered. However, no “Automated Functional Testing” or “Continuous Integration” courses. Our IT courses are so distant from reality. For over the last 5 years, there have been more functional testers than programmers in every software project I worked on or consulted. I was extremely fortunate to have the opportunity to pair-programed (for 6 weeks) with a world-renowned programmer, my mentor, who taught me automated functional testing and Continuous Integration, and also introduced Ruby, a wonderful scripting language.
Why do you want to do Agile? Certainly not for all sorts of charts, right? You want to be able to release high-quality software frequently. Yes, that’s real Agile, automated End-2-End regression testing makes “Release early, release often” possible, together with many other benefits. I have developed feature-rich ClinicWise, WhenWise, SiteWise web apps (plus the international award-winning BuildWise server app, TestWise desktop app) mostly in my spare time. The reason I could implement and maintain all of these apps: my Continuous Testing process: executing automated end-to-end Selenium/Appium tests regularly.
Agile software delivery companies charge clients for the development time. Most fake Agile software consulting companies might think releasing early would conflict with their commercial interests. However, it is wrong. That’s because they have never done a real agile project before. The result is quite the opposite. If software development was highly productive and the product was high quality, there would be more opportunities.
Just have a look at Apple’s product list after Steve Jobs back to Apple, one product helped another product: iPod -> iTunes; Xcode; iPhone -> iPad -> Apple Watch; Mac OS X -> iOS -> new macOS, .., etc. Some might argue: that it was Steve Jobs and Apple. How could anyone/company compare against it? That’s true. Take me, an ordinary IT engineer, as an example. Before 2007, I had no single software product of my own. After mastering test automation and Continuous Testing, I am now a proud owner of TestWise, ClinicWise, SiteWise, BuildWise, and WhenWise. So, if you hired/engaged agile coaches and DevOps engineers, and didn’t see a big improvement in productivity and software quality, oh well, you are not lucky: you got the fake ones.
How do fake agile software companies fake their UI test automation capabilities?
Showcase a perfect demo testing project
Once I was helping a friend (a PM) advising an outsourced project. The vendor was a small service delivery company who claimed that they had a good process of CI/CD with UI test automation. I spotted the problems immediately with their showcase project. The tests were too simple, and the scripts were not flexible. Not long after, we saw many rudimentary bugs made to the test server, passing their so-called ‘robust’ CI/CD process.
Record test scripts afterwards
The majority of automated UI functional testing efforts are on maintenance. Creating test scripts only took about 10–20% of your efforts. Test scripting maintenance, when the applications keep changing, requires high-level skills, efficiency, and dedication. The reward for that effort is great. check out this Facebook’s presentation), However, many never experienced it. Consequently, they regard test script maintenance as a burden. Hence, a common trick to fake test automation is, after the software is nearly ready, to create a set of automated tests with a recording tool (such as a Cypress recorder).
Those automated scripts have little value. They
did not provide valuable feedback when a feature was implemented
did not detect regression errors (early detection can greatly reduce development/debugging efforts)
will become obsolete soon, as the scripts are unmaintainable.
Another common trick is ‘swapping the concept’ using API (some with the fancier name: microservice) testing as ‘end-to-end testing’. This doesn't seem right if your end-users interact with UI. As the World Quality Report 2018–19 shows: “end-user satisfaction” is the top objective of quality assurance and software testing strategy.
Alarming keywords: UFT, Cypress, Cucumber, SpecFlow, Jenkins, Bamboo. When these frameworks/tools are used in UI Test Automation and Continuous Testing, be aware. I have worked in this field for 15 years. I have witnessed that every project that has used one of these technologies failed. According to Michael Feather’s observations (15 years ago), this is not news.
Now here comes the simple way to expose those lies. In the service contract, try to add this clause:
THE END-2-END FUNCTIONAL UI TESTING MUST BE CONDUCTED AT LEAST TWICE A DAY. THE TEST RESULTS MUST BE TRANSPARENT.
90+% of USER STORIES MUST HAVE END-2-END AUTOMATED TESTS.
Proper Reports should be like the ones below:
for each CT Build, full test results (and test script content) are displayed. Fixing the test failures is the team’s top priority (refer to Kent Beck’s famous Agile book: Extreme Programming Explained: Embrace Change).
Those ‘agile’ companies shall not object to the above, as that’s what they advertised. But they will be distraught; you put them into an objectively measurable cause. They would find it hard to say NO. Any excuses would put you in a much better position to negotiate.
Lastly, one real-life case. This is a government project (a sister project of ours) outsourced to a medium-sized software company. The PM saw how we did Continuous Testing (we were located on the same floor), and he then asked the company to add automated tests as a part of delivery (like my project). The company’s answer: “It will cost extra, double the current contract amount”. I visited this company later; they did not have the test automation capability at all.