Analyse of World Quality Report 2023–2024: Recommendations
A set of recommendations that a software company can adopt today.
Capgemini has recently published World Quality Report 2023–2024 (PDF format), which outlines the latest trends and state of Quality Engineering. Some of its key recommendations align with what I have been saying/writing for years.
This kind of official report often contains jargon or correct-but-lack-substance texts, such as “Test Automation reduces costs”. Because of the commercial factors of this report (done by Capgemini and Sogeti, tech consulting companies), it is not surprising that a hyped trend, such as AI, was included in the recommendations.
I remain skeptical about the effectiveness of AI in End-to-End Testing in real-world projects. Check out my articles: ChatGPT has been out for over 6 months (now 13 months as 2024–01). Have you implemented real test automation with AI?
I will just focus on practical ones that ambitious IT executives can transform and see the results in a matter of days.
Table of Contents:
· On Agile Quality Management
∘ 1. “Invest in robust training programs”
∘ 2. Integrate with DevOps/DevSecOps
∘ 3. “Focus on an E2E testing approach to ensure seamless customer experience”
· On Automation
∘ 1. “Continue to expand on what works. “
∘ 2. “Automation can be prioritized against legacy systems”
∘ 3. “Set up an Automation Marketplace platform”
On Agile Quality Management
1. “Invest in robust training programs”
“Organizations must adopt a build-or-buy mentality. Developing a robust training program reduces overall costs while setting your organization up to continue to adapt to the industry’s ever-changing technology and skillset requirements.”
While most IT executives believe training is important and will reduce costs, few put it into practice in the field of Software Testing. Check out my article, A Story: “Promised E2E Test Automation Training Never Came”.
Throughout my 14 years of consulting experience, I have observed numerous instances where management consistently overlooked the testing team’s requests for test automation training.
Test automation stands out as the most sensible and high-return investment when it comes to training, for the reasons below:
Test automation is rarely taught at the University
Test Automation knowledge is universally applicable (testing all websites the same way)
Only a short time is required (i.e. affordable), and the results can be verified quickly
My Selenium training is only one day and costs ~$400 p.p for six attendants. If taking into complimentary ebooks and software licences valued at $205, that's under $200 per person.
2. Integrate with DevOps/DevSecOps:
Forging a seamless integration with DevOps/DevSecOps practices through close collaborations can ensure the smooth incorporation of quality engineering assets into CI/CD pipelines.
E2E Test Automation has little value if it isn’t run often, i.e., regression testing. In other words, the whole E2E suite needs to be run daily, like this showcase.
A common mistake is that incorporating E2E Test automation into CI/CD is too late. At one company meeting, I heard one manager asking the DevOps lead, “When can your team get E2E tests running in the CI/CD pipeline? It has been over 9 months”.
Check out my daughter’s article, “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.
3. “Focus on an E2E testing approach to ensure seamless customer experience”
“- through effective utilization of abstraction of various architecture layers.”
E2E (via UI) tests should be written in a format that customers (end-users) can understand. Unfortunately, developer-focused automation frameworks have gained popularity in recent years, particularly using JavaScript.
For example, the below is written by a Cypress authority who rated this a good version.
it('adds numbers via aliases', () => {
cy.visit('public/index.html')
cy.get('[name=a]').invoke('val').then(parseInt).as('a')
cy.get('[name=b]').invoke('val').then(parseInt).as('b')
cy.get('#result')
.invoke('text')
.then(parseInt)
.as('result')
.then(function () {
expect(this.a + this.b).to.eq(this.result)
})
})Do you think customers will like () => { , }) , invoke , then , function()?
How about the version below, Selenium WebDriver + RSpec based on Maintainable Automated Test Design, using Page Object Model?
it "add numbers" do
driver.get("public/index.html")
calc_page = CalcPage.new(driver)
a = calc_page.a
b = calc_page.b
expect(calc_page.result).to eq(a + b)
endIt is much better. Not just more readable, but more importantly, much easier to maintain.
On Automation
1. “Continue to expand on what works. “
“ — don’t adopt new automation technologies or platforms just because they are new or trendy.”
Check out my recent article (awarded Boost status by Medium curators), Correcting Wrong “Playwright’s Advantage over Selenium” Part 1: “Playwright is Modern and Faster than Selenium”.
Also, two recent independent surveys yielded a similar result:
The 13-year-old Selenium WebDriver leads by a huge margin.
2. “Automation can be prioritized against legacy systems”
“– focus on automating the test scripts and technologies which yield a larger and faster return on investment first.”
Commonly, test automation attempts (in the form of Proof of Concept) take too long (e.g. months), and then the business loses interest. Web Test Automation is generic; businesses shall get ROI immediately if they engage a real test automation engineer.
E2E (via UI) test automation can be and shall be delivering visible values from Day 1 and every day onwards.
3. “Set up an Automation Marketplace platform”
“— Let your automation engineers own the development and maintenance of the scripts while other teams integrate them into their daily routines.”
Frequently, senior developers or architects lacking familiarity with E2E test automation are assigned to oversee test automation processes and frameworks/tools. This often leads to the manifestation of the following incorrect behaviours:
Force QA engineers to use the same language as developers.
Check out my articles:
- Automated Test Scripts Shall be in the Syntax of a Scripting Language, Naturally!
- Why JavaScript Is Not a Suitable Language for Real Web Test Automation?Use traditional CI servers (built for unit/integration testing) for executing E2E tests
Check out my book, “Practical Continuous Testing, make Agile/DevOps real”Design a new ‘test automation framework”
Check out my article, “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”.
The solution is really simple: test it out. Check out Test Automation and Continuous Testing Competition Week.
Related reading:
Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
Test Creation Only Account for ~10% of Web Test Automation Efforts