Analysis of "4th Edition Software Testing & Quality Report" by TestRail. Part 2 *

Trends and Findings from a recent software testing survey.

Jul 17, 2025

∙ Paid

Continue from Part 1 , more findings from the latest Software Testing & Quality Report (2025).

Finding 4: Most teams are still incorrectly using CI servers to run end-to-end (E2E) automated tests.

Jenkins and GitHub Actions (…) are CI servers primarily designed for running unit and integration tests (i.e., white-box tests). They are not well-suited for executing automated end-to-end (UI) tests, as they lack key Continuous Testing (CT) features such as dynamic test ordering, automatic reruns, manual reruns, delayed completion, and more. It's no surprise that failure rates are extremely high (~99.9%) when using CI servers for E2E test automation.

Check out my article:

Case Study: Rescue Unreliable 20 hours of Automated Regression Testing in Jenkins ⇒ 6-Minute Highly-Reliable in BuildWise CT Server

Finding 5: Software Test Automation pain points

The pain points highlighted by over 2,600 survey respondents are fairly common—and, in truth, not all that different from what I saw more than 15 years ago.

Keep reading with a 7-day free trial

Subscribe to The Agile Way to keep reading this post and get 7 days of free access to the full post archives.