A Robust Automated E2E Regression Testing Process Enables Real Performance Testing
A Safety Net That Empowers Exploration for Performance Enhancement.
In software development, performance testing is usually conducted towards the end, within a short period, e.g. one or two weeks. If the performance issues are identified, most software teams can only perform non-code-change optimizations, such as:
Add database indexes
If database-intensive operations, such as reporting, are taking a long time, check the database indexes. Indexing is a critical part of database optimization, and can dramatically increase query speed.Upgrade the machine
Faster CPU, more memory and machines help, to a certain degree.Specific to web apps
- Enable Gzip Compression
- Minify JavaScript and CSS
- Prefetching JavaScript
- CDN (Content Delivery Network)
The above are general performance-enhancement techniques, but they are limited because most performance issues usually stem from inefficient code. A well-structured software development process should empower skilled programmers to address performance optimization directly during coding.
As we know, the real and big performance gain is optimizing the code (front end and back end), in particular, the database structure. However, few software teams dare to do that. Why? New code changes almost certainly will cause regression issues, i.e., functional errors. The team could not afford another full-round regression testing, which usually takes weeks, unless this team has implemented Real Automated End to End Regression Testing (via UI).
I worked with many contractor programmers, and a fair percentage of them had worked on their side projects. Sometimes, we talked about that and demonstrated it to each other during the lunch breaks. One thing I noticed, their apps were quite slow compared to mine. Several colleagues were shocked after learning:
My apps were coded in Ruby
I worked as a Java Contractor for quite a long time. Java/C# programmers tend to think scripting languages such as Ruby and Python are much slower compared to compiled languages, which is true.Hosted in $10/month cloud machines
My hosting cost is also significantly lower than my colleagues. I knew that by hearing their complaints about the high AWS or Azure bills.
For example, my two test automation practice sites are hosted on a $6/month Vutlr instance. They are quite snappy.
How could I achieve much higher performance on much cheaper cloud infrastructure, and using a much slower coding language? I do “Continuous Performance Testing”, not every day, but rather, as needed base. Because it is easy to perform: trigger the “Build Now” button for a performance build project on the BuildWise CT Server.
After writing the above, I just triggered one round of performance testing (one suite) for my WhenWise app. Here is the test report (after 23 seconds).
Here is a test execution chart for a particular operation, “Reset database (quick)”, an important utility used in test automation (check out my article, My Innovative Solution to Test Automation: ‘The Simpsons’ Data Reset Pattern).
As you can see, I have done several rounds of performance optimization, reducing the time from ~1 seconds to 0.3 seconds, saving of 70%. Almost every Selenium test in the regression suite calls this utility. There are 559 Selenium tests, so this performance optimization alone saves 6 minutes of execution time for each regression run.
The enabler of my ability to do Continuous Performance Testing anytime: Real Automated End-to-End (via) UI regression testing. I don’t fear code changes.
Below is a recent regression testing report for my WhenWise app, running 559 Selenium tests in a BuildWise CT Server.
In summary, without a mature process for Daily Automated End-to-End Functional Regression Testing, many software professionals hesitate to explore the numerous available ways to enhance app performance and reduce infrastructure costs.
Related reading:
