Case Study: 548 Selenium Tests Help to Upgrade WhenWise to Rails 7 within 2 days
Real UI Test Automation and Continuous Testing is really useful, and tips on intensive automated regression testing.
I have witnessed a few software upgrade projects, such as Guidewire and SharePoint. Those projects usually took 6+ months for a 10+ staff team, functional testing is where the most effort (i.e. cost) is. I have done many framework upgrades (major versions) for my Ruby-on-Rails-powered web apps, which usually take 1 or 2 days, thanks to my Continuous Testing process.
Ruby on Rails 7.0 (web framework) was released on December 15th. I usually would upgrade my web apps to the latest Rails within the first week. This time was delayed because I was in the final stage of releasing my new book: ‘Practical Desktop App Test Automation with Appium’.
I started the upgrade on 25/12 (Xmas day) and finished today (27th, took two days). By the way, I did not work on this all the time, it is the Xmas break after all, and family time takes priority.
My upgrading strategy is simple: To pass the automated End-to-End test suite, which consists of 548 Selenium WebDriver tests. If all tests pass, the job is done, and deployed to production immediately. I have been doing this way for all my apps since 2012.
Learning test automation, if under proper guidance, it is easy. Maintaining a large test suite with application changes is hard. A major app upgrade is a big challenge for (software, test, DevOps) engineers. I usually complete the upgrade within one day, this time took a bit longer, 2 days, again, Xmas time.
I read the change log of Rails 7, and it seems will take some work. One welcome change in Rails 7 is that now we can be Node.js free. I really don’t like seeing the node_modules folder, apparently I am not alone.
Among all the changes that impact my app, the big one is the use of sessions. Rails 7 no longer supports persisting objects into session by default. I need to store them in the database and save the key to the session. Readers with programming experience will know this is not a trivial change.
It took me 7 regression builds to regain the green build.
After (and during the build), I rerun the failed tests (in TestWise IDE) against my local server, then work on the fixes.
The regression testing results on the BuildWise CT server:
2021–12–25 11:52, 41 test failures
2021–12–26 09:16, 42 test failures
2021–12–26 11:30, 17 test failures
2021–12–26 14:52, 71 test failures
I made a wrong change, which caused my regression failures! That’s OK, when this happens, I appreciate the automated regression test suite more.2021–12–26 15:51, 11 test failures
2021–12–26 17:05, 3 test failures
when the number reached a single digit, I felt much more relaxed2021–12–27 05:50, GREEN Build!
It felt good to see the green lava lamp (in my test lab) again. Then, deploy to production!
Stats
3836 test executions,~200,000 test steps. Each test step is a user operation, such as clicking a button or verifying a piece of text.
Tips on intensive automated regression testing
For an overview of automated regression testing, please check out this article. Here I will just share the tips on intensive regression testing, like this case.
1. Parallel Test Execution
Different from Unit Testing and API testing, you have to run automated tests in parallel. The reason is simple: time.
For example, by using 5 build agents, the overall test execution time is reduced from ~4 hours to <1 hour (see above screenshot).
Check out this article to see a picture of the super-impressive Facebook’s test lab.
2. Start to work on the fix while the build is in progress.
BuildWise CT server will run recently-failed tests first. So, I get early feedback while the build is still in progress. For example, below is a failed test execution I opened in BuildWise.
I click the “Copy error line to Clipboard”. Switch to TestWise, paste it into the “Go to File” dialog to navigate it quickly, and then run the test.
3. Developer and Tester Collaboration
Forget about defect tracking, you simply don’t have time for it if you want to complete the upgrade in days. Developers and Testers shall constantly keep a good eye on the build progress in the CT server.
Because I am the one person with all roles for WhenWise app, it is easier to convince myself to fix build errors ASAP. In a typical team, it will take some effort to train programmers unfamiliar with the fast-paced bug-fixing cycle.
The work is on both sides: programmers update the code; test engineers update the test scripts/test data. For easier communication, all team members shall use the same testing tool.
When the build is finished in the CT server, the team shall be ready to push a new build to the test servers. Then trigger another run in the CT server.
4. Don’t try to fix all in one go.
Instead, do it gradually. As a general rule, aim to reduce the failed test count first. This will help ease the stress. I suggest starting with seemingly easy-to-fix ones first. After that, choose ones that are related, so that they might be fixed with one code change.
Commonly, one bad check-in might cause more test failures. That’s OK. If you keep the changes small, it is easy to revert. “Slow and Steady wins the game”. From my experience, this is often the fastest way.
5. No Fear, with the safety net
Don’t be fear. Create a new Git backup branch, so that you can always go back. That’s what source control systems are for.
6. Be Productive
It is going to be a lot of work. Furthermore, there is time pressure (and a desire for a green build), and engineers must be familiar with the test tool to be productive. For example, prefer keyboard shortcuts over using a mouse when possible.
When I worked/consulted on an external project, people were often deeply impressed that I could implement a few real tests (for web apps I had never seen) on the first day. With good infrastructure, I set up the CT server to run these tests as well. Developing new automated tests is easy if you follow a good and proven formula. If a test automation engineer cannot write new automated tests efficiently (measured by minutes or hours), forget about this kind of intensive regression testing.
I cannot think of developing software any other way.
“We have been told many times that this is a very difficult discipline to put in place, but once it is working, no one would ever think of going back to the old way of doing things.”
- Implementing Lean Software Development: From Concept to Cash, by Mary Poppendieck and Tom Poppendieck