Upgrade WhenWise to Rails 8. Following 3 Rounds of Full Automated E2E (UI) Regression Testing, Deploy to Production (All in ~3 hours)
Thanks to Comprehensive E2E Automation Regression Testing.
Four days ago, Rails 8 was released.
I didn't upgrade WhenWise immediately because I've been exploring the new M4 Mac Mini lately and used the opportunity to optimize my Continuous Testing lab.
At 7 AM on November 12, 2024, I started the upgrade to Rails 8 (from version 7). Experienced software engineers understand that a major part of such framework upgrades is End-to-end regression testing.
How did I accomplish the entire upgrade in just 3 hours? Thanks to end-to-end (UI) test automation and continuous testing practices I've applied to all my web applications since 2012.
1. Upgrade
Refer to the Rails 8 upgrade guide and carry out the following steps:
Update Gemfile (library), e.g. `gem "rails", "~> 8.0.0`
bundle update railsUpdate the app
rails app:updateThen follow the prompt to update various configuration files. Some customized settings (e.g. STMP, Fake Google/Facebook authentication) needed to be kept and added back to the new default Rails configurations.
Also, I bumped the WhenWise version to 2.0.0.
2. Quick Local Check
I launched a local server instance ('DEV') with the updated code and ran into a minor issue.
This can be resolved by changing one environment configuration entry (in all my 10+ environments).
config.action_controller.raise_on_missing_callback_actions = falseAfter restarting the DEV server, I could sign in and perform a few simple operations without problems.
It seemed going to be a smooth upgrade.
I then executed several key workflow automated tests in TestWise IDE.
All three of them passed.
At 07:23, I committed the changes locally and pushed them to the parent repository.
3. Regression Testing
I ran the deployment script (a simple Ruby script) to deploy the latest build to my 10 test server instances, all now operating on the new M4 Mac Mini.
For this regression testing, only 7 instances were utilized.
I triggered a run on the BuildWise Server, and the main work commenced (I am not just sitting and waiting)
4. Ready for Regression Testing
This time, I pushed the limits by using up to 7 build agents on the newly configured App Server. Experienced Test Automation Engineers understand:
Infrastructure changes can impact server performance, making UI tests more sensitive.
Conducting a major version upgrade (Rails 7 to 8).
Using three more agents than usual increased the load, especially since all 7 instances were on the same server machine.
Run 1: Start time 7:47 AM (took 38 minutes), with 5 test failures.
BuildWise’s auto-retry feature helped prevent multiple false alarms (18 in total). The initial run resulted in only 5 test failures, which was better than expected. During the CT run, I monitored the test executions across different build agents (I do not run tests in headless mode).
For every test failure (while the run is still undergoing on BuildWise), I reran the test again the DEV server on TestWise. Then, analyse and update test scripts (or even the app) accordingly. This is challenging work under significant pressure, which many senior test automation engineers find difficult to manage. (Tip: seek out a mentor to help guide your growth in this area 😉).
The application itself performed well, and the failure analysis (based on error traces and screenshots from BuildWise) indicated timing-related issues.
Run 2: Start time 8:34 AM (took 36 minutes), with 3 test failures.
The same process.
Run 3: Start time 9:23 AM (took 29 minutes), SUCCESS.
This is the first time, from memory, that the total regression testing duration is under 30 minutes, after the number of automated E2E tests has surpassed 500.
Great, it is time to push it to production.
Deploy Production Server
As you can see, installing Rails 8 gems…
The whole deployment took only ~30 seconds (because there is no database migration required for this update)
Check the Production Server
Yes, all done before 10 AM. WhenWise is now running on Rails 8.
(Note: A day later, there have been no issues reported by customers, just like hundreds of past deployments.)
My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps realBenefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.