My Innovative Solution to Test Automation: ‘The Simpsons’ Data Reset Pattern
Data reset will greatly reduce a software team’s effort to test data management, and it is particularly useful for automated testing.
This is included in the “My Innovative Solution to Test Automation and Continuous Testing” series.
As one of the most iconic TV shows, ‘The Simpsons’ is the longest-running television series in the USA with its first episode aired in 1989. In this article, I would like to explore the mindset of the show’s genius creators that helped to make so many good episodes for over 30 years, and apply it to my software development and test automation.
At the beginning of a typical ‘The Simpsons’ episode:
Homer is working as a nuclear plant safety technician
Marge is a housewife
Bart is a naughty boy at Springfield Elementary School
Lisa is a smart girl in the same school as her brother
Maggie is a baby, with a dummy in her mouth most of the time
During the episode, Homer (and his family) might have all sorts of adventures, such as becoming rich, changing jobs, getting injured, becoming a movie star, relocating to a different city, …, etc.
Anyway, it does not matter what happened during the episode. At the end of the episode, the status will be reset back to its original state, just like what is at the beginning of the episode.
In software testing, test data is always a challenging (but is often neglected initially) problem in test automation. One day I came up with an idea to apply data reset in test automation while watching an episode of “The Simpsons”. Database refreshing is not a new idea, but invoking it for test automation (at the individual test level) was unheard of (at least for me). After many attempts, I managed to implement data reset successfully. Since 2008, I have been invoking database-reset before running automated tests (mostly, before each test case) for a number of web apps. It has become second nature to me in terms of automated testing.
To make the concept of database reset easy to remember, I called it “The Simpsons” pattern in Software Testing, particularly in test automation.
Table of Contents:
· Apply “The Simpsons” pattern in Software Testing
· Benefits of the “The Simpsons” pattern in Software Testing
· Team Members’ view on ‘The Simpsons’ Data Reset Pattern
∘ Business Analysts
∘ Architects
∘ Managers
· Main Challenge of Implementing ‘The Simpsons’ Data Reset Pattern
∘ PIF
∘ “No Data Reset, no real Agile”
· How to implement the ‘Simpsons Pattern’?Apply “The Simpsons” pattern in Software Testing
We can apply the same pattern to test data in software test automation. In this context, I refer to it as a “data reset pattern”. Let me illustrate it with an example.
Here is my automated test script for the Test Automation Scenario Interview Question that Most Candidates Failed (one of my most popular articles): User Can Change Password.
it "[4] User can change password, DB RESET" do
reset # invoke a helper function to reset data
sign_in("james@client.com", "test01")
click_avatar_menu("Edit Profile")
edit_user_page = EditUserPage.new(browser)
edit_user_page.enter_password("newpass")
edit_user_page.enter_confirm_password("newpass")
edit_user_page.click_save
relogin("james@client.com", "newpass")
try_for(2) {
expect(toast_text).to include("signed in successfully")
}
endThe main challenge for this test case is to leave no side effects after test execution, i.e., the subsequent logins will still work after executing this test (the password changed). With the “The Simpsons” pattern, the design of this test script (and others) is much simpler, and I don’t need to worry about the next login.
Other tests could use this reset too. For example, the test script below is to log in as the user james@client.com to perform other tasks. I just use the default password after calling reset .
it "[4] User can make a booking" do
reset
sign_in("james@client.com", "test01")
# ...
endThe reset function is defined in test_helper.rb.
def reset
visit("/reset")
endAs you can see, it simply visits a page /reset , which is a utility to reset the data on internal test servers, usually created by software engineers. (Of course, this is strictly prohibited in UAT and Production servers!)
You may view database reset in action by checking the article: Free Test Automation Practice Site with Database Reset, or try it out now at https://whenwise.agileway.net.
Benefits of the “The Simpsons” pattern in Software Testing
There are many benefits, but you probably will only appreciate database-reset after experiencing it. Here, I will just name a few:
1. A lot easier to write automated test scripts.
Starting with exactly the same status will make it much easier for script writers to create new episodes. Can you imagine the below:
In episode 1, Homer works at the nuclear station (original state)
In episode 2, Bart graduates from elementary school, now a middle school student
In episode 3, Marge found a permanent job
In episode 4, Maggie is sent to day-care
….
It would be very hard to continue, wouldn’t it?
Back to software testing, let’s take an insurance app as an example, with seeded data (the data that is used to refresh the database after reset),
4 users in the system: appliant01, applicant02, manager03, supervisor04.
App#001 is a fully filled application
App#002 is newly lodged
App#003 is an approved application
….
Putting the implementation of data reset aside (which I will cover shortly), don’t you think this will make software testing (manual/automated) a lot easier?
2. A huge productivity boot via parallelism
It is safe to assume that the producer (of The Simpsons) distributes ideas to different teams to work independently. The same practice can be directly applied to software testing. Different testers can work on test design/scripting for different test scenarios in parallel with no or little dependencies, based on the same data set.
3. A big saving on test execution time
I have often seen failed test automation attempts in which just a suite of ~50 automated end-to-end tests took 10+ hours to run. I will illustrate with the Insurance App example again.
Lodge app, ~ 5 minutes
Manager approves the app, ~5 mins on lodgement + 1 min on approving
Manager rejects the app, ~5 mins on lodgement + 30 sec on rejecting
…
If the app has implemented ‘The Simpsons’ pattern, the tests can be written in this way:
Lodge app, ~ 5 min
Manager approves the app, reset, go to ‘App#002" + 1 min on approving
Manager rejects the app, reset, go to “App#002”, + 30 sec on rejecting
…
That’s why my WhenWise Test Suite (548 raw Selenium tests) only takes ~4 hours when running all of them sequentially. With parallel execution on 7 Build Agents in the BuildWise CT server, the total time is reduced to 36 minutes.
Automated End-to-End Testing (raw Selenium) + Data Reset + TestWise + BuildWise (Continuous Testing) is my secret of developing/maintaining several highly acclaimed software products ( capable of daily production releases) at 100X+ productivity, in spare time.
Team Members’ view on ‘The Simpsons’ Data Reset Pattern
The benefits of data reset for automated testers are obvious. Apart from that, other team members found it useful too!
Business Analysts
Dean was a young Business Analyst who always tried to do the work better. It was his request on what data scenarios to be seeded. Dean printed out a list of application numbers (about a dozen) and stuck it on the edge of his monitor. Why? He used it too often.
I once asked him: “How do you value the importance of data reset on a scale of 1 to 10?”
Dean answered: “9, no, 10”.
Programmers and manual testers will find the same benefits too. Trust me, after they experience it, they would love it!
Architects
Once an architect came on board with knowledge of the previous expensive commercial workflow engine which was replaced by the in-house one designed by me (in plain Java). It was understandable that he had some grudges but our new workflow engine worked very well. Eventually, he changed his view.
Once I heard him talking to someone over the phone: “The reason we could do that so quickly is that a workflow can be reset to every specific state instantly in our workflow engine.”
Managers
Darren was a senior project manager with a solid programming background. Darren reviewed my first book: “Practical Web Test Automation”. He asked me to clarify further on the “The Simpsons’ Pattern, which I did.
A few years later, I asked Darren to review the 2nd edition of the book. Prior to that, we worked on one project together. Darren witnessed the power of Test Automation and Continuous Testing. He even wrote a formal letter to his boss, and the letter (he showed me) started with “I am seeing a software revolution, I know you wouldn’t believe it, but it is true. If you could come, …”. Not surprisingly, this executive director did not come.
Anyway, after the review (2nd time). He told me: “It is a funny feeling when I read the ‘Simpsons Pattern’ again. The first time, I had no idea. Now, it is so natural to me. I think, after implementing the ‘Simpsons Pattern’, 50% of software development work is completed”.
Main Challenge of Implementing ‘The Simpsons’ Data Reset Pattern
The short answer is Speed, i.e, how long does it take to reset the data back? The correct answer is a few seconds at maximum. To illustrate that, I will share two real stories: one failure and one success.
PIF, a failed attempt to data reset
I worked as a senior .NET developer on a large government project many years ago. The project was badly managed and the wrong choices were made on the technology stack. How typical!
On the daily basis, everyone struggled with the testing which was believed the major challenge of the project. We (developers, testers, and BAs) took a long time to find/create a data scenario. In a team meeting, I suggested the concept of ‘The Simpsons’ Data Reset Pattern. Some liked the idea, but most remained silent. As usual, I did not expect anything to happen.
The soft launch went badly, and the managing consulting company (a world-renowned one) was embarrassed. Somehow, one action item was to implement data reset under a project called: Project Integration Framework, in short, PIF. The job was assigned to a senior engineer J who sat in my neighbouring cubicle.
J worked on PIF for about 6 months, and I found he was getting more and more agitated, swearing more and more. His approach was backing up and restoring MS SQL server databases. Of course, it didn’t work, as the database was huge.
His work was rejected, as PIF’s best timing was about 2 hours or more. Later, PIF became a swear word in our teams, such as “Don’t PIF me”.
“No Data Reset, no real Agile”
In 2008, I joined a government project referred by a former colleague (Project Manager). The PM shared the same ambition with me to implement a proper Agile project: every user story is covered by automated end-to-end testing via UI.
However, the first obstacle was database reset. The architecture has been set. Microsoft SQL Server (MSSQL) was the database, which was fairly common at that time. I implemented a database reset utility for an equivalent database in MySQL, ~10 seconds. However, I couldn’t achieve that with the existing MSSQL database schema.
I expressed that we need help from MSSQL experts in the department. The PM asked: “Why?”
My answer: “Every automated End-to-End test will start with reset , we need that operation to be quick and reliable, in a matter of seconds. I was unable to achieve this with MSSQL. Without that, developing/maintaining automated tests will be a lot harder. Therefore, no real Agile”.
The PM managed to get some help. The best timing came about 10 minutes, unsatisfactorily.
One day, I came up with a solution.
Development/Test: MySQL Database
Staging/Production: Microsoft SQL Server database
I introduced Ruby on Rails’ database migration schema into our Java project (via JRuby). Though the database types were different between Dev and Production, they were based on the same database schema (in Ruby).
Long stories short, the hybrid approach worked! This project was very successful: the team ran a suite of 200+ Automated End-to-End Tests (in Watir) at least twice a day, which enabled daily updates to production.
How to implement the ‘Simpsons Pattern’?
Now comes the hard question, how could you implement database reset as testers? No, it has to be done by developers. If your team is doing Agile (or claiming so), go ask them, and it will be of great help for them as well.
The concept of the Simpsons Pattern is simple and generic, and a good software engineer shall be able to come up with a solution quickly. The main challenges are
Speed (explained earlier): fast, very fast!
Below is WhenWise’s quick data reset timing.
Ease of change
The seeded data is not static. With application changes, there will be inevitable changes to the seeded data. In other words, maintaining seeded data is an ongoing process. Therefore, it cannot be a simple database-backup or a large number of SQL statements.Absolute Safety
can’t reset the production database, obviously. There are many ways to safeguard that.
- In code: the reset controller check for environment variables
- In code: abort the reset if the total number of records above the
- In packaging: Remove reset classes for a production build
- In Database control: a database user that only works for testing databases.
Generally speaking, with the above protection scheme, it shall be quite safe. I never had a bad incident with using database-reset.
If you want my ready-to-use solution, there will be a cost. Not money directly, it is more to prove that you have the determination and the mindset for it. I will provide an example of the wonderful Ruby language. Yes, I have done this in Java and could implement it in C#, Python, or JS. However, after over 25 years of work in software, I value my remaining IT work highly and I prefer using Ruby whenever it is possible.
I will offer an info pack including
Guide (like a book chapter)
A sample Ruby on Rails app with ‘The Simpsons’ Data Reset Pattern
free (upon request) for customers who purchased any two of my products and services:
One-day Training (from $600 p.p)
- “Web Test Automation with Selenium WebDriver”
- “Practical Continuous Testing”
- “Practical Continuous Performance/Load Testing” (upcoming)
You can access the above material free (my articles) or nearly free (my books)One-on-one Coaching $1650/day
free complimentary coaching is available for TestWise Customers: limited to the test cases on the sandbox WhenWise site.TestWise IDE $30/month
free (forever) mode: execute up to 15 tests then relaunch.BuildWise Agent $30/month
free (forever) mode: run the agent for up to 40 mins (like Zoom free mode) then relaunch.
If some readers think: “You are trying to sell your test automation products/services”, please read the above (italic text) again, I have offered completely free options (for you to make commercial gains from your work).
I set the condition for two reasons.
AgileWay, my company, also has several commercial web apps, such as WhenWise and TestWisely. The ‘data-reset’ utility was considered our kind of ‘trade secret’, a big competitive advantage over competitors. I now make it free for people who would value it like us.
The working (data reset) example app is in Ruby. If someone wants to master this data reset skillset, they are expected to have some degree of Ruby knowledge or be open to learning Ruby (then re-implement it in C#/Java/Python/JavaScript).
Related reading:
My eBooks: