A Story of 100X API Testing I Implemented in a Matter of Days
What a fully automated continuous regression testing can achieve.
A report of my past article on Medium in 2023
100X? Some readers might think, “Zhimin, you are bragging”. No, it is actually quite conserved for this case; you might agree with me after reading this story.
Over a decade ago, I worked (contracting) as a test automation engineer in a large government department. As was often the case, the testing manager did not understand Test Automation (even though he specifically borrowed me from another department after attending one of my presentations). The first task assigned to me was more like manual testing.
The testing is to verify APIs. Every tester was assigned a few API names (in a big spreadsheet, I remember two A3 pages in print). For each API, there was a pre-defined test plan that outlined many test scenarios.
The instruction was to use JMeter to perform the task. Some readers would be puzzled (like I was), “JMeter is for performance/load testing!” There was a reason: the API was protected, and needed to be invoked after signing with a set of keys (PKCS12 format) as different vendors. The technical architect came up with this solution, using JMeter to load a keystore to encrypt the requests.
A keystore can be a repository where private keys, certificates and symmetric keys can be stored.
The testing team consisted of about a dozen of manual testers. As I said earlier, I was counted as one as well. The main-frame system has been used for a few decades (at least), and the test cases for each API were documented well (but looking up test data is challenging).
This round of regression testing was planned for one month for 10+ testers, a total of ~3000 test executions.
I was comfortable with JMeter (see my JMeter Case Study) and XML. A typical test execution is as below:
Find a data record ID.
Construct a request XML based on the test case design.
Launch JMeter and load a specific keystore (PKS12 file).
Construct a request XML in JMeter and invoke the target API.
Verify the response (also XML) returned.
Exit JMeter, because you only can set keystore once (at least it was the case then)
It was really manual API testing, in a very inefficient way.
After doing a few, I realized I couldn’t continue this way for a month.
So, I started to work on the solutions, automated solutions, with the following two goals in mind:
Perform the testing in a 100% automated way using test scripts, i.e., no human intervention.
The closing and launching of JMeter ~100 times a day would drive me crazy.Run all tests (assigned to me) in a continuous fashion
The reason is simple: when the developers fix one defect, they might introduce another.
The latter was no problem, as I have already been using my BuildWise CT Server (not released publicly then) for my personal projects.
In terms of scripting API tests, I was comfortable too. The only challenge was to get around the security: signing the request XML with a PKS12 key store.
The test scripting language was supposed to be Java because PKS12 is known as a Java keystore. I started programming in 1997 and contracting as a senior Java developer for 10+ years. However, I decided to use Ruby, as it would be simpler for my colleagues to understand and use. Also, for myself, it’s delightful to use Ruby.
It took me about half a day to work out the solution, a 100% automated test script to use different KeyStores to invoke the API. I can’t recall the exact techniques I employed, but I remember:
There is a need to do a handshake first.
Convert PKS key stores to a different format that my Ruby scripts could use.
Then, set up a BuildWise CT server to run those tests.
By the way, because I had already created a few automation utilities to find test data, plus being proficient with XML and JMeter, I was still on par with my colleagues in terms of progress despite spending half a day on research.
At about noon on the second day, after refining (extracting a few reusable functions), I completed all my work that was planned for one month.
A Huge Success
I ran all my API tests in BuildWise (just a click of a button) and got the test report in under 1 minute. There were a few failures.
There was a process to raise defects in an online form, which was quite simple to use, just filling
Which API
Request XML
The response XML
Description: What's wrong?
I also automated most of the “filling the defect form” using Selenium WebDriver.
So, I raised a few defects, in a semi-automated way.
A few minutes later, the senior dev lead came to our section and asked, “Who is Zhimin?”. He did not expect such prompt testing feedback, and the defects were raised one by one so quickly. I showed him BuildWise and did a run in front of him. The dev lead was deeply impressed. He noted down the BuildWise server address (running on my machine on port 3618).
Of course, the test manager and the whole team quickly became aware of my work. I told the manager, “I completed the task”. The test manager told others to give some workload to me. Eventually, I completed most of the work (the whole two A3 pages were assigned to me) within a few days. There was synergy when using real automated scripts. (Remind you, that was 10+ testers’ work for one whole month.)
Please note, here, I do not belittle the value of this manual testing. In this case, the poor testers were assigned to regression testing manually. In fact, once our 7-minute automated API regression testing was implemented, some senior manual testers had time to perform high-quality manual testing, and found out a couple of subtle issues existed for 20 years.
Rather, I would highlight the importance of
The power of good scripting, especially using the best E2E scripting language: Ruby.
Fully automated regression testing (no human intervention)
Continuous Testing, execute the whole regression suite regularly
A huge success, technically, surely it was. The previously 1-month regression testing time (by 10+ testers) is reduced to just 7 minutes (that’s surely more than 100X, right?)
However, there are human factors at work, right? The story continues. Stay tuned.
Related reading:
My Universal, High-Efficient and Free Approach to API Testing
My Books:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps real
- API Testing Recipes in RubyBenefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
Without Solid Automated End-To-End Regression Testing, Exploratory Testing is Largely Compromised