Practical Advice To Starting Continuous Testing
Practical Tips for beginners to get started with Continuous Testing.
Continuous Testing, different from Continuous Integration, is a process of executing Automated E2E (preferably UI) tests frequently as regression testing.
Goal: Run a large number of tests on a daily basis
Below is a screenshot of one BuildWise agent participating in executing a suite of 577 End-to-End (UI) tests (Selenium). Real Continuous Testing!
It is a nice goal. However, even for experienced software engineers, it is important to take small steps when setting up Continuous Testing.
Tip 1. Get your tests running fine from command line first.
A Continuous Testing server, e.g. BuildWise, is designed to execute automated E2E tests. Of course, A CT Server has specific features, such as Dynamic Ordering, Parallel Execution, …, etc in BuildWise. But for new projects, put that aside for now — the first priority is getting the tests executed, reliably.
As all test automation engineers know, there are many things that can go wrong with test execution setup. With my recommended simple Selenium + RSpec tech stack, there are only a few things to set up, namely:
Ruby installation and its version
Selenium WebDriver gem and its version
RSpec Gem
Chrome version
ChromeDriver (which isn’t even required separately for newer Chrome versions)
Test project structure (missing certain files)
If any of the above is not correct (or not matching), the tests in your CT server might fail. When using an unfamiliar CT tool, it is easy to give up and blame the tool unfairly. What a pity!
The good news is that you can verify pretty much all of the above by simply executing one test script from the command line.
Tip 2. Don’t Start With Running your own Tests; Try out the sample projects first (e.g. AgileTravel)
A common mistake that new-to-CT software engineers make is that they are too eager to see their tests running on a CT server.
When something goes wrong, which is quite common in CT as there are many variables, plus frustration combined with eagerness, many people give up.
The best way is to start with one of two sample projects, e.g. AgileTravel, which is publicly available.
This has two benefits:
Boost confidence by confirming the test environment is set up.
Understanding CT and the tool better by freely playing with configuration/s.
Furthermore, Continuous Testing knowledge is mostly generic, in other words, many things you learned from playing with sample projects can be directly applied to your own test scripts, with much less stress.
Tip 3. Don’t Attempt Parallel Execution until it is necessary.
Test architects tend to make big plans (took a lot of time), with this mindset, CT rarely works. Why? CT is super practical, if it is not set up on Day 1 (again, read above, you can start small), the chances are you will fail.
The purpose of Parallel Execution is to speed up execution. However, if your test suite has only a handful of tests, don’t worry about parallel execution yet, just get them running quickly in a CT server first. In fact, when the number of tests is small, you might find that parallel execution may actually be slower.
Tip 4. Try executing a single test with configured set up
Besides executing your tests reliably from the command line, executing in a CT tool (e.g. BuildWise Agent) requires some configuration (such as PATH, target server environment.
Before triggering a run of the whole suite on a CT Server, do a test run (as above) for a single test (on Agents), the aim of this is to verify the configuration is all set up.
Related reading:
