My Test Automation Journey
Java/C# Programming ⇒ Test Automation + Continuous Testing; Now I do coding, testing, and CT daily for my own apps and coach test automation and CT for clients.
This article is a reflection of my 15-year journey in test automation since 2005. Before that, I had worked mostly as a Java Programmer (contractor) for 8 years.
(2005–2006) JWebUnit (HTMLUnit) using Intellij IDEA
(2006–2011) Watir with RSpec
(2011–2012) RWebSpec on top of Watir and Selenium WebDriver
(2011–2013) Load Testing with LoadWise
In 2010, I switched to daywork from programming to test automation and Continuous Testing. My love for programming was not changed, just do it differently: coding for my own products in my spare time. Thanks to my CT process, my productivity in software development is an order of magnitude better. Please read “Reflections of Software I Created over the Last 14 Years in My Spare Time”.
(2012-present) raw Selenium WebDriver with RSpec
(2012-present) API/WebServices Testing with Ruby
(2019-present) Appium with WinAppDriver for testing Win desktop apps
(2020-present) Performance/Load testing using browser-based Selenium test to run in BuildWise CT server
1. JWebUnit on top of HTMLUnit (2005–2006)
At that time, though most IT people had heard of automated functional testing, it was considered a luxury as the testing tools such as Mercury Interactive’s QTP (yes, later acquired by HP) were super expensive.
In 2005, I was extremely fortunate to have a chance to pair program with a world-class programmer (Martin Fowler often mentioned him) for 6 weeks. He showed me TDD (which I was aware of) and wrote automated functional tests in Java, which was a new concept to me. I liked it instantly.
The framework we used was JWebUnit , a syntax layer on top of HtmlUnit.
Realization:
Automated functional testing greatly enhances programmers’ productivity
Automated functional testing increases job satisfaction and is fun
Automated functional testing helps software design skills because I need to think more from different perspectives
Sample test (JWebUnit):
@Test
public void testLogin() {
beginAt("/home");
clickLink("login");
assertTitleEquals("Login");
setTextField("username", "test");
setTextField("password", "test123");
submit();
assertTitleEquals("Welcome, test!");
}2. Watir (2006–2011)
Once I ran a JWebUnit test in front of a manual tester. I pointed to the green tick in IntelliJ IDEA and said: “See, it passed”. This poor tester was confused, “How?”. I scrolled down and showed him the test output. He was half-convinced, “I want to see the test execution”. Hence, I made one of the biggest realizations of my IT career: “Seeing is believing applies to automated test execution”.
Fortunately, I found Watir and did exactly what I needed to do. Furthermore, after forcing myself to learn Ruby, I fell in love with this beautiful language quickly.
After selecting Watir as the automation framework, I naturally looked for a JUnit equivalent in Ruby as a veteran Java programmer, Test::Unit . It worked. Shortly after, I found RSpec, a BDD framework with more human-readable syntax, while still 100% Ruby (this is very different from Gherkin frameworks). I liked RSpec. In fact, I have been using RSpec mostly ever since.
Realization:
Automated test execution needs to be visible
Automated test scripts shall be in a scripting language
Need to think outside of the programmer’s perspective
Java IDE such as IntelliJ IDEA is too complex for non-programmers to use
Ruby is a great scripting language for writing automated tests.
Need to run all tests in CI server with auto-retry features (will be covered in ‘My Continuous Testing Journey’)
Sample test (Watir):
it "Login OK" do
browser.goto "http://travel.agileway.net"
browser.text_field(name: "username").set(user_name)
browser.text_field(name: "password").set(password)
browser.button(value: "Sign in").click
try_for(3) { expect(browser.text).to include("Welcome agileway") }
endTool:
I created iTest2, a testing IDE that supports developing/debugging Watir tests. One of its unique features is “Functional Test Refactoring”.
Other testing tools (tried/reviewed):
IBM Rational Functional Tester, QTP (neither is good)
Selenium v1 (good support with Firefox, but with limitations; JavaScript is not a good language for writing automated tests)
Publications:
“Refactoring Automated Functional Test Scripts with iTest2” Article on InfoQ (2009–07–22). This was my first published article, which greatly boosted my confidence in writing.
Practical Web Test Automation (2009). My first book, and my favourite one, too. I spent countless hours on it with many revisions. To me, writing English was much harder than coding and writing test scripts.
Acknowledgment:
iTest2 was demonstrated at AA-FTT Meeting at Agile 2009, with good feedback.
iTest2 was listed as the first testing tool in the presentation “Trends in Agile Testing” (2009) by Lisa Crispin, co-author of the Agile Testing book.
iTest2 was listed in Ward Cunningham (co-author of Agile Manifesto)’s Functional Testing Tools list.
Speaker at software testing conferences:
ANZTB 2010, Melbourne
Tianjin Software Testing Conference (TIST) 2010, China
Tianjin Software Testing Conference (TIST) 2011, China
Participated:
CITON (Continuous Integration and Testing Conference) 2009, Brisbane
3. RWebSpec on Watir and Selenium WebDriver (2011–2012)
Selenium WebDriver, a.k.a. Selenium 2, was released in 2011. This was big news for me. While I have been using Watir to test in IE6 fine then, I would like to use Mozilla Firefox which was my preferred browser for its speed. With Selenium 2, finally, I could run automated tests in Firefox.
Selenium 2 comes with Firefox support, and there was no separate driver required then. Selenium with IEDriver was unreliable though. Also, the idea of cross-browser testing became popular. I created RWebSpec which will use Watir for IE and Selenium 2 for Firefox, in the same test script.
Sample test (RWebSpec):
before(:all) do
open_browser :base_url => "http://testwisely.com/demo",
:browser => :firefox
endit "Login" do
page_title.should == "AgileTravel"
enter_text("username", "agileway")
enter_text("password", "testwise")
click_button("Sign in")
expect(page_text).to include("Welcome agileway")
endOther testing tools tried/reviewed:
Selenium WebDriver in Java, using Intellij IDEA
Publications:
Watir Recipes (2012)
4. Load Testing with LoadWise (2011–2013)
I have used JMeter for Load Testing before. It worked but was quite complicated. Therefore, I created my own performance/load testing tool: LoadWise. It did attract some commercial interest.
However, LoadWise, using the traditional HTTP-request simulations approach, does not support AJAX and more web apps were adopting AJAX. I deprecated LoadWise in 2013.
5. Selenium WebDriver (2012–Present)
With Selenium WebDriver becoming more reliable with IE (and Chrome dominating the browser market), I deprecated RWebSpec to use raw Selenium WebDriver + RSpec. I was quite pleased after converting all existing RWebSpec tests to raw Selenium. The scripts are actually more intuitive.
Over the last 8 years, I used raw Selenium + RSpec to test my own apps and help others.
Sample test (Selenium WebDriver + RSpec):
it "[1] Can sign in OK" do
driver.get("https://travel.agileway.net")
driver.find_element(:id, "username").send_keys("agileway")
driver.find_element(:id, "password").send_keys("testwise")
driver.find_element(:name, "commit").click
try_for(3) { expect(driver.page_source).to include("Welcome")}
driver.find_element(:link_text, "Sign off").click
endTool:
Renamed iTest2 to TestWise, and added support for Selenium WebDriver
Realization
Test syntax framework shall be in 100% scripting language like RSpec, while Cucumber is not.
Presenting at international software testing conferences, in my opinion, is not effective in terms of influence. I met many attendants who liked my presentation, but few of them dared to try test automation at work (lack of confidence). So I decided to spend more time on writing books that come with good exercises, creating video demos, etc. rather than speaking at conferences.
I worked on many projects as a contractor/consultant. I didn’t care about job titles, just wanted to work on different projects for testing challenges. After a few years, I could answer and solve other people’s testing problems quickly, usually in under a minute.
While test automation was being adopted by more projects, every single project I visited failed to adopt the automation properly. Some large companies failed a few attempts.
With my increased efficiency in test automation, my productivity as a programmer also grew significantly. Since 2012, I have created ClinicWise, SiteWise, TestWise v4, SupportWise, BuildWise, WhenWise and TestWisely apps, in my spare time.
Other testing tools (tried/reviewed)
UFT, Coded UI Test, Ranorex, Sahi Pro, Cypress. (none of them was good)
Speaker at software testing conferences
StarWest 2013, Los Angeles, USA
Publications:
Selenium WebDriver Recipes in C# (2015)
Practical Continuous Testing — make Agile/DevOps real (2021)
6. API/WebServices Testing with Ruby
While I focused mostly on web test automation, quite often, I was required to perform API/WebServices testing. As a veteran programmer, writing API test scripts is quite easy for me. However, after I worked on web test automation with many manual testers, I learned to look beyond the correct API tests: easy to understand (for non-programmers) and easy to maintain.
A common mistake with API Testing is to use tools such as SoapUI/Postman. Every project I knew of that uses these kinds of tools failed for the reasons below:
Not flexible (fact: programming provides the ultimate flexibility)
Hard to maintain
Inefficient. I know some might find it hard to believe that using a good scripting language such as Ruby can be far more efficient than GUI tools.
Difficult (or impossible) to run the tests in a CT server. Many projects did not even realize how to manage executions of a large suite. Therefore, tests went out of date quickly. As time went on, it was getting harder and harder to maintain. Eventually, there was only one outcome: the tools were dumped.
My approach with API (Soap/Rest/Microservices/GraphQL) has always been the same (for over 12 years): raw Ruby with RSpec and execute them in BuildWise.
At one project, I completed an assigned 2-month’s API testing (SoapUI) workload under one day (including initial start time), using raw Ruby.
With quick training, most manual testers can work on the scripts following my examples. For more, please read my book (listed below).
Other testing tools (tried/reviewed)
SoapUI, ReadyAPI, Postman (none of them was good)
Publications:
API Testing Recipes in Ruby (2016)
7. Appium with WinAppDriver for testing win apps (2019-present)
I created a small framework RFormSpec on top of AutoIT3 for automating windows native apps. AutoIT3 has some limitations, it worked well for some scenarios. I developed a dozen of RFormSpec tests for TestWise, my desktop app. However, it wasn’t easy.
In 2019, I saw a recommendation from Microsoft: “Selenium for web apps, Appium for desktop apps”.
After quick work on proof of concept, I liked it. Therefore, I switched the automated tests of TestWise (my desktop app) to Appium + Desktop. To see a test execution of Appium+WinAppDriver, check out this article “Demo of Automated Testing for Windows Desktop App: TestWise tests TestWise”.
As of 2021–03–26, the regression suite of TestWise has 276 automated tests running in the BuildWise CT server. This reaches Level 3 of AgileWay Continuous Testing Grading.
This CT process enables me to release TestWise daily if required.
Sample test (Appium + RSpec):
it "Plus two numbers in Calculator" do
desired_caps = {
caps: {
platformName: 'Windows',
deviceName: ENV["DEVICE_NAME"] || 'MacBook',
app: "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"
}
}
driver = Appium::Driver.new(desired_caps, true).start_driver
driver.find_element(:name, "One").click
driver.find_element(:name, "Plus").click
driver.find_element(:name, "Three").click
driver.find_element(:name, "Equals").click
result = driver.find_element(:accessibility_id, "CalculatorResults").text
expect(driver).to eq("Display is 4")
driver.quit
end2020–03–25, TestWise 6 is released. TestWise 6 is a total rewrite (the work started in 2018–07) using C++ language, and runs on Windows, macOS and Linux.
If you are interested, see TestWise in action in this article: “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”.
8. Continuous Performance/Load testing using browser-based Selenium testa to run in BuildWise CT server (2020-present)
Load testing is moving from Protocol-Simulation to Browser-based as modern web apps are dynamic and use AJAX. Since last year, I have combined my past experience of developing LoadWise with Web Test Automation and Continuous Testing and come up with a new way to conduct load testing.
The engineer develops a load test script as a functional UI test first, in raw Selenium WebDriver +RSpec. The engineer can run the testing tool such as TestWise, and view (or from a load testing perspective, preview) test execution to ensure correctness.
Add benchmark (timings) of operations
Make the test script load-testing aware of virtual users
Run a set of load test scripts in the BuildWise CT server which will distribute tests to run them in multiple build agents in parallel.
The test reports (and history) are viewable on BuildWise’s web interface.
Below is a report of the ‘User Login’ load test with 5 virtual users on BuildWise.
Currently, I am working on a new book: “Practical Performance and Load Testing”. I usually make the book available (on Leanpub) after the whole book is completed. If some readers want to access it now, please contact me.