Successful Windows Desktop App Test Automation with Appium

The Continuous Testing Process of TestWise, a Desktop App

Over the past 15 years, I primally worked on web test automation. At the software testing conferences, a common question from the audience was “How do you develop automated tests for desktop apps?”. My answer: “I haven’t yet seen one successful and sustainable solution for desktop apps”. From today, I can say: “I found a solution” (the solution is Appium with WinAppDriver + RSpec, sample tests are provided below).

I have done some research on automated testing Desktop Apps using AutoIT3 and others since 2008, I even created a free, open-source framework RFormSpec. I had some degrees of success with RFormSpec on two projects, but I knew that the underlying framework wouldn’t scale.

Last year, I saw an announcement from Microsoft: “We recommend using Selenium for testing web apps and Appium with WinAppDriver for testing desktop and UWP apps.” So I started:

• learning and researching Appium and WinAppDriver

• applying the same design techniques in my book “Practical Web Test Automation”, such as reusable help functions and the page object pattern.

• adding Appium support in TestWise IDE

• developing automated GUI tests in TestWise for TestWise

As of 2021–03–26, the regression suite of TestWise IDE (a Desktop App) has 276 automated GUI (Appium) tests, running quite reliably in our CT server: BuildWise.

276 end-to-end tests running in CT on a daily basis, this is Level 3 of AgileWay Continuous Testing Grading. Personally, I think 200 is a magic number. This is purely based on my experiences, nothing scientific about it. When a test suite reaches 200 automated end-to-end first time, overcoming the challenges of maintenance helped me to reach a deeper understanding of test automation. It was not easy. I transformed my understanding into creating TestWise IDE and international award-winning BuildWise, to help testers to reach that goal much easier.

In test automation, the number of tests matters, as it can verify:

• the efficiency of developing new tests,

• more importantly, the efficiency of maintaining test scripts,

• the quality of test design (including project layout and script convention), and

• the overall reliability of test frameworks, toolsets, and the CI server.

To provide a perspective, over the last 15 years, I worked/consulted many software projects, despite claims (Agile/DevOps) of some projects, before I joined, no single project implemented 50 reliable automated UI tests that ran reliably.

Furthermore, for most medium-size software projects, 200 user story level UI tests, if the team really trusts, provide a quite comprehensive safety net (via regression testing), which enables the team to release frequently, i.e. Real Agile, Real DevOps.

What does this continuous testing mean to software development? I pushed out 30 quality TestWise releases within 71 days.

Most recent build:

The build time is 21 minutes with 5 build agents. If running all tests on one machine sequentially, it would take 77 minutes.

Build History:

It detected many regression errors. For people who have worked on a real Agile project, the importance of regression UI testing is hard to describe in words. Simply speaking, it is a must! (otherwise, you are doing fake Agile).

Total Test Executions (50,000+):

Sample Test (Appium + RSpec)

load File.dirname(__FILE__) + "/../test_helper.rb"
require "fileutils"
describe "Refactor - Extract Function" do
  include TestHelper
  before(:all) do
    close_all_testwise
    tmp_refactor_dir = tmp_dir("refactor-proj")
    test_proj_file = File.join(tmp_refactor_dir, "refactor.tpr")
    caps = testwise_caps(:appArguments => test_proj_file)
    @driver = $driver = Appium::Driver.new(caps, true).start_driver
    @main_window = MainWindow.new(@driver, "Selenium WebDriver RSpec Refactor - " + testwise_version)
  end

  before(:each) do
    sleep 0.15
    @main_window.send_keys(:control, "t")
    goto_file_dialog = GotoFileDialog.new(driver)
    goto_file_dialog.enter_name("01_login_spec.rb:29")
    goto_file_dialog.press_enter
  end

  after(:all) do
    driver.quit unless debugging?
  end

  it "Extract Function current line no,  no parameters - Ruby" do
    main_window.invoke_refactor_extract_function
    extract_function_dialog = ExtractFunctionDialog.new(driver)
    extract_function_dialog.click_cancel
    main_window.invoke_refactor_extract_function
    extract_function_dialog = ExtractFunctionDialog.new(driver)
    extract_function_dialog.enter_function_name("print_one_line")
    extract_function_dialog.click_ok
    sleep 1
    editor_text = get_current_editor_text()
    text_lines = editor_text.split("\n")
    expect(text_lines[23].strip).to eq("def print_one_line")
    expect(text_lines[24].strip).to eq("puts(\"login OK test\")")
    expect(text_lines[25].strip).to eq("end")
    expect(text_lines[32].strip).to eq("print_one_line")
  end
end

with Page Objects

require File.join(File.dirname(__FILE__), "abstract_window.rb")
class ExtractFunctionDialog < AbstractWindow
  def initialize(driver)
    super(driver, "Extract Function")
    0.5
  end
  def enter_function_name(func_name)
    elem = @win_elem.find_element(:xpath, "//Edit[@Name=\"e.g. enter_card_holder_name\"]")
    elem.click
    elem.send_keys(func_name)
  end
  def click_ok
    @win_elem.find_element(:name, "OK").click
  end
  def click_cancel
    elem = @win_elem.find_element(:name, "Cancel")
    elem.click
  end
 
  # ...
end

Testing Tool: TestWise

Continuous Testing Server: BuildWise

with parallel testing via 5 Build Agents.

This article was initially published on LinkedIn, 2020–06–04

Update: My eBook “Practical Desktop App Test Automation with Appium” is available from 2021–12–20.