Understanding the Components of End-to-End Test Automation and My Practical Suggestions
My opinionated suggestion to choose a correct technology stack for your E2E test automation.
Table of Contents:
· 1. Scripting Language
· 2. Automation Frameworks
∘ 2.1 Web
∘ 2.2 Mobile
∘ 2.3 Desktop
∘ 2.4 API
∘ Synergy
· 3. Test Syntax Framework
· 4. Build Script
· 5. Test Data Creation
· 6. Test Scripting development and Debugging tool
· 7. Execution in a Continuous Testing Server
1. Scripting Language
My Choice: Ruby
Why Ruby is the Best Scripting Language for End-to-End Test Automation?Not recommended: a complied language such as Java and C#
Automated Test Scripts Shall be in the Syntax of a Scripting Language, Naturally!Avoid: JavaScript
Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
2. Automation Frameworks
An automation framework drives the application. As a general rule, use a free and open-source automation framework, preferably following an international standard, in web development, W3C, of course.
Don’t confuse tools with frameworks. Commercial tool vendors love to see that. For example, Quick Test Pro can be seen as a framework and a tool altogether. Avoid that.
“Choose a popular, proven, free and open-source automation framework; Then choose a dedicated tool (preferably, with free and commercial editions) for high productivity.” — Zhimin Zhan
2.1 Web
My Choice: Selenium WebDriver
Proof:
“Facebook releases twice a day
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucialto making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Not recommended: all others
Why Raw Selenium Syntax is better than Cypress and Playwright? series
Avoid: Creating a so-called custom framework on top of Selenium. Google’s Angular.js team tried (Protractor) and failed.
Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver
2.2 Mobile
My Choice: Appium + (XCUITestDriver or EspressoDriver)
Why? Based on WebDriver
Proof: Check out this great Facebook Presentation.
Not recommended: All others
2.3 Desktop
My Choice: Appium + WinAppDriver (for Windows platform)
There shouldn’t be any argument for alternatives after seeing the announcement from Microsoft, right? at least for Windows App testing.
2.4 API
My Choice: mature HTTP Clients, such as HTTPClient, HTTParty, RestClient, …, etc.
For API Testing, being simple and flexible is the key. After all, HTTP API is well-defined, quite straightforward and unchanged for decades. Don’t complicate it.
Avoid: GUI-based ones such as SoapUI and Postman; those are more suited for ad-hoc or manual API tests. Also, mixing with others, such as Cypress.
Synergy
You will get synergy using the same programming language for Web, Desktop, Mobile and API Testing. e.g. invoke an API to create a job number and check it on the Web to verify the job, all in the same test script file.
3. Test Syntax Framework
A test syntax framework provides the structure and assertions.
My Choice: RSpec
Why? Simple and Proven, more importantly, an RSpec test itself is a Ruby script, offering great flexibility.
OK ones: JUnit and Pytest
Avoid Gherkin-style so-called BDD frameworks, such as Cucumber and SpecFlow.
Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
“Cucumber is Dying”, What did we learn?
4. Build Script
My Choice: Rake
Why? Rake’s build file (Rakefile) is a Ruby script, offering great flexibility (using any libraries) and almost no learning curve.
OK ones: Apache Ant
Avoid: all others
5. Test Data Creation
My Choice: Faker (https://github.com/faker-ruby/faker)
Faker is also available in multiple languages; just search for “Faker <language of your choice>”.
6. Test Scripting development and Debugging tool
My Choice: TestWise IDE
Yes, it’s created by my father, I might be a bit subjective here. The fact: TestWise has gained international recognition.
Frankly, I didn’t realize the greatness of TestWise as I started learning test automation with it. When I started my intern work, my colleagues were shocked by my efficiency. But I took that for granted. The key reason: TestWise.
A maybe worthy upcoming choice: JetBrains Aqua
JetBrains, the №1 development tool vendor, released Aqua’s preview version, its first-ever testing IDE, in November 2022. I have high expectations for it; I use IntelliJ IDEA daily at work (as a software engineer) and love it.
OK: Visual Studio Code
Visual Studio Code, while a general coding IDE, can also be used to develop automated E2E tests.
7. Execution in a Continuous Testing Server
My Choice: A dedicated CT Server such as BuildWise and Facebook’s Sandcastle
Why? They contain dedicated CT features to address the challenges of executing Automated E2E (via UI) testing.
Avoid: CI servers such as Jenkins, Bamboo, TeamCity, …, etc
Why? They were designed for executing unit/integration tests.
Related reading: