‘Shift Left Testing’ Clarified
Shift Left Testing is good, but implement Automated E2E (UI) testing and Continuous Testing first.
This article is one of the “IT Terminology Clarified” series.
“Shift-Left Testing” has become a hot term in the software industry since 2019. For example, SmartBear even released a product named “TestLeft” (BTW, this product is a mistake, see the explanation below).
According to Wikipedia, Shift-left testing is an approach to software testing and system testing in which testing is performed earlier in the lifecycle (i.e. moved left on the project timeline). It is the first half of the maxim “Test early and often.”
If my understanding of Shift-Left testing is correct, I have been practising it since 2008. Therefore, I would like to share my understanding of Shift-Left testing here.
Table of Contents:
· The concept of Shift-left Testing is not new
· The benefits of Shift-left Testing?
· What kind of testing do we do in Shift-Left Testing?
∘ Shift-Left Testing must be E2E Functional Testing
∘ Shift-Left Testing must be Test Automation
· Who is conducting Shift-Left Testing?
· How left exactly is in ‘Shift-Left Testing’?
· How to succeed in Shift-left Testing?
∘ Prerequisite: Test Automation and Continuous Testing process
∘ Automated Test Script is easy to read
∘ Testing Tool is easy to use
∘ Relies on solid Continuous TestingThe concept of Shift-left Testing is not new
Below is a slide from a great presentation: “Continuous Integration at Facebook” (in 2015). As you can see, “Test Locally” (functional testing) has been moved to the immediate next step of the “Write Code” at Facebook. I have been developing software in a similar way for over 10 years.
Shift-Left Testing is really another name for “Test Early”, though the definition of Shift-Left Testing is vague just like many fancy new terms, such as:
CI/CD (deliver to where? production?)
“Release (assuming, to production) often”, but how often exactly? every day or every month? Does it need to pass whatever level of regression testing?
Facebook releases twice a day, which would totally change the whole development process. I don’t believe Facebook teams will waste lots of time on estimating user story points and worry about velocity charts. The reason I know is that all my software (TestWise, BuildWise, SiteWise, ClinicWise and WhenWise) is released on a daily basis too.
When I started working on my own software development (since 2008, before that, I coded for others), I didn’t have a clear distinction between writing code and tests. To me, it is the Yin-Yang of one activity. On my computer screen, a programming IDE (IntelliJ IDEA, Visual Studio, Xcode or TextMate) was on the left and TestWise (functional testing IDE) on the right. A user story was only considered done when I passed the automated E2E tests (newly created).
In my opinion, test automation should also be discussed with the capability of production release. Without that context, it will be talking only.
The benefits of Shift-left Testing?
You have probably seen a long list of benefits of Shift-left testing elsewhere. Here, I will just emphasize one: “The earlier a defect is found, the cheaper it is to fix it”. Actually, this is one of a few things I still remember from my software engineering course at Uni. And, of course, this rule is absolutely true in practice.
In my view, this is a good enough reason for projects to adopt Shift-left testing.
What kind of testing do we do in Shift-Left Testing?
The answer is automated E2E (UI) functional testing.
Shift-Left Testing must be E2E Functional Testing
The term “shift-left” indicates moving the testing activities that were previously done on the right (i.e. at the end of a development cycle). This means functional testing that was performed by testers. Unit testing is not included in Shift-left testing. According to Test Driven Development, unit testing has already been on the left, 😊. (for more information, check out my article, Unit Testing Clarified)
Some may ask: “How about API testing?”. My answer is “It depends. But mostly, it is not”. If your final testing phase includes API testing, then you may improve it by shifting it left. In my experience, API testing is commonly conducted solely by developers. Regardless, if your app has UI, focus on UI functional tests first.
Non-functional testing, such as performance and load testing, generally does not fit in Shift-left Testing. We often need to perform non-functional testing in a special and controlled environment. Though it is still possible to do shift-left performance/load testing in an innovative way which is covered in my upcoming new book “Practical Performance and Load Testing”. There is no point in shifting non-functional testing if functional testing was not done first.
For E2E tests here, what I mean is a user story level test via UI (in a browser for web apps).
Shift-Left Testing must be Test Automation
The definition of Shift-Left testing (or Test-Early) is vague on whether the tests are executed manually or automated test scripts. In my opinion, Shift-Left testing only makes sense if it is done via automation.
Some might argue “I can move manual testing left”. No, you cannot, at least you won’t be able to do it in a practical repeatable process. Don’t forget that, according to the definition, Shift-Left testing is the first half of “Test Early, Test Often”. How can you do “Test Often” (which includes the meaning of regression testing) with manual testing?
Here is an example. For an insurance app (in which I worked with a few clients): lodging one claim might take 5–10 minutes by an experienced manual tester/business analyst, without making a single mistake. Therefore, testing often is not possible with manual regression testing.
After all, test automation is a fundamental Agile practice, the enabler.
Who conducts Shift-Left Testing?
Programmers, manual/automated testers or business analysts.
The role of programmers in Shift-left Testing is obvious. As there is no heavyweight design stage in agile projects, programming activity is the left-most activity that involves a visible user story implementation. In other words, Shift-left testing moves left to get programmers involved.
Since I have been practising shift-left testing since 2008, in group or solo projects, I believe a programmer can do both development and testing If he/she has a testing mindset/skills (automation), I will write a separate article on how I practise Shift-left testing.
How left exactly is it in ‘Shift-Left Testing’?
You might have heard of “Acceptance Test Driven Development” (ATDD, which is rarely mentioned nowadays, replaced by BDD) before? In theory, testers can design or even write test scripts before a user story is implemented (based on mock-ups or meetings). However, in reality, it makes sense that real testing starts with a first-draft implementation that has been deployed on the dev/test server.
How to succeed in Shift-left Testing?
Prerequisite: Solid Test Automation and Continuous Testing process
Shift-left testing requires a high level of test automation. You simply cannot do shift-left testing well by executing tests manually based on a spreadsheet. Testing is about feedback, “Test early, Test Often” means much quicker and more often feedback. Manual testing is simply unable to meet the criteria. Therefore, there was a good reason that testing activities were put at the end in waterfall projects.
How will a tester demonstrate a defect to a programmer? Remember, we are in ‘Shift-left testing” mode. Programmers are supposed to react quickly based on early feedback. Commonly, a manual tester could not describe the defect correctly or replicate it reliably. When this happens, it is a waste of a programmer’s time. Therefore, test automation is a must.
Besides working on test automation for the user story, the team must have a solid continuous testing process to run an automated regression testing suite. As we know, the byproducts of a programmer’s code check-in are regression defects. I will explain this more later.
Some readers might wonder: “but we don’t have the skills in test automation, nor continuous testing”. The answer is: learn them. This is not about shift-left testing. In my opinion, every software engineer needs to master basic test automation, otherwise, he/she will not even qualify as an engineer. By the same token, if a team does not have a continuous process to run a comprehensive automated regression suite, the team is not qualified to use ‘Agile’ or ‘DevOps’.
Automated Test Script is easy to read
In Shift-left testing, there will be a lot of communication over test scripts between the programmers and other team members. It is important that the automated test scripts are easy to read for all. My recommendation: raw Selenium WebDriver + RSpec. (Check out AgileWay Test Automation Formula)
Common mistakes:
Using Gherkin (Cucumber, SpecFlow, …, etc)
Gherkin's syntax is easy to read. However, it comes at a huge cost: test maintenance. (Check out Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?)Test Scripts are in a compiled language such as Java and C#.
Test scripts need to be in a scripting language. (Checkout Automated Test Scripts Shall be in the Syntax of a Scripting Language, Naturally!)Test Scripts are in JavaScript
(Check out Why JavaScript Is Not a Suitable Language for Real Web Test Automation?)
Testing Tool is easy to use
With a hyped term in the software industry, there will be a flood of tools for it. There is no exception for Shift-Left testing, such as SmartBear’s TestLeft and Micro Focus’s UFT developer. However, these tools are wrong. First of all, I have never seen one single successful test automation attempt with this kind of commercial tool with proprietary syntax and offering wrong practices:
Record-and-Playback
The fact is that the test automation trend is moving towards free/open-source test frameworks. The popularity of Selenium WebDriver is proof.
My recommendation is TestWise. (Disclaimer: I created it. TestWise was a finalist of the Ruby Award in 2010 and was listed as the first testing tool in the “Trends in Agile Testing” presentation by Lisa Crispin, co-author of the Agile Testing book)
You may download and use fully functional TestWise in a free mode (not trial), to run, edit or debug raw Selenium WebDriver tests. Many manual testers and business analysts found it easy to use. (Check out “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”)
Of course, with a test automation framework in plain text, such as Selenium WebDriver, you have the freedom to choose any testing IDE or programmer’s editor (such as Visual Studio Code). However, please note that the choice shall be based on the preferences of the team members with the least technique skills.
Relying on solid Continuous Testing
If “Shift-Test Testing” is “Test Early”, then “Test Often” is “Continuous Testing”. By shifting testing left, programmers tend to make more code changes, inevitably leading to more regression errors. Blaming a programmer is not fair as the defects are due to the nature of software development. One code change might impact other parts of the software. But automated regression testing will provide a safety net.
Check out “How to Implement Real Automated Regression Testing?”.