BDD Clarified: BDD ≠ “Given-When-Then” (Gherkin)
BDD with RSpec, no Gherkin, is much better.
Background image credit: Futurama S2E4
Cucumber is the first Gherkin BDD framework, and it gained popularity gradually. Cucumber is written in Ruby, naturally, its clones were created, such as SpecFlow and JBehave, in other languages. However, many people think Cucumber (and its clones) is equivalent to BDD, and that is wrong!
RSpec is the first BDD framework
First of all, Cucumber (or Gherkin) is not the first BDD framework, developed in Ruby. In Ruby’s world, the RSpec BDD framework was released nearly two years earlier than Cucumber.
RSpec v1.0 was released on 2009–07–25.
Cucumber v1.0 was released on 2011–06–20.
Putting aside the argument on syntax differences (which I will explain later), without a doubt, RSpec is the first BDD framework. Please note, I am comparing apples to apples here, both frameworks were released on RubyGems.
RSpec is still very active, in fact, its downloads far exceed Cucumber.
Cucumber 7.1 download count: 823630
RSpec 3.10 download count: 42526424The reason that RSpec downloads are so much higher: RSpec can be used in the Unit, API, and functional testing.
Non-Ruby programmers were not aware of RSpec. When Cucumber clones were available on Java and C#, they mistakenly thought the syntax = BDD, which now you know, is wrong. This is like, many years ago, many people thought Internet Explorer was the first web browser and how a web browser should be, but the reality is: Netscape started web browsing.
What is BDD?
Behaviour-Driven Development. We can’t talk about BDD without mentioning Test-Driven Development (TDD). In fact, BDD was inspired by TDD to apply to functional testing. There was also a short-lived intermediate term: ATDD, Acceptance Test-Driven Development.
TDD -> ATDD -> BDD.
In TDD,
Programmers write automated unit tests,
and write the test first; let it fail
then implement the code to pass the test.
If passed, must run all unit tests immediately.
For more, please read my other article: Unit Testing Clarified.
BDD is actually quite different.
Software Engineers/ Test Automation Engineers write automated functional test scripts
Business Analysts/Customers may run (or even edit) automated functional tests
mostly don’t create the tests beforehand
preferably, but rarely achieved, run the whole functional tests daily as automated regression testing, typically in a Continuous Testing (not CI) server.
As you can see, The Whos, Whats, and Hows are all different. The ‘driven’ in BDD, in practice, has no meaning in the context of automated testing. People might argue that the cucumber features, that Business analysts wrote in Jira, drive the development, i.e. programmers coded based on that. But that’s a different context (requirement-driven), we could say that in software projects using Waterfall methodology three decades ago as well.
In TDD, the unit tests serve two purposes for code:
as the requirement (upfront)
as the immediate verification
The concepts in BDD are the same for user stories. However, in reality, many people talk about BDD without enforcing the verification part.
Why do so many people talk about BDD, without seeing automation?
Most IT people know TDD is about the execution of automated tests. But BDD in many software projects means:
a term frequently mentioned but not visible result.
‘Given-When-Then’ texts in Jira’s user stories
Of course, it is wrong. BDD ≠ User Stories are written in a certain syntax. If lacking the meaning of automation and verification, there is no BDD. Frankly, purely for requirement, I don’t see the Given-When-Thensyntax’s advantage over As …, X can syntax introduced in the classic “User Stories Applied: For Agile Software Development” book.
Why can’t we use Gherkin BDD for test automation? The answer is simple, few test engineers are capable to accomplish that.
Let me illustrate this with an example. Below is what a BA wrote in Jira for the login user story.
Feature: User Authentication
As a registered user
I can log in Scenario: Deny access due to invalid password
Given I am on the home page
When enter user name "agileway" and password "badpass"
And click "Sign in" button
Then I should see an log in error message
Scenario: Registered user can log in successfully
Given I am on the home page
When enter user name "agileway" and password "testwise"
And click "Sign in" button
Then I am logged inAfter being assigned, a programmer started to work on that. During the development (for this user story), he tested it (manually) a few times. After being happy about it, he drags the user story to the “To be tested” column in Jira.
A tester might have planned the test design based on the content on Jira. On most projects, they would just test it manually.
Why won’t they create a Cucumber test for it? In theory, they could. In reality, most engineers don’t have the capability. Please read the quotes from the creator of Cucumber.
“Using UI testing tools together with Cucumber? Please don’t — or at least do it very sparingly.” — Aslak Hellesøy, creator of Cucumber (link)
“If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. There are other tools that are designed to do this with far less abstraction and typing overhead than Cucumber.” — Aslak Hellesøy, creator of Cucumber (link)
Why is it hard? When used for test automation, the English text syntax does not come for free. Test engineers need to create a parser for it, but that’s still relatively easy. The main challenge is maintaining that. Over the last 10 years, every test automation attempts with Gherkin that I witnessed failed badly. That’s the price for not listening to the creator of Cucumber and DHH. For more, check out my other article: Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
BDD with RSpec makes sense
We have seen a sample cucumber feature. How does it look in RSpec?
describe "a model or business feature" do
it "behaviour 1" do
# ...
end
it "behaviour 2" do
# ...
end
endDon’t you think this is also a good (yet simple) way to describe business behaviors? So, BDD ≠ Gherkin. Different from Gherkin, developing/maintaining automated E2E tests in RSpec is a lot easier.
A full (and executable) test script for the above login user story in RSpec:
describe "User Login" do
include TestHelper
before(:all) do
@driver = Selenium::WebDriver.for(:chrome)
driver.get(site_url)
end
after(:all) do
driver.quit
end
before(:each) do
goto_page("/login")
end
it "[1] Can sign in OK" do
login_page = LoginPage.new(driver)
login_page.enter_username("agileway")
login_page.enter_password("testwise")
login_page.click_sign_in
try_for(3) { expect(driver.page_source).to include("Welcome") }
driver.find_element(:link_text, "Sign off").click
puts "[stdout] Signed out"
end
it "[1] User failed to sign in due to invalid password" do
login_page = LoginPage.new(driver)
login_page.enter_username("agileway")
login_page.enter_password("badpass")
login_page.click_sign_in
expect(driver.page_source).to include("Invalid password")
end
endIt is quite readable, isn’t it? Moreover, the test script is in Ruby language, which means many goodies:
syntax validation
syntax-highlighting
auto-reformatting (pretty-printing)
top flexibility (can’t beat programming)
good support in testing tools, such as snippets, auto-complete, script navigation, functional test refactorings, …, etc
dynamic test data generation, such as
send_keys(Faker::Internet.email)use of utility libraries, such as date manipulation
Date.today.next_weekand decoding BASE64 stringshighly maintainable if well designed, such as popular Page Object pattern (see Maintainable Automated Test Design)
easy to integrate, such as calling API tests
easy to run in CI servers such as Jenkins, and CT servers such as BuildWise
…
To wrap it up, ditch Gherkin and use RSpec to do real BDD. If you have to use Gherkin, I suggest doing both RSpec and Cucumber. By starting writing tests in RSpec first, you can maintain them easily before the app’s UI is established. When the time is right, write another set of Cucumber tests, reusing the helper and page classes created previously (in RSpec), which accounts for the majority of the work.