Behat in stages

From time to time, I have this bright idea of learning Behat, the acceptance testing framework. In 2016, I actually did that, and then promptly forgot it all as I never had to use it again. This time, I have done it again and writing down a summary of what I have learned, which hopefully will help me remember it better.

My approach to relearning Behat this time had two stages: first I read about Cucumber and Gherkin from the excellent The Cucumber Book and then dived into the documentation of the Behat project. While the Behat documentation is good at explaining the syntax and semantics of writing test code, The Cucumber Book touches subjects that goes far beyond that. For example, the When Cucumbers Go Bad chapter explains how tests fall into decay or the importance of using domain specific terminology in your test description. I thoroughly recommend this book to serious users of Behat or similar other acceptance testing tools.

Everything described in this article applies to Behat version 3.4

Cucumber and Gherkin

Any tutorial on Behat will be incomplete without mentioning Cucumber. Cucumber is the original BDD framework. It is used to write test code in Ruby. Behat is the PHP equivalent. But they all have one thing is common - Gherkin. Gherkin is a very loose syntax for describing your tests in plain English. Here is an example of Gherkin describing the behaviour of a GPS navigator.

Feature: Take me home
  I am in the middle of nowhere.
  Actually, I am lost :(

  Background:
    Given I have no idea where I am

  Scenario: Happy ending
    Given I know my home address
    When I enter my postcode
    Then I am given route direction

  Scenario: Extreme case
    Given I know my home address
    But I am not paying much attention
    When I enter an invalid postcode
    Then my mistake is pointed out

Describing tests in the Gherkin format is a semi-art form. You cannot be too specific and you cannot be over general. I will use two examples to illustrate this. First, a test description that is too specific:

Feature: Find out the temperature
  Cannot tell you how important this is to me :)

  Background:
    Given I am setting off for work soon

  Scenario: I want to know the current outside temperature.
    Given bbc.co.uk carries up-to-date weather data
    When I open bbc.co.uk/weather
    And locate the search box whose DOM Id is "search"
    And fill in the search box with "London"
    And click the submit button
    Then the resulting page will show the temperature in a div
    And that div will have an id of "current-temperature"

If you are not a web developer, you may struggle with terms like DOM, Id, etc. Now let's do the opposite and present an example that is over general:

Feature: Find out the temperature

  Scenario:
    Given I am leaning on a couch
    When I want to know the temperature
    Then just tell me the temperature

While this may serve well for the Amazon Alexa or Apple's Siri, it certainly is over general for the rest. Here is my less ambitious take at finding a middle ground:

Feature: Find out the temperature

  Background:
    Given I am in London

  Scenario:
    Given I am at bbc.co.uk/weather
    When I search for "London"
    Then I am presented with the current temperature of London.

Learn more about Gherkin from the excerpts of The Cucumber Book

The whole idea of Behat is to connect test code written in PHP with the plain English tests described in Gherkin. In the rest of this write up, I try to explain how we can write those PHP test code. We progress from basic to advance in several stages. All code is available in Github.

Stage zero: Concepts

Behat is a PHP implementation of Cucumber, so not surprisingly, most concepts come from Cucumber. Key concepts include:

Test profile: This combines all our test suites under one roof. But multiple profiles are allowed as well.
Test suite: Gathers tests on a certain topic. For example, all tests for the authentication operation of a website can live under a single suite.
Feature: Describes a specific activity. For example, website login can be a feature, password reset can be another feature, and so on.
Scenario: Describes a specific use case. For example, a login failure where the username is non-existent.
Step: Describes a small part of the whole operation we execute to test a scenario. For example, stating that we need to load the login page is one of the steps we take to test a login failure. Each Scenario comprises several steps in most cases.
Step definition: The actual PHP code that executes a step. For example, the step definition for loading the login page will be a PHP class method that will fetch the login form.

The following hierarchical diagram shows the relationship between the profiles, suites, features, scenarios, and steps:

Profile A
  Suite A
    Feature A
      Scenario A
        Step 0
        Step 1
      Scenario B
        Step 0
    Feature B
      Scenario C
        Step 0
  Suite B
    Feature C
      Scenario D
        Step 0

I will also introduce the behat.yml file early in this article. It configures our tests. This is where we name our test profiles, test suites, services and allocate PHP classes to each test suite. If we do not provide this file, Behat will try to execute all the test suites under the default profile (which is fine). But it will also expect Feature files to reside in the features/ directory and step definitions to be inside the features/bootstrap/ directory which is not always ideal. For maximum clarity, I will recommend adding a behat.yml file in everything but a simple project. Behat documentation provides decent coverage of this file, so I won't go any deeper.

Stage one: Step definitions

Step definitions are the meat of a test suite. They are the class methods that do the actual heavy lifting. I cannot tell you what you should be writing inside these methods because that is specific to what you are trying to test. But I can tell you what you should do to relate a step defined in a Gherkin feature file with a step definition from your PHP class. Here is a simple example:

Feature: Mary and the Selfish giant
  Scenario: The chase
    Given Mary has a little lamb
    When the lamb runs into the garden of the Selfish giant
    Then Mary goes after the lamb
    And both get chased out by the giant

And then the corresponding step definitions:

class TheChase extends Context {
  /**
   * @Given Mary has a little lamb
   */
  function hasLamb() {}

  /**
   * @When the lamb runs into the garden of the Selfish giant
   */
  function runsIntoTheGarden() {}

  /**
   * @Then Mary goes after the lamb
   */
  function followTheLamb() {}

  /**
   * @Then both get chased out by the giant
   */
  function getsChased() {}
}

As you can see, Behat uses annotations to relate PHP methods acting as step definitions with Gherkin steps. It is really that simple. All the class methods above are empty because they are here for illustration purposes only.

Stage two: Arguments

Time goes by and now Mary has three little lambs. Let's update our scenario:

Feature: Mary and the Selfish giant
  Scenario: The chase
    Given Mary has 3 little lamb
    ...

This forces us to change the annotation in our PHP code:

...
/**
 * @Given Mary has 3 little lambs
 */
function hasLamb() {}
...

At this rate of growth in the lamb population, we can assume Mary will soon have a few more lambs which will force us to update the annotation once again. We can break out of this cycle by introducing arguments:

...
/**
 * @Given Mary has :lambCount little lamb
 */
function hasLamb($lambCount) {}
...

This is of course grammatically incorrect when she has more than one lamb. In that case we should write lambs instead of lamb. Optional parts of words, which are indicated by surrounding parts of a word in brackets, offers one of the solutions:

...
/**
 * @Given Mary has :lambCount little lamb(s)
 */
function hasLamb($lambCount) {}
...

We could have also used alternative words. These are indicated by the slash character separating all the possible words. To capture both lamb and lambs, we would have to write @Given Mary has :lambCount little lamb/lambs. An even more flexible solution is to use regular expressions.

All of the previous three solutions covers both Mary has a little lamb and Mary has 3 little lambs. But the solution that uses optional parts of word is the simplest and hence preferred.

As you can see in the previous code snippet, I have used a placeholder called :lambCount to capture the actual number of lambs. Behat makes such placeholders available to step definitions as method arguments. Regular expression sub-patterns are also allowed. The above code could also have been written as:

...
/**
 * @Given /^Mary has (a|\d+) little lambs*$/
 */
function hasLamb($lambCount) {}
...

Behat accepts three annotations for relating step definitions to steps:

@Given
@When
@Then

Cucumber allows @And and @But, but not Behat. For these, use any of the above. You can even use more than one annotation on a single method. Example follows:

...
/**
 * @Given Mary has :lambCount little lamb(s)
 * @Then Mary ended up with :lambCount little lamb(s)
 */
function hasLamb($lambCount) {}
...

Behat passes all arguments as strings. So both "a" and "3" will be made available to $lambCount as strings. But we get a chance to convert this string argument to an integer using a Transform method:

/**
 * @Transform :lambCount
 */
function toInt($lambCount) {

  if (ctype_digit($lambCount)) {
    return (int) $lambCount;
  } else if ($lambCount === "a") {
    return 1;
  } else {
    return 0;
  }
}

In fact, we can transform any argument to anything using a @Transform method.

Stage 3: Getting real

We will say bye to Mary, the lambs, and the Selfish giant in favour of something more realistic - A Drupal site. We will take a newly minted Drupal site and add a page or two.

Behat has a cousin named Mink whose job is to make programmatic interaction with websites easier. To use Mink in our Behat tests, we need to install the Behat Mink extension and reference it from our behat.yml file. Mink provides a single API to drive a virtual web browser. Under the bonnet, it is Goutte or Selenium that does the actual browsing. The code for this article uses Goutte as it involves zero setup. The same code will work with Selenium with the added advantage that part of the Mink API that requires JavaScript support will also become available for use. I won't bore you with more details and instead point you towards the Mink documentation.

Now a new feature:

Feature: Manage node
  Add a page node.

  Background:
    Given I am logged in as an admin user.

  Scenario: Create a page.
    Given this piece of content
      | title | Foo bar |
      | body  | Baz qux |
    When I fill in the form to create a page node
    Then I end up at page 1 with title "Foo bar"

  Scenario: Create another page node.
    Given this piece of content with title "blah blah" and the following body copy:
      """
      This is the
      Body of the
      Blah blah page.
      """
    When I fill in the form to create a page node
    Then I end up at page 1 with title "Blah blah"
    And feel very very happy :)

Now an outline of its step definitions:

class DrupalNodeCreation extends RawMinkContext {
  /**
   * @Given I am logged in as an admin user.
   */
  function login() {}

  /**
   * @Given this piece of content
   */
  function prepare(TableNode $table) {}

  /**
   * @Given this piece of content with title :title and the following body copy:
   */
  function prepareAnother($title, PyStringNode $body) {}

  /**
   * @When I fill in the form to create a page node
   */
  function createNode() {}

  /**
   * @Then I end up at page 1 with title :expectedPageTitle
   */
  function checkNewPage($expectedPageTitle) {}

  /**
   * @Then feel very very happy :)
   */
  function feelHappy() {}

  /**
   * Setup the Drupal database file.
   *
   * @BeforeScenario
   */
  function setupSite(BeforeScenarioScope $scope) {}

  /**
   * @AfterScenario
   */
  function logout() {}

  /**
   * @Transform :title
   */
  function toTitleCase($title) {}
}

I just want you to concentrate on one method in the above code and that is setupSite(). It is a hook method which I haven't mentioned anything about yet. Worry not, because they are simple. They run at different stages of a test suite's execution. There is a whole bunch of them and I present all of them in this family photo:

Profile
  @BeforeSuite
  Suite
    @BeforeFeature
    Feature
      @BeforeScenario
      Scenario
        @BeforeStep
        Step
        @AfterStep
      @AfterScenario
    @AfterFeature
  @AfterSuite

Back to Mink. All method calls inside DrupalNodeCreation::login() come from the RawMinkContext class. These methods are used to load the Drupal login page, find the username and password form fields, and finally submit the form.

You may have noticed the use of PyStringNode in the signature of the DrupalNodeCreation::prepareAnother() method. PyStringNode is used to capture multiline text arguments.

Stage 4: Create a page

We will go straight to the step definition:

/**
 * @Given this piece of content
 */
function prepare(TableNode $table) {

  $map = $table->getRowsHash();
  $this->pageTitle = $map['title'];
  $this->pageBody  = $map['body'];
}

Seen that Behat\Gherkin\Node\TableNode object? It captures a whole table full of data. Extremely useful for adding tabular data to Gherkin tests. The TableNode class has some useful methods for extracting data out of the given table. Here are a few:

TableNode::getColumnsHash(): Fetches one row at a time. Useful where table headers are on the first row as is the case with most tabular data.
TableNode::getRowsHash(): Again, fetches one row, but expects table headers on the first column.
TableNode::getIterator(): Useful for using the table object in loops. Expects table headers on the first row.

Stage 5: Create some more pages

Gherkin's scenario outlines are useful for applying the same step on several different data sets. Example:

...
Scenario Outline:
  Given I ate <count0> biscuits
  When I thought of eating <count1> more
  Then I discovered I have lost appetite :(

  Examples:
    | count0 | count1 |
    | 2      | 2      |
    | 4      | 1      |
...

The above is a shorthand for the following:

...
Scenario:
  Given I have eaten 2 biscuits
  When I thought of eating 2 more
  Then I discovered I have lost appetite :(

Scenario:
  Given I have eaten 4 biscuits
  When I thought of eating 1 more
  Then I discovered I have lost appetite :(
...

Scenario outlines have been most useful while adding different pages and users using the same scenario.

Stage 6: Create users

We carry on and create one more Gherkin feature for adding users to our Drupal site. Having a second feature allows me to demonstrate code reuse between our test classes. But first, the feature text:

Feature: Create user

  Background: I have to have the right privileges to create users.
    Given I am logged in as an admin user who can create users

  Scenario Outline: Add a new user
    Given the desired "<Username>" and "<Email>" of a user
    When I fill in the form to create that user
    Then the user "<Username>" appears in the users list

    Examples:
      | Username | Email           |
      | Foo      | foo@example.net |
      | Bar      | bar@example.net |
      | Baz      | baz@example.net |

The step definitions are very similar to those for page creation. Let's compare their outlines:

DrupalCreateUser

login()
prepare()
createUser()
checkUserList()
setupSite()
logout()

DrupalCreateNode

login()
prepare()
prepareAnother()
createNode()
checkNewPage()
feelHappy()
setupSite()
logout()
toTitleCase()

Some of the above methods have exactly the same code. Here they are:

setupSite()
login()
logout()

We have two problems with these two PHP classes. First of all, Behat just won't execute them. In both features, we have a common step Given I am logged in as an admin user. This is fine. But we also have two step definitions (i.e. PHP methods) for the same step. This cannot happen because Behat just wouldn't know which method to execute.

The second problem is, by keeping the same code in two PHP classes, we are violating the DRY principle. We solve the first problem by tweaking the login step definition for the user creation scenario. Now it says Given I am logged in as an admin user who can create users. This is a minor annoyance that we will have to live with. Let's move to the second problem - DRY violation. I can think of two solutions to this:

Put the common code in a PHP trait.
Provide the common code as a service.

Both solutions are useful under different circumstances, so I have provided two sets of code for the same problem. Behat recently has gained the ability to autowire services. This is quite convenient, so I have added a third set of code that uses services but through autowiring.

That, pretty much ends our journey.

Good to know

Drupal acceptance testing

I have used a stock Drupal installation as the target of the acceptance tests. That doesn't mean I am presenting a Drupal specific solution in this article. I could have used any other framework. I have used Drupal because that is what I am most familiar with. On the other hand, if you are trying to figure out how to write automated acceptance tests for Drupal, definitely take a look at the DrupalExtension Behat extension.

Most Drupal sites use MySQL as its database although I have used SQLite. This is because SQLite keeps all its data in a single file which makes it easy to setup a fresh Drupal site within seconds. If you have lots of scenarios, the setup time for a fresh site becomes a matter of concern. Quick setup time makes it realistic to provide a fresh site for each Scenario allowing us to write Independent Scenarios. The author of The Cucumber book has this to say about independent scenarios -

...independent scenarios, ensures they put the system into a clean state and then add their own data on top if it. This makes them able to stand on their own, rather than being coupled to the data left behind by other tests or shared fixture data.... We can't stress enough how fundamental independent scenarios are to successful automated testing. Apart from the added reliability of scenarios that independently set up their own data, they're also clearer to read and debug...

I will explain the site setup process in detail in case it gives you some ideas for your own projects:

First checkout a copy of your Drupal site: $ git clone git@your-git-host/drupal-repo-path
Install using Drush: $ cd drupal-repo-path; drush --yes site-install --db-url=sqlite:///path/to/db.sqlite --account-pass=RANDOM-PASSWORD DRUPAL-PROFILE-NAME
Make a copy of the database file: $ cp /path/to/db.sqlite /path/to/behat-in-stages/tests/test-db/
Tell a web server to serve the Drupal site or use Drush to do that for now: $ drush runserver
Every time you need a fresh Drupal site, just copy the saved database file: $ cp /path/to/behat-in-stages/tests/test-db/db.sqlite /path/to/db.sqlite

Class autoloading

Behat is capable of autoloading PHP classes. The default location for classes is features/bootstrap/. Using the autoload key in your behat.yml file, you can tell Behat to look elsewhere. The autoload path can be mentioned once per profile. This means classes for all test suites belonging to a test profile must live in the same directory. I don't like this at all. I prefer to keep classes for each suite in their own directory instead. I would also prefer composer doing PSR4 autoloading over Behat doing PSR0 autoloading. The following snippet from our composer.json grants my wishes:

{
    ...
    "autoload": {
        ...
        "psr-4": {
          "StepDefs\\": "tests/step-definitions/"
        }
    }
}