Cucumber, BDD, and Automated Testing

Behavior Driven Development (BDD) is a software development process, which has its roots in Test Driven Development, and Agile processes in general. When done correctly, BDD is intended to ease collaboration between developers, QA, and less technical business stakeholders. This is achieved by describing software features in terms of their actual behavior, using natural language sentences, rather than technical implementation details, or specifications. Although different than the originally intended use, BDD has become very popular in software test automation practices and has been used to help structure the test automation project for iTMS (our in-house TMS system for loads)

BDD is made possible by using standardized syntaxes for describing system behaviors. The most well-known and widely used syntax is specified in the domain-specific language Gherkin.

Cucumber is one of the most popular BDD frameworks which utilizes the Gherkin syntax. Cucumber allows for the system behaviors to be written in Gherkin sentences, each of which is executable, tied with corresponding code, which implements the tests and behaviors. Cucumber is essentially, executable documentation, and this ability is the main reason why Cucumber has become popular in test automation. It allows testers to write business requirements in English, and then to write test automation code behind it, and run the code from the actual Cucumber file, instead of running from a main method or test case in the programming language of choice.

A Step refers to each executable Gherkin sentence in a Cucumber file, and is the most fine-grained part of the Cucumber file. A Step Definition is the corresponding “glue code”, which is tied to the step and is supposed to implement the behavior or test described in the Step.

Keywords:

Cucumber/Gherkin uses a set of keywords to describe behaviors, generalized in a Given, When, Then flow:

• Given - describes a precondition, predefined state, and/or a context to the behavior or test.
• When - gives the action to be performed by the user.
• Then - the verification of the expected outcome.

The keywords And and But can be used to separate other keywords, for better readability.

The Feature keyword starts each “feature file” (the Cucumber file which is being written). It is the top level of each behavior described in the feature of the system. A description of the feature should be given after the Feature keyword (the description is not executable). The Scenario keyword starts the description of the more fine-grained parts of the system, and describes each behavior and outcome of the behavior, e.g. acceptance criteria. A description of the scenario should also be given after the Scenario keyword. Each separate behavior for the system should be specified in separate scenario blocks. Comments in Cucumber are delineated by the ‘#’ character.

The below Cucumber feature exemplifies the above keywords, in the context of iTMS (our in-house TMS system for loads):

Feature: A broker without all permissions can create both reefer and non-reefer loads in iTMS

    Scenario: create a reefer load and verify that it appears in future loads
        Given the iTMS login page opens
        Given a broker without all permissions logs in to iTMS
        When a broker creates a new reefer load
        Then a new reefer load appears in the future loads section and is highlighted in yellow

    Scenario: create a non-reefer load and verify that it appears in future loads
        Given the iTMS login page opens
        Given a broker without all permissions logs in to iTMS
        When a broker creates a new non-reefer load
        Then a new non-reefer load appears in the future loads section and is highlighted in yellow

#example of a comment

More Keywords:

• Background - defines a set of steps which are common among the whole set of scenarios in the feature. In Cucumber, the Background is executed before each scenario is executed.
• Scenario Outline - a type of Scenario which allows for parameterized variables, used in conjunction with the Examples keyword. The number of times the Scenario Outline will execute is equal to the number of rows under each variable in the table. It will loop through each value, and substitute it with the variable placeholder in the step.
• Examples - defines a table of parameterized variable values, which are inserted into a step. The first row of the Examples table is the name of the variable (not substituted), and the columns are the values (substituted).

Using the above three keywords, we can refactor the first example from before, resulting in a shorter (if somewhat less readable) format, while testing the same feature:

Feature: A broker without all permissions can create both reefer and non-reefer loads in iTMS

    #this will be executed before every scenario under the Scenario Outline
    Background:
        Given the iTMS login page opens
        Given a broker without all permissions logs in to iTMS
    
    Scenario Outline: create reefer load and non-reefer load. verify that they appear in future loads
        When a broker creates a new "<loadType>" load
        Then a "<loadType>" load appears in the future loads section and is highlighted in yellow
          
    Examples:
    |loadType   |
    |reefer     |
    |non-reefer |

Cucumber and Test Automation - Pros, Cons, Complaints:

Pros: Cucumber, as originally intended by its creators, is first and foremost a collaboration tool, and this is its most valuable advantage when used with automated testing. After the Gherkin steps and scenarios are written in an easily understood format, the discussion and presentation of the test cases becomes much easier than discussing the lower level implementation details of each test, therefore making collaboration among testers and others easier.

The BDD nature of Cucumber can also lend easier translation of business requirements into tests cases. Typically, when writing the code for the test behaviors, it helps to understand the requirements easily, if they are first written in the Given, When, Then format - instead of starting to code without a clear understanding of the behavior and expected outcomes first.

Cucumber steps are also easily reusable. When a mature set of steps are defined, which cover most of the user behaviors in the system, new feature test cases can sometimes be easily assembled from the preexisting steps, with the addition of a minimal amount of new steps to define.

The three above points illustrate the separation of concerns which is facilitated by Cucumber, when applied to software test automation. The extremely useful combination of a domain-specific language for describing tests cases naturally (Gherkin), with the ability run the test cases from an organized file, has led to the widespread adoption of Cucumber for automated testing.

Cons: One of the main complaints about Cucumber when applied to automation, is that the extra layer of abstraction is unnecessary, adding more complexity and time to write tests. If the business requirements for a feature are written in a BDD style elsewhere, then the test cases can be implemented directly in code, without the overhead of rewriting the Gherkin sentences as well.

It can be difficult or impossible to share state between steps, if needed. The Cucumber steps are not a sophisticated programming language, so their functionality is limited. There will almost inevitably be behaviors, wherein the user uses data across multiple states in an application flow. This can be easily implemented in standard programming, with method arguments and return values, but is not straightforward in the world of Cucumber steps and their accompanying step definitions.

Cucumber is also easily misused, resulting in long sets of hard to understand scenarios, which have no business value or logic, or scenarios which are not actually testing the intended features. It requires constant attention to best practices and standards, to write meaningful tests.

Complaints: Despite the fact that Cucumber is extremely popular in test automation, its founders and purist adherents decry the use of Cucumber for test automation in isolation in any context. Their opinion is that Cucumber is not really a testing tool at all, and should only be used as a collaboration tool among team members, with test automation being only one part of the collaboration. The whole team should be involved in the BDD process for it to make sense. So, to them, it would not make much sense to retrofit Cucumber tests for mature, established tests, such as those in regression test suites, which have no prior BDD context (a top-down approach). In other words, Cucumber features and BDD are meant to drive the implementation, and not reflect it. Regression type tests would then be built up over time, through the BDD driven features (a bottom-up approach).

Cons and complaints notwithstanding, Part Two of this post will be a guide on using the automation tool Katalon to create a Cucumber feature file, implement steps in step definitions, and run them.

References and Further Reading:

https://cucumber.io/docs/cucumber/

https://dannorth.net/introducing-bdd/

https://docs.behat.org/en/v2.5/guides/1.gherkin.html

https://en.wikipedia.org/wiki/Behavior-driven_development

https://www.scaledagileframework.com/behavior-driven-development/

https://saucelabs.com/resources/articles/bdd-framework-cucumber

https://cucumber.io/blog/collaboration/the-worlds-most-misunderstood-collaboration-tool/