Misusing Cucumber

Andrey Mikhaylov (lolmaus)

frontend developer & EmberJS enthusiast

@ Mainmatter

https://lolma.us

https://github.com/lolmaus

https://mainmatter.com

repuprposing a BDD library for conventional acceptance testing

💖 tests

Give confidence
Detect regressions
Enable refactoring
Have business value

🤮 tests

Low quality code
Overly complex code
Overly complex seeding patterns

I have a dream

of a test suite that would:

standardize test code
make tests truly readable
and understandable
by non-developers
focus on intention, hide implementation details
speed up writing tests
make writing tests enjoyable

What is BDD?

assert.isEqual(actual, expected)

expect(actual).to.equal(expected)

actual.should.equal(expected)

What is BDD?

An assertion style?

What is BDD?

The double loop dev workflow?

BDD = user stories

Feature:
Login and Signup

Scenario:

Logging in successfully

As an anonymous user,

When I visit the login page,

Fill valid credentials

And submit the login form,

Then I should be logged in

And I should be on my profile

User stories:

are written by all team members together,
in human-readable language,
from the user's perspective.
Thread through the whole feature lifecycle: from conception to deployment.

User stories for signup and login

Visiting app sections:

Anonymous user visits a public section, proceeds to a login page by clicking a link
Anonymous user visits a restricted section, gets redirected to the login page
A logged-in user visits a restricted section successfully
A logged-in user visits the login page, gets redirected to their profile
A logged-in user visits a signup page, gets redirected to their profile
A logged-in user visits a password recovery page, gets redirected to their profile

Login page, anonymous user:

fills in correct credentials, authenticates, gets redirected to their profile
fills in incorrect credentials, sees an error message
attempts to login without filling credentials, the button is inactive
attempts to login with an invalid email, a validation error message appears, the button is inactive

A logged-in user visits a signup page, gets redirected to their profile

On the signup page, anonymous user:

fills in the form successfully, signs up and authenticates, gets redirected to their profile
fills in an invalid login, sees a validation error message, the button is inactive
fills in an occupied login, sees an async validation error message, the button is inactive
fills in an invalid password, sees a validation error message, the button is inactive
fills in a non-matching password, sees a validation error message, the button is inactive
visits password recovery page

Password recovery page, anon user

fills in a valid email address, sees a request to check the email, the email message gets sent
fills in an invalid email, sees a validation error message, the button is inactive

Password recovery email

visits a recovery link successfully
visits an expired link
visits a used link
visits a link while logged in

Benefits of
user stories and BDD

Reveal the true scope of the feature.
More realistic estimations.
Serve as a spec.
Serve as a single source of truth.
Easy to distribute work.
Easy to track progress.
Prioritize users' needs over developers' needs.
Integrate well with automated testing.

«The Inmates Are Running the Asylum» by Alan Cooper

Started over 14 years ago as a Ruby library, on top of RSpec.
Evolved into a full-fledged testing framework.
Ported to most popular programming languages.
JavaScript has the official CucumberJS and the unofficial YaddaJS.

Cucumber feature file

tests/acceptance/blog.feature Syntax is called "Gherkin"

@setupAcceptanceTest
@setupMirage

Feature: The blog

Background:
    Given there's a user Tom
    And I'm authorized as Tom

Scenario: Visiting the blog with posts
    Given there are 3 posts in the database
    When I visit the blog page
    Then I should see 3 posts
    And I should not see the empty blog message
    
Scenario: Visiting an empty blog
    Given there are 0 blog posts in the database
    When I visit the blog page
    Then I should see 0 posts
    And I should see the empty blog message

Step implementations

When I visit the blog page
Then I should see 3 posts

import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see 3 posts' () {
    const posts = find('.post');
    this.assert.equals(posts.length, 3);
  },
}

import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see (\\d+) posts' (countStr) {
    const posts = find('.post');
    const countInt = parseInt(countStr, 10);
    this.assert.equals(posts.length, countInt);
  }
}

import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see $number posts' (count) {
    const posts = find('.post');
    this.assert.equals(posts.length, count);
  }
}

tests/acceptance/blog.feature

tests/cucumber/steps.js

Multiline steps

Then the declaration of independence should say: 
  ---
  The unanimous Declaration of the thirteen united States of America,
  When in the Course of human events, it becomes necessary for one people
  to dissolve the political bands which have connected them with another,
  and to assume among the powers of the earth, the separate and equal
  station to which the Laws of Nature and of Nature's God entitle them,
  a decent respect to the opinions of mankind requires that they should
  declare the causes which impel them to the separation.
  ---

And there are records of type "accessory" with the following properties:
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  | id        | name           | code       | sorting | thumbnailImage | priceType  | priceValue  | accessoryCategoryId  | teaser    | minTotalWidth | maxTotalWidth | profileIds       |
  | "tshirts" | "tshirts-name" | "tshsirts" | 1       | "tshirts.jpg"  | "constant" | 100         | "2"                  | "tshirts" |               |               | ["p1","p2","p3"] |
  | "tshirtm" | "tshirtm-name" | "tshsirtm" | 2       |                | "constant" | 110         | "2"                  | "tshirtm" |               |               | ["p1","p2"]      |
  | "tshirtl" | "tshirtl-name" | "tshsirtl" | 3       | "tshirtl.jpg"  | "constant" | 120         | "2"                  | "tshirtl" |               |               | []               |
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Test case mattresses

matrices

Test case matrices

Scenario: Exceeding min/max value on the Roof-Width-Field, input [User Input], angle: [Angle]
  When I visit URL "/configurator/unterkonstruktion"
  And I select item "Individual" in the dropdown Preset-Field
  And I fill "[Angle]" into the Input of the Angle-Field
  And I fill "[User Input]" into the Input of the Roof-Depth-Field

  Then the Input in the Roof-Depth-Field should have text "[Actual Value]"
  And the Input in the Rafter-Length-Field should have text "[Rafter Length]"

  Where:
    -------------------------------------------------------
    | Angle | User Input | Actual Value   | Rafter Length |
    | 5     | 999        | 1000           | 1000          |
    | 5     | 7001       | 7000           | 7000          |
    | 25    | 999        | 1000           | 1097          |
    | 25    | 7000       | 6383           | 7000          |
    | 25    | 7001       | 6383           | 7000          |
    | 15    | 25000      | 6758           | 7000          |
    | 25    | 25000      | 6383           | 7000          |
    | 35    | 25000      | 5922           | 7000          |
    | 45    | 25000      | 5439           | 7000          |
    -------------------------------------------------------

Separation of concerns
Improved readability
Ease of validation
...by all team members
Code reusability
High speed of writing tests
Simple maintenance
Enforced discipline
Enforced code coverage
Uniform code style
Integration into workflow

Cucumber benefits

sucks

Cucumber issues

The library of step implementations keeps growing indefinitely.

Finding implementations for steps is difficult.

Cucumber issues

The library of step implementations keeps growing indefinitely.
Finding implementations for steps is difficult.
Every step is a black box.

Scenario: Viewing blog posts
    Given there are 3 posts in the database
    When I visit the blog
    Then the title of the 1st post should be "Hello World"

Cucumber issues

The library of step implementations keeps growing indefinitely.
Finding implementations for steps is difficult.
Every step is a black box.
Steps implicitly depend on each other.

  When I expand the 2nd post
  Then the expanded post should contain a comments section

Cucumber issues

The library of step implementations keeps growing indefinitely.
Finding implementations for steps is difficult.
Every step is a black box.
Steps implicitly depend on each other.
CSS selectors are counter-productive.

When I click on div p ul li:nth-child(2) a

Cucumber issues

The library of step implementations keeps growing indefinitely.
Finding implementations for steps is difficult.
Every step is a black box.
Steps implicitly depend on each other.
CSS selectors are counter-productive.
Debugging is notoriously difficult.

Cucumber issues

The library of step implementations keeps growing indefinitely.
Finding implementations for steps is difficult.
Every step is a black box.
Steps implicitly depend on each other.
CSS selectors are counter-productive.
Debugging is notoriously difficult.

Nobody cares about BDD anyway.

Solutions

Abandon BDD.

Focus on the convenience of acceptance testing
for myself as a developer.

Solutions

Step library size: limit to a small amount of tiny, fully reusable steps.

For example:

click
fill text into field
hover the mouse pointer
read text from an element
read text from a text field
select an item from a dropdown list

Solutions

Step library size: limit to a small amount
of tiny, fully reusable steps.
Hidden logic: extract "the truth"
from step implementaions into feature files.

    Given there are Posts in the database:
        -------------------------------------
    	| title         | body  | authorId  |
        | "Hello World" | "..." | "alice"   |
        | "Lorem Ipsum" | "..." | "bob"     |
        | "Foo Bar Baz" | "..." | "charlie" |
        -------------------------------------
    Then the title of the 1st post should be "Hello World"

    Given there are 3 posts in the database
    Then the title of the 1st post should be "Hello World"

Before:

Then the 1st product should be selected

After:

Then ".product:nth-child(1)" should have HTML class "is-selected"
And I should see ".product:nth-child(1) .comments"

Solutions

Step library size: limit to a small amount
of tiny, fully reusable steps.
Hidden logic: extract "the truth"
from step implementations into feature files.
Tangled implementations: steps reference
each other in the feature file,
not inside step implementations.

  Then the 4th post should contain a comments section

  Then the expanded post should contain a comments section

Solutions

Step library size: limit to a small amount
of tiny, fully reusable steps.
Hidden logic: extract "the truth"
from step implementations into feature files.
Tangled implementations: steps reference
each other in the feature file,
not inside step implementations.
CSS selector hell: enforce a restriction to
use only semantic data-test selectors.

Before:

When I click on div p ul li:nth-child(2) a

After:

When I click on "[data-test-post]:nth-child(2) [data-test-title]"

4. Use only semantic
data-test selectors

[data-test-main-menu]

 data-test-main-menu

           main-menu

           Main-Menu

4. Use "Labels": a convenient DSL
for data-test selectors

When I click [data-test-post]:nth-child(2) [data-test-expand-button]

4. Reverse the order
in compound selectors

When I click the Expand-Button in the 2nd Post

Before:

After:

<div class="container">
    <article data-test-post>
    </article>
</div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>

:nth-child() does not work as expected

2nd Post

[data-test-post]

:nth-child(2)

<div class="container">
    <article data-test-post>
    </article>
</div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>

Use :eq(2) instead of :nth-child(2)

2nd Post

[data-test-post]

:eq(2)

Solutions

Step library size: limit to a small amount
of tiny, fully reusable steps.
Hidden logic: extract "the truth"
from step implementations into feature files.
Tangled implementations: steps reference
each other in the feature file,
not inside step implementations.
CSS selector hell: enforce a restriction to
use only semantic data-test selectors.
Debugging: imrpove the output of test results.

5. Improving test output

Step name
Implementation
Assertion error
Arguments
Stack trace

Thank you!

» ^_^ «

Andrey Mikhaylov

lolma.us

github.com/lolmaus

mainmatter.com

ember i ember-bdd

A drop-in Cucumber solution for your Ember app, powered by YaddaJS

https://github.com/lolmaus/ember-bdd

A secret slide 🤫

Misusing Cucumber

Andrey Mikhaylov (lolmaus)

I have a dream

of a test suite that would:

What is BDD?

What is BDD?

An assertion style?

What is BDD?

The double loop dev workflow?

BDD = user stories

Feature: Login and Signup

Scenario:

Logging in successfully

User stories for signup and login

Visiting app sections:

Login page, anonymous user:

On the signup page, anonymous user:

Password recovery page, anon user

Password recovery email

Benefits of user stories and BDD

Cucumber feature file

Step implementations

Multiline steps

Test case mattresses

Test case mattresses

Test case mattresses

matrices

Test case matrices

Test case matrices

Cucumber benefits

ember i ember-bdd

A drop-in Cucumber solution for your Ember app, powered by YaddaJS

https://github.com/lolmaus/ember-bdd

Feature:
Login and Signup

Benefits of
user stories and BDD