Misusing Cucumber

Andrey Mikhaylov (lolmaus)

frontend developer & EmberJS enthusiast

@ Mainmatter

 

https://lolma.us

https://github.com/lolmaus

https://mainmatter.com

repuprposing a BDD library for conventional acceptance testing

💖 tests

  • Give confidence
  • Detect regressions
  • Enable refactoring
  • Have business value

🤮 tests

  • Low quality code
  • Overly complex code
  • Overly complex seeding patterns

I have a dream

of a test suite that would:

  • standardize test code
  • make tests truly readable
  • and understandable
    by non-developers
  • focus on intention, hide implementation details
  • speed up writing tests
  • make writing tests enjoyable
  • Started over 14 years ago as a Ruby library, on top of RSpec.
  • Evolved into a full-fledged testing framework.
  • Ported to most popular programming languages.
  • JavaScript has the official CucumberJS and the unofficial YaddaJS.

Cucumber feature file

tests/acceptance/blog.feature           Syntax is called "Gherkin"

@setupAcceptanceTest
@setupMirage

Feature: The blog

Background:
    Given there's a user Tom
    And I'm authorized as Tom

Scenario: Visiting the blog with posts
    Given there are 3 posts in the database
    When I visit the blog page
    Then I should see 3 posts
    And I should not see the empty blog message
    
Scenario: Visiting an empty blog
    Given there are 0 blog posts in the database
    When I visit the blog page
    Then I should see 0 posts
    And I should see the empty blog message

Step implementations

When I visit the blog page
Then I should see 3 posts
import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see 3 posts' () {
    const posts = find('.post');
    this.assert.equals(posts.length, 3);
  },
}
import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see (\\d+) posts' (countStr) {
    const posts = find('.post');
    const countInt = parseInt(countStr, 10);
    this.assert.equals(posts.length, countInt);
  }
}
import { find, visit } from '@ember/test-helpers';

export default {
  'When I visit the blog page' () {
    return visit('/blog');
  },
  
  'Then I should see $number posts' (count) {
    const posts = find('.post');
    this.assert.equals(posts.length, count);
  }
}

tests/acceptance/blog.feature

tests/cucumber/steps.js

Multiline steps

Then the declaration of independence should say: 
  ---
  The unanimous Declaration of the thirteen united States of America,
  When in the Course of human events, it becomes necessary for one people
  to dissolve the political bands which have connected them with another,
  and to assume among the powers of the earth, the separate and equal
  station to which the Laws of Nature and of Nature's God entitle them,
  a decent respect to the opinions of mankind requires that they should
  declare the causes which impel them to the separation.
  ---
And there are records of type "accessory" with the following properties:
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  | id        | name           | code       | sorting | thumbnailImage | priceType  | priceValue  | accessoryCategoryId  | teaser    | minTotalWidth | maxTotalWidth | profileIds       |
  | "tshirts" | "tshirts-name" | "tshsirts" | 1       | "tshirts.jpg"  | "constant" | 100         | "2"                  | "tshirts" |               |               | ["p1","p2","p3"] |
  | "tshirtm" | "tshirtm-name" | "tshsirtm" | 2       |                | "constant" | 110         | "2"                  | "tshirtm" |               |               | ["p1","p2"]      |
  | "tshirtl" | "tshirtl-name" | "tshsirtl" | 3       | "tshirtl.jpg"  | "constant" | 120         | "2"                  | "tshirtl" |               |               | []               |
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Test case mattresses

Test case mattresses

Test case mattresses

matrices

Test case matrices

Test case matrices

Scenario: Exceeding min/max value on the Roof-Width-Field, input [User Input], angle: [Angle]
  When I visit URL "/configurator/unterkonstruktion"
  And I select item "Individual" in the dropdown Preset-Field
  And I fill "[Angle]" into the Input of the Angle-Field
  And I fill "[User Input]" into the Input of the Roof-Depth-Field

  Then the Input in the Roof-Depth-Field should have text "[Actual Value]"
  And the Input in the Rafter-Length-Field should have text "[Rafter Length]"

  Where:
    -------------------------------------------------------
    | Angle | User Input | Actual Value   | Rafter Length |
    | 5     | 999        | 1000           | 1000          |
    | 5     | 7001       | 7000           | 7000          |
    | 25    | 999        | 1000           | 1097          |
    | 25    | 7000       | 6383           | 7000          |
    | 25    | 7001       | 6383           | 7000          |
    | 15    | 25000      | 6758           | 7000          |
    | 25    | 25000      | 6383           | 7000          |
    | 35    | 25000      | 5922           | 7000          |
    | 45    | 25000      | 5439           | 7000          |
    -------------------------------------------------------
  • Separation of concerns
  • Improved readability
  • Ease of validation
  • ...by all team members
  • Code reusability
  • High speed of writing tests
  • Simple maintenance
  • Enforced discipline
  • Enforced code coverage
  • Uniform code style
  • Integration into workflow

Cucumber benefits

sucks

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.

    Finding implementations for steps is difficult.

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.
    Finding implementations for steps is difficult.
     
  2. Every step is a black box.
Scenario: Viewing blog posts
    Given there are 3 posts in the database
    When I visit the blog
    Then the title of the 1st post should be "Hello World"

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.
    Finding implementations for steps is difficult.
     
  2. Every step is a black box.
     
  3. Steps implicitly depend on each other.
  When I expand the 2nd post
  Then the expanded post should contain a comments section

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.
    Finding implementations for steps is difficult.
     
  2. Every step is a black box.
     
  3. Steps implicitly depend on each other.
     
  4. CSS selectors are counter-productive.
When I click on div p ul li:nth-child(2) a

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.
    Finding implementations for steps is difficult.
     
  2. Every step is a black box.
     
  3. Steps implicitly depend on each other.
     
  4. CSS selectors are counter-productive.
     
  5. Debugging is notoriously difficult.

Cucumber issues

  1. The library of step implementations keeps growing indefinitely.
    Finding implementations for steps is difficult.
     
  2. Every step is a black box.
     
  3. Steps implicitly depend on each other.
     
  4. CSS selectors are counter-productive.
     
  5. Debugging is notoriously difficult.
     
  1. Nobody cares about BDD anyway.                                 

Solutions

  1. Abandon BDD.

    Focus on the convenience of acceptance testing
    for myself as a developer.

Solutions

  1. Step library size: limit to a small amount of tiny, fully reusable steps.

    For example:
  • click
  • fill text into field
  • hover the mouse pointer
  • read text from an element
  • read text from a text field
  • select an item from a dropdown list

Solutions

  1. Step library size: limit to a small amount
    of tiny, fully reusable steps.
     
  2. Hidden logic: extract "the truth"
    from step implementaions into feature files.
    Given there are Posts in the database:
        -------------------------------------
    	| title         | body  | authorId  |
        | "Hello World" | "..." | "alice"   |
        | "Lorem Ipsum" | "..." | "bob"     |
        | "Foo Bar Baz" | "..." | "charlie" |
        -------------------------------------
    Then the title of the 1st post should be "Hello World"
    Given there are 3 posts in the database
    Then the title of the 1st post should be "Hello World"

Before:

Then the 1st product should be selected

After:

Then ".product:nth-child(1)" should have HTML class "is-selected"
And I should see ".product:nth-child(1) .comments"

Solutions

  1. Step library size: limit to a small amount
    of tiny, fully reusable steps.
     
  2. Hidden logic: extract "the truth"
    from step implementations into feature files.
     
  3. Tangled implementations: steps reference
    each other in the feature file,

    not inside step implementations.
  Then the 4th post should contain a comments section
  Then the expanded post should contain a comments section

Solutions

  1. Step library size: limit to a small amount
    of tiny, fully reusable steps.
     
  2. Hidden logic: extract "the truth"
    from step implementations into feature files.
     
  3. Tangled implementations: steps reference
    each other in the feature file,

    not inside step implementations.
     
  4. CSS selector hell: enforce a restriction to
    use only semantic data-test selectors.

Before:

When I click on div p ul li:nth-child(2) a

After:

When I click on "[data-test-post]:nth-child(2) [data-test-title]"

4. Use only semantic
data-test selectors

[data-test-main-menu]
 data-test-main-menu
           main-menu
           Main-Menu

4. Use "Labels": a convenient DSL
for data-test selectors

When I click [data-test-post]:nth-child(2) [data-test-expand-button]

4. Reverse the order
in compound selectors

When I click the Expand-Button in the 2nd Post

Before:

After:

 

<div class="container">
    <article data-test-post>
    </article>
</div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>

:nth-child() does not work as expected

2nd Post

[data-test-post]

:nth-child(2)

1

2

3

4

5

<div class="container">
    <article data-test-post>
    </article>
</div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>
    
    
<div class="container">
    <article data-test-post>
    </article>

    <article data-test-post>
    </article>
 </div>

Use :eq(2) instead of :nth-child(2)

2nd Post

[data-test-post]

:eq(2)

1

2

3

4

5

Solutions

  1. Step library size: limit to a small amount
    of tiny, fully reusable steps.
     
  2. Hidden logic: extract "the truth"
    from step implementations into feature files.
     
  3. Tangled implementations: steps reference
    each other in the feature file,

    not inside step implementations.
     
  4. CSS selector hell: enforce a restriction to
    use only semantic data-test selectors.
     
  5. Debugging: imrpove the output of test results.

5. Improving test output

  • Step name
  • Implementation 
  • Assertion error
  • Arguments
  • Stack trace

Thank you!

» ^_^ «

Andrey Mikhaylov

lolma.us

github.com/lolmaus

mainmatter.com

ember i ember-bdd

A drop-in Cucumber solution for your Ember app, powered by YaddaJS

 

https://github.com/lolmaus/ember-bdd

A secret slide 🤫

Made with Slides.com