Voice Assistant

Our project

• Bucket of services
• NLP-based ecosystem
• Complex back-end
• Intent recognition
• Dialog as Directed Acyclic Graph

Directed Acyclic Graph

Software

• Python 3.7+
• PyTest tests
• Flask Admin
• Allure

OK, got it!

What about testing?

Behavior-Driven Development

Main idea

Tech + Business

=

Perfect Product

BDD goals for project

• Perfect for testing of dialogs 
• Finite number of steps
• Clear user and system spec

Behavioral specification

• Title
• Narrative
• Scenarios
  • Given steps
  • When steps
  • Then steps

Example

Background:

        Given I am user with debit card TinkoffBlack

        And I use iOS mobile app version 5.2

Scenario: Voice Assistant transfers money

        When I tell 'Send 1000 rubles to Max'

        Then bot answers 'You're going to send 1000 rubles to                                                         Maxim Ivanov?'

        And bot displays buttons [Cancel; Confirm]

        When I press 2nd button

        Then bot answers 'Success'

Get BDD started!

BDD ecosystem

• Fixtures
• CLI entrypoint
• BDD scenarios
• Regression
• Tool for testing

Wait...

Questions

• How to get rid of dependences?
  • IDE
  • Code
  • Environment 
• How many people?
• How much code?
• How difficult scenario creation?
• What's the price of support?

Behavior-Driven Development

is brainteaser

Problems

1. Hardly scalable / shareable

2. Difficult maintenance

3. How about pretty reports?

4. Not suitable project !

5. Not interested team !

1. Hardly

Scalable / Shareable

• Need to train people
• Code couldn't be shared
• IDE dependence
• Environment dependence

2. Difficult maintenance

• Not synchronous development
• Documentation update
• Tests for testing tool too!

3. Reports

• Poor or very complex tools
• Not pretty reports
• Not BDD solutions
• * Not integratable with code

4. Project

Not suitable

5. Team

Not interested

Analyze

your own case

Let's form the requirements

1. Scaling & Sharing

• IDE independence
• ENV independence
• Gherkin UI for people
• Execution possibility
• Explanation and support

2. Maintenance

• Synchronous development
• Project code reuse
• Based on PyTest
• CI / CD integration

3. Reports

• Built-in decorators
• CI / CD support
• Allure framework

4. Project team

• BDD for task setting
• Features coverage
• Debug possibility
• Improvement opportunity

Our solution

BDD-testing service

based on Flask Admin

Ingridients

• Flask Admin
• Gherkin editor (Ace)
• pytest_bdd
• Allure
• SQLAlchemy

Service

Flask Admin goals

• Simple and powerful
• jinja2 template out-of-box
• SQLAlchemy out-of-box
• Already in use

Features

Ace editor goals

• based on JS
• Gherkin support
• Easy implementation
• Customizable

Editing panel

pytest_bdd goals

• Powerful as PyTest
• Test code reuse

Steps

Allure goals

• PyTest support
• Customizable decorators
• Pretty BDD reports
• Statistics out-of-box

Report

Conception

Create

Run

Analyze

Deploy

Demonstration

Service as

powerful testing tool

Service gives an ability to

• Create and manage scenarios
• Run tests and provide experiments
• Compile results and analyze reports 
• Add scenarios to regression
• Share with other specialists
• Set development tasks as scenarios