The 'Toolbox'

Testing Approach

James Gibbard

@jgibbarduk

#PNSQC

Introduction

  • Deliver value to the customer
  • Ensure high quality throughout
  • Increase release speed (maybe using agile)

Stressed engineering teams

+
++
=
==

Recent trend to use more complex software stacks

Software engineering teams are asked to…

Value

Quality

Software Tester

Speed

Complex Stack

Flexible Pragmatic
High Quality

The Challenges

The 'Thin Vertical Slice'

COTS Test Automation Tools are not flexible enough for today's complex software stacks

Layers need testing separately & together

Test with a... 'Toolbox' Approach

Toolset User Stories

  1. Be usable in multiple testing situations
     
  2. Be usable for testing multiple types of applications
     
  3. Be flexible enough to run on a test/dev box
     
  4. Be able to be plugged into Continuous Integration
     
  5. Be able to allow test case reviews by people who may not understand programming

 As a tester, I want my test automation toolbox to...

What makes the Toolbox approach work...

  • Generic keyword-driven test case language
     

  • Use keywords derived from different libraries or tools
     

  • The whole team can review, learn and suggest changes to automated test cases
     

  • Test cases or parts of test cases can be re-used

High-level,
Abstracted Language,
​Test Automation Frameworks

Reusing Keywords

Promotes keyword resuse Reduces test case creation time Reduces maintenance costs

Robot Framework

Introducing..

Robot Framework

  • Open-source & completely free
     
  • Easy-to-use tabular test data syntax
     
  • Uses keyword-driven testing approach
     
  • Extended by test libraries implemented either with Python or Java
     
  • Test cases written so stake-holders can understand

Robot Stack

Features & Benifits

  • Multi-platform support
     
  • Continuous Integration
     
  • Configuration Management
     
  • Probably the best documentation I've ever seen!
     
  • Simple but powerful test result reporting/logging
     
  • Many more...

Robot Reporting

Robot Reporting

Solving the...
Thin Vertical Slice Problem

Identify & Standardise 

on

Testing Tools

Build Your Tools / Libraries

Putting it all together

Global settings, including importing your built-in and custom libraries

*** Settings ***
Library    OperatingSystem
Library    Database_Lib
Library    UI_Lib
Library    Common_Lib
*** Variables ***
${API-URL}	http://localhost/api/user/create
${UI-URL}http://localhost/ui/users

*** Test Cases ***
TC01 – Verify new user is created (in Database) using API
    [Tags] API Regression
    Create User	    'qa-user'    ${API-URL}
    Verify User in Database    'qa-user'
    [Teardown] Purge User from Database    'qa-user'
TC02 – Verify new user is created (in UI) using API
    [Tags] API Regression
    Create User    'qa-user'    ${API-URL}
    Open Browser    ${UI-URL}    Chrome
    Verify User in UI    'qa-user'
    [Teardown] Purge User from Database    'qa-user'

Global variables that can be reused in multiple test cases

Test Case 01: Initially created to test the API & DB

Test Case 02: Reusing the ‘Create User’ keyword from TC01, but testing the UI

Augmenting Test Cases

Tool/Library is unable to interact with software under test (e.g. bugs or poorly implemented functionality)

Situation

Make use of another tool/library to fill in the gap (i.e. augment your test case)

Solution

Augmenting Test Cases

*** Settings ***
Library    OperatingSystem
Library    Database_Lib
Library    UI_Lib
Library    Common_Lib


*** Variables ***
${API-URL}    http://localhost/api/user/create
${UI-URL}     http://localhost/ui/users

*** Test Cases ***

TC02 – Verify new user is created (in UI) using API
    [Tags] API Regression
    Create User    'qa-user'    ${API-URL}
    Open Browser    ${UI-URL}    Chrome
    Verify User in UI    'qa-user'
    [Teardown] Purge User from Database    'qa-user'
*** Test Cases ***

TC02 – Verify new user is created (in UI) using API
    [Tags] API Regression
    Create User    'qa-user'    ${API-URL}
    UI_Lib.Open Browser    ${UI-URL}    Chrome
    Verify User in UI    'qa-user'
    [Teardown] Purge User from Database    'qa-user'
*** Settings ***
Library    OperatingSystem
Library    Database_Lib
Library    UI_Lib
Library    Common_Lib
Library    CMD_Lib

*** Test Cases ***

TC02 – Verify new user is created (in UI) using API
    [Tags] API Regression
    Create User    'qa-user'    ${API-URL}
    CMD_Lib.Open Browser    ${UI-URL}    Chrome
    Verify User in UI    'qa-user'
    [Teardown] Purge User from Database    'qa-user'
*** Test Cases ***

TC02 – Verify new user is created (in UI) using API
    [Tags] API Regression
    Create User    'qa-user'    ${API-URL}
    UI_Lib.Open Browser    ${UI-URL}    Chrome
    Verify User in UI    'qa-user'
    [Teardown] Purge User from Database    'qa-user'
*** Settings ***
Library    OperatingSystem
Library    Database_Lib
Library    UI_Lib
Library    Common_Lib

Future Direction &
Conclusion

Future Direction

  • Customers or business stakeholders involved in test case reviews (of automated tests)
     
  • Automated UAT tests could be created by the user
     
  • Commonly used libraries and keywords can be refactored to be globally usable. Providing a vast array of keywords to use on new projects

Conclusion

  • Enables teams to have a flexible, pragmatic approach to writing automated test cases
     
  • It no longer has to be a daunting task to create test automation for complex software stacks
     
  • Teams will be able to write test cases faster, by not having to learn different test tools for different layers of the stack
     
  • Start to break down the barriers to test case automation

 

Any Questions?

@jgibbarduk

linkedin.com/in/jgibbarduk

jgibbard.me.uk

http://xkcd.com/327/

Made with Slides.com