The 'Toolbox'
Testing Approach
James Gibbard
@jgibbarduk
#PNSQC
James Gibbard
QA Technical Lead at Intel Security
Introduction
- Deliver value to the customer
- Ensure high quality throughout
- Increase release speed (maybe using agile)
Stressed engineering teams
+
=
Recent trend to use more complex software stacks
Software engineering teams are asked to…
Value
Quality
Software Tester
Speed
Complex Stack
Flexible Pragmatic
High Quality
The Challenges
The 'Thin Vertical Slice'
COTS Test Automation Tools are not flexible enough for today's complex software stacks
Layers need testing separately & together
Commercial off the shelf (COTS) test automation tools are used by many software companies
Typically they are...
- Expensive
- Inflexible
- High cost of maintenance
Test Automation Tools
Test with a... 'Toolbox' Approach
Toolbox Approach
- Not a tool
- Not really a framework
-
You can't download it
Its an approach to writing automated test cases
Database
Testing
UI
Testing
+
(Could even be called a Philosophy)
Toolset User Stories
- Be usable in multiple testing situations
- Be usable for testing multiple types of applications
- Be flexible enough to run on a test/dev box
-
Be able to be plugged into Continuous Integration
- Be able to allow test case reviews by people who may not understand programming
As a tester, I want my test automation toolbox to...
A small lie...
What makes the Toolbox approach work...
-
Generic keyword-driven test case language
-
Use keywords derived from different libraries or tools
-
The whole team can review, learn and suggest changes to automated test cases
-
Test cases or parts of test cases can be re-used
High-level,
Abstracted Language,
Test Automation Frameworks
Reusing Keywords
| Promotes keyword resuse | Reduces test case creation time | Reduces maintenance costs |
Robot Framework
Introducing..
Robot Framework
- Open-source & completely free
- Easy-to-use tabular test data syntax
- Uses keyword-driven testing approach
- Extended by test libraries implemented either with Python or Java
- Test cases written so stake-holders can understand
Robot Stack
Features & Benifits
- Multi-platform support
- Continuous Integration
- Configuration Management
- Probably the best documentation I've ever seen!
- Simple but powerful test result reporting/logging
- Many more...
Robot Reporting
Robot Reporting
Solving the...
Thin Vertical Slice Problem
Identify & Standardise
on
Testing Tools
Build Your Tools / Libraries
Putting it all together
Global settings, including importing your built-in and custom libraries
*** Settings ***
Library OperatingSystem
Library Database_Lib
Library UI_Lib
Library Common_Lib*** Variables ***
${API-URL} http://localhost/api/user/create
${UI-URL}http://localhost/ui/users
*** Test Cases ***
TC01 – Verify new user is created (in Database) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
Verify User in Database 'qa-user'
[Teardown] Purge User from Database 'qa-user'TC02 – Verify new user is created (in UI) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
Open Browser ${UI-URL} Chrome
Verify User in UI 'qa-user'
[Teardown] Purge User from Database 'qa-user'
Global variables that can be reused in multiple test cases
Test Case 01: Initially created to test the API & DB
Test Case 02: Reusing the ‘Create User’ keyword from TC01, but testing the UI
Augmenting Test Cases
Tool/Library is unable to interact with software under test (e.g. bugs or poorly implemented functionality)
Situation
Make use of another tool/library to fill in the gap (i.e. augment your test case)
Solution
Augmenting Test Cases
*** Settings ***
Library OperatingSystem
Library Database_Lib
Library UI_Lib
Library Common_Lib
*** Variables ***
${API-URL} http://localhost/api/user/create
${UI-URL} http://localhost/ui/users
*** Test Cases ***
TC02 – Verify new user is created (in UI) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
Open Browser ${UI-URL} Chrome
Verify User in UI 'qa-user'
[Teardown] Purge User from Database 'qa-user'*** Test Cases ***
TC02 – Verify new user is created (in UI) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
UI_Lib.Open Browser ${UI-URL} Chrome
Verify User in UI 'qa-user'
[Teardown] Purge User from Database 'qa-user'*** Settings ***
Library OperatingSystem
Library Database_Lib
Library UI_Lib
Library Common_Lib
Library CMD_Lib
*** Test Cases ***
TC02 – Verify new user is created (in UI) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
CMD_Lib.Open Browser ${UI-URL} Chrome
Verify User in UI 'qa-user'
[Teardown] Purge User from Database 'qa-user'*** Test Cases ***
TC02 – Verify new user is created (in UI) using API
[Tags] API Regression
Create User 'qa-user' ${API-URL}
UI_Lib.Open Browser ${UI-URL} Chrome
Verify User in UI 'qa-user'
[Teardown] Purge User from Database 'qa-user'*** Settings ***
Library OperatingSystem
Library Database_Lib
Library UI_Lib
Library Common_Lib
Future Direction &
Conclusion
Future Direction
- Customers or business stakeholders involved in test case reviews (of automated tests)
- Automated UAT tests could be created by the user
- Commonly used libraries and keywords can be refactored to be globally usable. Providing a vast array of keywords to use on new projects
Conclusion
- Enables teams to have a flexible, pragmatic approach to writing automated test cases
- It no longer has to be a daunting task to create test automation for complex software stacks
- Teams will be able to write test cases faster, by not having to learn different test tools for different layers of the stack
- Start to break down the barriers to test case automation
Any Questions?
@jgibbarduk
linkedin.com/in/jgibbarduk
jgibbard.me.uk
http://xkcd.com/327/
The 'Toolbox' Testing Approach
By James Gibbard
The 'Toolbox' Testing Approach
In an ever changing and increasing world of agile software development, automated testing approaches have struggled to keep up with the amount of software that can be developed. This paper sets out a solution to the problem, with the use of a ‘toolbox’ test approach, leveraging the power of Robot Framework. The key and the purpose of this paper, is to help you understand how to fit all the tools together, to form your ready-for-anything toolbox.
