1 - Comon Testing Approaches & Model-Based Testing

 

 

Classic Testing Processes Overview

Online / Offline MBT Approach & steps Mapping

Model-Based Testing Process Benefits

Making Tests Executable: Principles of Test Adaptation

Definition of Testing

Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behavior. (IEEE Definition)

 

Dynamic: Execute the program with specific input values to find failures in its behavior.

Finite: Choose a smallish number of tests so that we can run the tests in the available time.

Selected: How to select the tests that are most likely to expose failures in the system.

Expected: We must decide whether the observed behavior of the system was a failure or not. This is called the oracle problem.

Classic Testing Processes Overview - 1

Classic Testing Processes Overview - 2

Model-Based Testing Process

The model-based testing process can be divided into the following five main steps:

1. Model the SUT and/or its environment.
2. Generate abstract tests from the model.
3. Concretize the abstract tests to make them executable.

4. Execute the tests on the SUT and assign verdicts.
5. Analyze the test results.

Model-Based Testing Domain

Offline MBT

Automate test case generation

Offline MBT means generating a finite set of tests and execute those later

This allows automatic test execution in third party test execution platform

Makes possible to create the Tool Chain

 

Modeler -> Test Generator -> Test Executor

Online MBT

Test case generation and
execution in motion
Next step is design after the output receiving
Testing nondeterministic systems
Infinite test suite running

Online / Offline Steps Mapping

MBT Benefits

 - Easier test suite maintenance
 - Automated test design

      Save effort

 - Better test quality

      no human faults
      Computer can find more combinations for complex
      systems than human brain

 - Online MBT provides also
      Testing nondeterministic systems
      Infinite test suite

MBT Limitations

A fundamental limitation of model-based testing is that it cannot guarantee to

find all the differences between the model and the implementation, not even if we generate an infinite stream of tests and run them on 1000 machines for

thousands of hours. This is a limitation of all kinds of testing.

 

A more practical limitation on adopting model-based testing is that it requires somewhat different skills from manual test design. The model designers must be able to abstract and design models, in addition to being expert in the application area

 

Pain Factors

Outdated Requirements: If some model-based testers start working from those outdated requirements, they will build the wrong model and find lots of “errors” in the SUT

Inappropriate use of model-based testing: The risk here is that it takes some experience to know which aspects of the SUT should be modeled and which should be tested manually

Time to analyze failed tests: When one of the generated tests fails, we must decide whether the failure is caused by the SUT, the adaptor code, or an error in the model.

Useless metrics: It is easy to generate huge numbers of tests, so the number-of-tests metric is not useful. It becomes necessary to move toward other measurements of test progress, such as SUT code coverage, requirements coverage, and model coverage metrics

Principles of Test Adaptation

Principles of Test Adaptation

The adaptation approach is to manually write some adapter code
that bridges the gap. This is essentially a wrapper around the SUT that
provides a more abstract view of the SUT to match the abstraction level
of the model.

The transformation approach is to transform the abstract tests into
concrete test scripts.

Principles of Test Adaptation

The mixed approach is a combination of the other two approaches.
It is sometimes useful to add some adapter code around the SUT to raise its abstraction level part of the way toward the model and make testing easier, and then transform the abstract tests into a more concrete form that matches the adapter interface.

 

Some benefits of this mixed approach are that the transformation can be easier, since the levels of abstraction are closer, and the adapter can be less model-specific, which may allow it to be reused for several different models or different testing scenarios.

2 - LTS Base Theory (Hints)

 

 

 

 

 

Labelled Transition Systems Definition

Internal / External Actions

Preface

fMBT is an online MBT tool, which use the labelled transaction system theory

 

The transition systems are an abstract relational model based on primitive notions of state and transition and, for the modelling scope, you must be able to learn about the capabilities of such systems to perform actions belonging to a predetermined set Act

 

Now, let's define more formally a Labelled Transaction System

LTS Definition

An LTS is a 4-tuple (Q, q0 , Act ∪ {τ }, R) where:
• Q is a countable, not-empty set of states;
• q0 is the initial state;
• Act is a countable, not empty set of visible actions; τ is an action that does not belong in Act, used to represent internal actions of the structure.
• R ⊆ Q × (Act ∪ {τ }) × Q is the transition relation such that an element (s, α, r) ∈ R if there is the possibility of moving from the state s to the state r through the action α

 

A LTS can be represented by a tree whose root is the Initial state q0 ( the initial states are represented by a small circle inside) , the transition relations are represented by arcs between nodes ( arcs labeled with the shares belonging to Act ∪ { τ } ) , the nodes represent the states finally belonging to Q .

LTS Definition

• Act is used to denote the set of visible actions: a,b,c ... will denote the elements of this set; the actions constitute a visible interaction with the external environment through system ports .So, if we denote the set of doors in the system considered with the symbol Λ, we represent the set of all visible actions visible as Act = {α : α ∈ Λ}

• Act∗ is the set of all finite sequences of elements of Act: s1 , s2 , ... represents the elements of this set, called traces or sequences and ε represent the empty trace.

 

In an LTS, a path (or computation) π is a sequence (q0 , α0 , q1 ) (q1 , α1 , q2 ) (q2 , α2 , q3 ) ... where every 3-tuple (qi , αi , qi+1 ) ∈ R.

LTS Example

From the starting node q0, we've two distinct arcs (transactions  (q0 , play, q1 ) and (q0 , work, q2 ) ) that reach two different states q1 & q2. From them, we can reach q3 which has only one outgoing arc (transaction (q3 , τ, q4 ) ), with which you reach the end node q4

3 - The fMBT Framework

 

 

 

The Value of the MTB approach 

Introduction to the fMBT Project

fMBT Architecture

fMBT Model

fMBT Test generation and execution & AAL/Python

Coverages, Heuristics and End Conditions

The value of the Approach

 

 - An MBT framework generates tests for you.

Anything that can be tested with automated test cases, can be tested using model-based testing, too.

 

 - fMBT also allow inspecting the state of the SUT using preconditions analysis

This enables, for instance, generating tests that look which buttons are on the display, choose a test step that clicks one of them, and then look again what could be the next test step.

 

The value of the Approach

 

The other way around, model-based testing is not beneficial, if:

 - the system under test is stateless
 - there are no different combinations (such as parameter values or   environment configurations) that should be tested.

For a stateless systems it's enough to test every input only once. The order in which test steps are executed and inputs sent does not matter.

There is no reason to use model-based testing tools for generating many test step sequences in this case.

 

fMBT

The fMBT (free Model-Based Testing) tool automatically generates and executes tests.

 

fMBT is targeted for a wide audience, including application, system and service developers, integrators and quality assurance teams. The system under test can be anything from device drivers to operating systems, to mobile devices and distributed services.

 

This Project Provides: A set of tools for fully automatic test generation and execution, and a collection of utilities and libraries that support high level of test automation.

This includes Python libraries for multiplatform GUI testing, a tool for editing, debugging, running and recording GUI test scripts, and a tool for editing and visually analyzing test models and generated tests.

fMBT Internal Architecture

What they do?

Model Component

Specifies which actions (test steps) are enabled.

 

Heuristic Component

Makes decision which action should be executed next.

Test Engine Component

Test generation & execution mainloop. Most important connections for the test engine are the heuristic and the adapter components.

 

1. Check that tags in the current state are ok (Tagged states are verified when entered by executing adapter blocks of all tags in the state).

2. Loop until meeting an end condition:

   2.1. Observe outputs from the adapter as long as there are new observations. Update model, check tags and end conditions after each observation.

   2.2  Ask heuristics to suggest an input action and the adapter to execute it. If the adapter reported an unknown action or an illegal action, the test fail. If it reported an allowed action, update model state, check tags and end conditions.

Adapter Component

Adapters are responsible for translation between actions in test models and what actually happens in systems under test (SUT) and test environments.

 

Adapters have two responsibilities in the fMBT tool:

 

1. Read an input action suggested by the test engine, try to execute it, and finally report the result back to the engine in the form of any (input or output) action, or a special "unknown response" action.

2. Make observations on SUT and the test environment, and report the results to the test engine in the form of output actions (precondition analysis).

Local & Remote Adapter

Local adapters are adapters which implement the fMBT adapter API, and are instantiated inside the process that also generates the test, that is fMBT.

 

Remote adapters are adapters that implement the adapter protocol. They are running on a separate process, but may or may not run on the same host. There is always at least one local adapter in a test run. The name and the parameters of the adapter are specified in a test configuration file.

 

fMBT Test Configuration File

Configuration file tells fMBT:

 

- how to load model and adapter,

- which test generation heuristics will be used,

- how to measure coverage,

- when to end test generation and execution, and

- what to do on different verdicts.

 

More details on the example configuration, complete reference on the fMBT Configuration section

Test generation & execution

Load test conguration, most importantly model,           adapter and end conditions.

Test generation & execution

Load test conguration, most importantly model,           adapter and end conditions.

Loop:

 - choose one of possible test steps

 - try executing it in an adapter

 - validate executed test step reported by the adapter

 - execute the test step in the test model

until any of the end conditions is met

AAL/Python

input " name " {

    guard { code: return true if the test step can be executed }

    adapter { code: interact with the system under test }
    body { code: update variables after successful execution }
}

fMBT transforms AAL/Python into a labelled transition system (LTS) before deciding which test step it should choose to be executed next.

The LTS that would correspond to complete AAL/Python model is often infinite or too big to be generated completely. Therefore fMBT generates LTSes only up to the depth that is necessary for making the decision.

"The current state of the AAL" means values of all variables that are initialized in the initial_state block of the AAL definition file.

AAL/Python in Test Generation

4 - fMBT Usage Example: Android UI (With Demo)

 

 

Description

regressiontest.conf

Prerequisites

Generating tests

Running a test

Live Demo

 

Description

Phone.aal enables testing

• activating and backgrounding the phone application.

• calling, two simultaneous calls at max.

• putting calls on hold, switching between two calls.

• hanging up calls in any order.

regressiontest.conf

model = aal_remote(<command>)

uses a child process that executes <command> as a model. Communicates with the child using fMBT AAL protocol
adapter = aal

the adapter is implemented by the same AAL instance that is configured as a model.

heuristic = lookahead(6)

lookahead will simulate execution of all possible paths of length 6
coverage = perm(2)

measures the percentage of coverage of all permutations of 2 actions in the test model.

Prerequisites

In order to use fMBT for Android testing, you will need three things:

 

fMBT (of course), obtained from the GitHub Page

Android SDK, obtained from the Android Developer Website

Android Emulator (Nexus S - Android 4.4.2 with SD Card emulation enabled)

 

If you use an Android device, you will need to enable USB debugging in Settings.

Your Android environment is ready when you can login to the device or emulator with adb shell.

Generating tests

In the example, fMBT generates tests from test steps defined in phone.aal according to test configuration in regressiontest.conf

The configuration aims to cover all possible input pairs (perm(2)). You can change this, for instance, to cover five unique input sequences from launching the phone application to returning to the homescreen, replace coverage = ... line in regressiontest.conf with

 

coverage  = uinputs(from 'i:launch.*' to 'i:return.*')
pass      = coverage(5)

 

Tip: d.enableVisualLog(...) in phone.aal starts logging whatever happens in the Device instance. The log is written to devicelog.html.

Running the test

Run the test in fmbt-androidtest example:

 

1. Start Nexus-S emulator in Android SDK.
   • Make sure adb shell connects to it.
   • Make sure Nexus-S is in the homescreen and there are no calls in progress.
2. Execute fmbt regressiontest.conf in the fmbt-androidtest directory.

 

Once the test run is over, open devicelog.html with your browser  to see the details in each executed test step and verified states.

 

Live Demo

5 - Conclusions

In the future we expect to see model-based testing expanding from focusing mostly on functional testing into other kinds of testing, such as security testing, performance testing, load testing, robustness testing and GUI testing.

Three major trends in the software industry seem to be conspiring to make the increased adoption of model-based testing inevitable:

 

 - The increasing level of complexity in modern software

 - The increasing demand for software that is less buggy, more secure, and more
robust.

 - The increasing use of models in software development.

 

4.1 References

 

Books & Papers:

Mark Utting - Practical Model-Based Testing - A tools Approach

Model-Based Testing of Reactive Systems - Advanced Lectures

Jonathan Jacky - Model-Based Software Testing and Analysis with C#

Mark Timmer - Model Based Testing (Formal methods & Tools)

 

External References:

fMBT Page

fMBT GitHub Repository

fmbt-androidtest GitHub Repository

4.2 Acknowledgement

Antti Kervinen, for the support and the presentation review. Thank you!!

Simone Piunno, Stefano Guandalini & the whole B! QA Staff for the support

My Dad, for conveying to me his passion for technology

My beloved girlfriend, for the patience and the coffee