Towards Structured Testing in Machine Learning Applications

Cristina Morariu

PhD, PMP, Mo2

What we want to achieve...

Understand different types of Machine Learning (ML)

Formulate a testing strategy for a ML application


What we will not cover in this talk...

Machine Learning Algorithms and how to choose the best

ML project phases



Today's environment

Definitions and Types of Machine Learning

Testing your Learning Algorithm and Possible Problems to Consider

Useful Resources



Today's Environment


“ML is a core, transformative way by which we’re rethinking how we’re doing everything”
Sundar Pichai, Google

“AI is the new electricity. Just as electricity transformed many industries roughly 100 years ago, AI will also now change nearly every major industry — healthcare, transportation, entertainment, manufacturing — enriching the lives of countless people.”
Andrew Ng, Standford








Today's Environment



Processing Power

Advanced algorithms

Available data








Machine Learning


Arthur Samuel (1959):

Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.


Tom Mitchell (1998):

Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.








Types of Machine Learning

Supervised Learning


Inferring a function from training data

Labelled training data

Each training example is a pair of an input object and an output values

Map new examples


Supervised Learning










Supervised Learning Examples


Based on some prior knowledge (when its sunny, temperature is higher; when its cloudy, humidity is higher, etc.) weather apps predict the parameters for a given time.

Based on past information about spams, classify if an email is spam or not.










Unsupervised Learning


Inferring a function from "unlabelled" data

​There is no evaluation of the accuracy, since the examples are unlabelled

To cluster the data, or visualize the data


Unsupervised Learning








Unsupervised Learning Examples


Google news clustering the articles


Data visualisation








Reinforcement Learning



Reward the agent's action

Maximize cumulative reward


Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.



Reinforcement Learning














Reinforcement Learning Examples

Game playing. Player knows whether it won ("yay!'') or lost ("aww''), but not how it should have moved.









A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.

MountainCar-v0 defines "solving" as getting average reward of -110.0 over 100 consecutive trials.

Biologically Inspired Algorithms


At the core of HTM are learning algorithms that can store, learn, infer and recall high-order sequences.

Limitations: It only works on one dimension, time-series, for now





Biologically Inspired Algorithms






What to Look for (Possible Problems)

Possible Bugs - Overfitting






Possible Bugs - Underfitting






Training and Test Set in Supervised Models








Training Data



Cross Validation Data


Test Data

Testing Unsupervised Models


1. The Gold Standard: Hand label an external data set

2. By evaluation the performance of subsequent systems


Possible Bugs - It's an Evolving System






Testing the Overall System - Ceiling Analysis

Camera Image

Preprocess (remove background)

Face detection




Logistic Regression




Overall System


Face detection

Eyes Seg.

Nose Seg.

Mouth Seg.

Logistic Regression











Useful Resources

Andrew Ng - Machine Learning, free course on Coursera

OpenAi Machine Learning Curriculum, on Github

Thank you!



Towards Structured Testing in AI Applications

By Cristina Morariu

Towards Structured Testing in AI Applications

  • 1,691