Towards Structured Testing in Machine Learning Applications

Cristina Morariu

PhD, PMP, Mo2

What we want to achieve...

Understand different types of Machine Learning (ML)

Formulate a testing strategy for a ML application


What we will not cover in this talk...

Machine Learning Algorithms and how to choose the best

ML project phases




Today's environment

Definitions and Types of Machine Learning

Testing your Learning Algorithm and Possible Problems to Consider

Useful Resources




Today's Environment


“ML is a core, transformative way by which we’re rethinking how we’re doing everything”
Sundar Pichai, Google

“AI is the new electricity. Just as electricity transformed many industries roughly 100 years ago, AI will also now change nearly every major industry — healthcare, transportation, entertainment, manufacturing — enriching the lives of countless people.”
Andrew Ng, Standford








Today's Environment



Processing Power

Advanced algorithms

Available data








Machine Learning


Arthur Samuel (1959):

Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.


Tom Mitchell (1998):

Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.








A Look into the Past and the Future









8000 Years

Agricultural Revolution

120 Years

Industrial Revolution

90 Years

Light Bulb

22 Years

Moon Landing

9 Years


Human Genome Sequenced

Types of Machine Learning

Supervised Learning


Inferring a function from training data

Labelled training data

Each training example is a pair of an input object and an output values

Map new examples


Supervised Learning










Supervised Learning Examples


Based on some prior knowledge (when its sunny, temperature is higher; when its cloudy, humidity is higher, etc.) weather apps predict the parameters for a given time.

Based on past information about spams, classify if an email is spam or not.










Unsupervised Learning


Inferring a function from "unlabelled" data

​There is no evaluation of the accuracy, since the examples are unlabelled

To cluster the data, or visualize the data


Unsupervised Learning








Unsupervised Learning Examples


Google news clustering the articles


Data visualisation








Reinforcement Learning



Reward the agent's action

Maximize cumulative reward


Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.



Reinforcement Learning














Reinforcement Learning Examples

Game playing. Player knows whether it won ("yay!'') or lost ("aww''), but not how it should have moved.









A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.

MountainCar-v0 defines "solving" as getting average reward of -110.0 over 100 consecutive trials.

What to Look for (Possible Problems)

Possible Bugs - Overfitting












Possible Bugs - Underfitting












Possible Bugs - Real Examples






Training and Test Set in Supervised Models





Raw Data

Pre-processing (feature extraction/ missing data)

Training Set

Test Set


Data Processing (feature scaling/ dimension reduction)

Hypothesis definition and optimization

Final Classification/ Regression Model

Test error

Real Input Data

Cross validation


Training and Test Set in Supervised Models








Training Data



Cross Validation Data


Test Data

Testing Unsupervised Models


1. The Gold Standard: Hand label an external data set

2. By evaluation the performance of subsequent systems


Testing the Overall System - Ceiling Analysis

Camera Image

Preprocess (remove background)

Face detection

Eyes Segmentation



Logistic Regression




Overall System


Face detection

Eyes Seg.

Nose Seg.

Mouth Seg.

Logistic Regression








Useful Resources

Andrew Ng - Machine Learning, free course on Coursera

OpenAi Machine Learning Curriculum, on Github

Thank you!


Don't forget to send me your feedback using #1331


DevTalks 2017 - Towards Structured Testing in AI Applications

By Cristina Morariu

DevTalks 2017 - Towards Structured Testing in AI Applications

  • 787
Loading comments...

More from Cristina Morariu