Towards Structured Testing in Machine Learning Applications
Cristina Morariu
PhD, PMP, Mo2
What we want to achieve...
Understand different types of Machine Learning (ML)
Formulate a testing strategy for a ML application
What we will not cover in this talk...
Machine Learning Algorithms and how to choose the best
ML project phases
Agenda
Today's environment
Definitions and Types of Machine Learning
Testing your Learning Algorithm and Possible Problems to Consider
Useful Resources
Today's Environment
“ML is a core, transformative way by which we’re rethinking how we’re doing everything”
Sundar Pichai, Google
“AI is the new electricity. Just as electricity transformed many industries roughly 100 years ago, AI will also now change nearly every major industry — healthcare, transportation, entertainment, manufacturing — enriching the lives of countless people.”
Andrew Ng, Standford
Today's Environment
Processing Power
Advanced algorithms
Available data
Machine Learning
Arthur Samuel (1959):
Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.
Tom Mitchell (1998):
Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
Types of Machine Learning
Supervised Learning
Inferring a function from training data
Labelled training data
Each training example is a pair of an input object and an output values
Map new examples
Supervised Learning
Regression
Classification
Supervised Learning Examples
Based on some prior knowledge (when its sunny, temperature is higher; when its cloudy, humidity is higher, etc.) weather apps predict the parameters for a given time.
Based on past information about spams, classify if an email is spam or not.
Unsupervised Learning
Inferring a function from "unlabelled" data
There is no evaluation of the accuracy, since the examples are unlabelled
To cluster the data, or visualize the data
Unsupervised Learning
Unsupervised Learning Examples
Google news clustering the articles
Data visualisation
Reinforcement Learning
Reward the agent's action
Maximize cumulative reward
Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
Reinforcement Learning
Agent
Environment
Action
State
Reward
Reinforcement Learning Examples
Game playing. Player knows whether it won ("yay!'') or lost ("aww''), but not how it should have moved.
A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.
MountainCar-v0 defines "solving" as getting average reward of -110.0 over 100 consecutive trials.
Biologically Inspired Algorithms
At the core of HTM are learning algorithms that can store, learn, infer and recall high-order sequences.
Limitations: It only works on one dimension, time-series, for now
Biologically Inspired Algorithms
What to Look for (Possible Problems)
Possible Bugs - Overfitting
Possible Bugs - Underfitting
Training and Test Set in Supervised Models
60%
Training Data
20%
Cross Validation Data
20%
Test Data
Testing Unsupervised Models
1. The Gold Standard: Hand label an external data set
2. By evaluation the performance of subsequent systems
Possible Bugs - It's an Evolving System
Testing the Overall System - Ceiling Analysis
Camera Image
Preprocess (remove background)
Face detection
Eyes
Nose
Mouth
Logistic Regression
Label
Component
Accuracy
Overall System
Pre-processing
Face detection
Eyes Seg.
Nose Seg.
Mouth Seg.
Logistic Regression
85%
85.1%
91%
95%
96%
97%
100%
1
0.1
5.9
Useful Resources
Andrew Ng - Machine Learning, free course on Coursera
OpenAi Machine Learning Curriculum, on Github
Thank you!
cristina.morariu@softvision.ro
morariu-cristina
Towards Structured Testing in AI Applications
By Cristina Morariu
Towards Structured Testing in AI Applications
- 1,997