Towards Structured Testing in Machine Learning Applications
PhD, PMP, Mo2
What we want to achieve...
Understand different types of Machine Learning (ML)
Formulate a testing strategy for a ML application
What we will not cover in this talk...
Machine Learning Algorithms and how to choose the best
ML project phases
Definitions and Types of Machine Learning
Testing your Learning Algorithm and Possible Problems to Consider
“ML is a core, transformative way by which we’re rethinking how we’re doing everything”
Sundar Pichai, Google
“AI is the new electricity. Just as electricity transformed many industries roughly 100 years ago, AI will also now change nearly every major industry — healthcare, transportation, entertainment, manufacturing — enriching the lives of countless people.”
Andrew Ng, Standford
Arthur Samuel (1959):
Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.
Tom Mitchell (1998):
Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
Types of Machine Learning
Inferring a function from training data
Labelled training data
Each training example is a pair of an input object and an output values
Map new examples
Supervised Learning Examples
Based on some prior knowledge (when its sunny, temperature is higher; when its cloudy, humidity is higher, etc.) weather apps predict the parameters for a given time.
Based on past information about spams, classify if an email is spam or not.
Inferring a function from "unlabelled" data
There is no evaluation of the accuracy, since the examples are unlabelled
To cluster the data, or visualize the data
Unsupervised Learning Examples
Google news clustering the articles
Reward the agent's action
Maximize cumulative reward
Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
Reinforcement Learning Examples
Game playing. Player knows whether it won ("yay!'') or lost ("aww''), but not how it should have moved.
A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.
MountainCar-v0 defines "solving" as getting average reward of -110.0 over 100 consecutive trials.
Biologically Inspired Algorithms
At the core of HTM are learning algorithms that can store, learn, infer and recall high-order sequences.
Limitations: It only works on one dimension, time-series, for now
Biologically Inspired Algorithms
What to Look for (Possible Problems)
Possible Bugs - Overfitting
Possible Bugs - Underfitting
Training and Test Set in Supervised Models
Cross Validation Data
Testing Unsupervised Models
1. The Gold Standard: Hand label an external data set
2. By evaluation the performance of subsequent systems
Possible Bugs - It's an Evolving System
Testing the Overall System - Ceiling Analysis
Preprocess (remove background)
Towards Structured Testing in AI Applications
By Cristina Morariu