Towards Structured Testing in Machine Learning Applications

Cristina Morariu

PhD, PMP, Mo2

What we want to achieve...

Understand different types of Machine Learning (ML)

Formulate a testing strategy for a ML application

 

What we will not cover in this talk...

Machine Learning Algorithms and how to choose the best

ML project phases

Agenda

 

Today's environment

Definitions and Types of Machine Learning

Testing your Learning Algorithm and Possible Problems to Consider

Useful Resources

 

 

Today's Environment

 

“ML is a core, transformative way by which we’re rethinking how we’re doing everything”
Sundar Pichai, Google


“AI is the new electricity. Just as electricity transformed many industries roughly 100 years ago, AI will also now change nearly every major industry — healthcare, transportation, entertainment, manufacturing — enriching the lives of countless people.”
Andrew Ng, Standford

 

 

 

 

 

 

 

Today's Environment

 

 

Processing Power

Advanced algorithms

Available data

 

 

 

 

 

 

 

Machine Learning

 

Arthur Samuel (1959):

Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.

 

Tom Mitchell (1998):

Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

 

 

 

 

 

 

 

Types of Machine Learning

Supervised Learning

 

Inferring a function from training data

Labelled training data

Each training example is a pair of an input object and an output values

Map new examples

 

Supervised Learning

 

 

 

 

 

 

 

Regression

Classification

Supervised Learning Examples

 

Based on some prior knowledge (when its sunny, temperature is higher; when its cloudy, humidity is higher, etc.) weather apps predict the parameters for a given time.


Based on past information about spams, classify if an email is spam or not.

 

 

 

 

 

 

 

 

 

Unsupervised Learning

 

Inferring a function from "unlabelled" data

​There is no evaluation of the accuracy, since the examples are unlabelled

To cluster the data, or visualize the data

 

Unsupervised Learning

 

 

 

 

 

 

 

Unsupervised Learning Examples

 

Google news clustering the articles

 

Data visualisation

 

 

 

 

 

 

 

Reinforcement Learning

 

 

Reward the agent's action

Maximize cumulative reward

 

Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

 

 

Reinforcement Learning

 

 

 

 

 

 

 

 

Agent

Environment

Action

State

Reward

Reinforcement Learning Examples

Game playing. Player knows whether it won ("yay!'') or lost ("aww''), but not how it should have moved.


 

 

 

 

 

 

 

 

A car is on a one-dimensional track, positioned between two "mountains". The goal is to drive up the mountain on the right; however, the car's engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.

MountainCar-v0 defines "solving" as getting average reward of -110.0 over 100 consecutive trials.

Biologically Inspired Algorithms

 

At the core of HTM are learning algorithms that can store, learn, infer and recall high-order sequences.

Limitations: It only works on one dimension, time-series, for now

 

 

 

 

Biologically Inspired Algorithms

 

 

 

 

 

What to Look for (Possible Problems)

Possible Bugs - Overfitting

 

 

 

 

 

Possible Bugs - Underfitting

 

 

 

 

 

Training and Test Set in Supervised Models

 

 

 

 

 

 

60%

Training Data

 

20%

Cross Validation Data

20%

Test Data

Testing Unsupervised Models

 

1. The Gold Standard: Hand label an external data set

2. By evaluation the performance of subsequent systems

 

Possible Bugs - It's an Evolving System

 

 

 

 

 

Testing the Overall System - Ceiling Analysis

Camera Image

Preprocess (remove background)

Face detection

Eyes

Nose

Mouth

Logistic Regression

Label

Component

Accuracy

Overall System

Pre-processing

Face detection

Eyes Seg.

Nose Seg.

Mouth Seg.

Logistic Regression

85%

85.1%

91%

95%

96%

97%

100%

1

0.1

5.9

Useful Resources

Andrew Ng - Machine Learning, free course on Coursera

OpenAi Machine Learning Curriculum, on Github

Thank you!

 

cristina.morariu@softvision.ro

 

morariu-cristina

Towards Structured Testing in AI Applications

By Cristina Morariu

Towards Structured Testing in AI Applications

  • 737
Loading comments...

More from Cristina Morariu