DEEP LEARNING 101

Andrew Beam, PhD

Head of Machine Learning/Senior Fellow

Flagship Pioneering

March 3rd, 2019

twitter: @AndrewLBeam

WHY DEEP LEARNING?

DEEP LEARNING HAS MASTERED GO

Source: https://deepmind.com/blog/alphago-zero-learning-scratch/

Human data no longer needed

DEEP LEARNING HAS MASTERED GO

DEEP LEARNING CAN PLAY VIDEO GAMES

DEEP LEARNING CAN DIAGNOSE PATIENTS

What does this patient have?

A six-year old boy has a high fever that has lasted for three days. He has extremely red eyes and a rash on the main part of his body in addition to a swollen and red strawberry tongue. Remaining symptoms include swollen lymph nodes in the neck and Irritability

DEEP LEARNING CAN DIAGNOSE PATIENTS

What does this patient have?

Image credit: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

DEEP LEARNING CAN DIAGNOSE PATIENTS

What does this patient have?

EVERYONE IS USING DEEP LEARNING

WHAT IS DEEP LEARNING?

Deep learning is a specific kind of machine learning

- Machine learning automatically learns relationships using data

- Deep learning refers to large neural networks

- These neural networks have millions of parameters and hundreds of layers (e.g. they are structurally deep)

- Most important: Deep learning is not magic!

HOW DID WE GET HERE?

NEURAL NETWORKS ARE AN OLD IDEA

One of the very first ideas in machine learning and artificial intelligence

Date back to 1940s
Many cycles of boom and bust
Repeated promises of "true AI" that were unfulfilled followed by "AI winters"

Are today's neural nets any different than their predecessors?

"[The perceptron is] the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence." - Frank Rosenblatt, 1958

IN THE BEGINNING... (1940s-1960s)

Rosenblatt's Perceptron, 1957

Initially very promising
Came with provably correct learning algorithm
Could recognize letters and numbers

THE FIRST AI WINTER (1969)

Minsky and Papert show that the perceptron can't even solve the XOR problem

Kills research on neural nets for the next 15-20 years

THE BACKPROPAGANDISTS EMERGE (1986)

Rumelhart, Hinton, and Willams show us how to train multilayered neural networks

REBRANDING AS 'DEEP LEARNING' (2006)

Unsupervised pre-training of "deep belief nets" allowed for large and deeper models

Image credit: https://www.toptal.com/machine-learning/an-introduction-to-deep-learning-from-perceptrons-to-deep-networks

THE BREAKTHROUGH (2012)

Imagenet Database

Millions of labeled images
Objects in images fall into 1 of a possible 1,000 categories
Relatively high-resolution
Bounding boxes giving exact location of object - useful for both classification and localization

Large Scale Visual

Recognition Challenge (ILSVRC)

Annual Imagenet Challenge starting in 2010
Successor to smaller PASCAL VOC challenge
Many tracks including classification and localization
Standardized training and test set. Competitors upload predictions for test set and are automatically scored

THE BREAKTHROUGH (2012)

Pivotal event occurred in the 2012 ILSVRC which brought together 3 critical ingredients:

Massive amounts of labeled images
Training with GPUs
Methodological innovations that enabled training deeper networks while minimizing overfitting

THE BREAKTHROUGH (2012)

In 2011, a misclassification rate of 25% was near state of the art on ILSVRC

In 2012, Geoff Hinton and two graduate students, Alex Krizhevsky and Ilya Sutskever, entered ILSVRC with one of the first deep neural networks trained on GPUs, now known as "Alexnet"

Result: An error rate of 16%, nearly half what the second place entry was able to achieve.

The computer vision world immediately took notice

THE ILSVRC AFTERMATH (2012-2014)

Alexnet paper has ~ 30,000 citations since being published in 2012!

WHY NOW?

MODERN DEEP LEARNING

Several key advancements have enabled the modern deep learning revolution

Advent of massively parallel computing by GPUs.

MODERN DEEP LEARNING

Several key advancements have enabled the modern deep learning revolution

Advent of massively parallel computing by GPUs.

21 TFLOPs of computing power. Would have been the fastest super computer on Earth around the year 2000!

You can rent one on the Amazon cloud for $3/hour!

Several key advancements have enabled the modern deep learning revolution

Methodological advancements have made deeper networks easier to train

Architecture

Optimizers

Activation Functions

MODERN DEEP LEARNING

Several key advancements have enabled the modern deep learning revolution

Robust frameworks and abstractions make iteration faster and less error prone

Automatic differentiation allows easy prototyping

MODERN DEEP LEARNING

CASE STUDY

DIABETIC RETINOPATHY DETECTION

DIABETIC RETINOPATHY

Why did this work so well?

- Huge dataset of over 100,000 images

- High quality annotations - each image was rated by 3-7 opthamologists

- Transfer learning - neural network was originally trained on Imagenet!

- For the cost of a GPU (~$1,000) it's possible to read 240 million images/day at accuracy on par with best ophthalmologists!

DIABETIC RETINOPATHY

Implications

- Many subsequent studies have followed this formula

- Ingredients: Deep learning + high quality database of ~100,000 medical images + transfer learning

- Many medical imaging tasks in radiology, pathology, dermatology, and opthamology can be fully automated in a similar manner

- Similar results emerging from non-image data

How will this technology change medical practice, reimbursement, and other policies?

SUMMARY & CONCLUSIONS

DEEP LEARNING AND YOU

Barrier to entry for deep learning is actually low

... but a few things might stand in your way:

Need to make sure your problem is a good fit
- Lots of labeled data and appropriate signal/noise ratio
Access to GPUs
Must "speak the language"
- Many design choices and hyper parameter selections
Know how to "babysit" the model during learning phase