DeepLearning1

Machine Learning - Training

Courtesy: Keelin Murphy

@MurphyKeelin

Machine Learning - Testing

Courtesy: Keelin Murphy

@MurphyKeelin

Hand-Crafted Features

What's in the Box?

Courtesy: Keelin Murphy

@MurphyKeelin

Hand-crafting Features

Texture

Curvature

Courtesy: Keelin Murphy

@MurphyKeelin

Deep Learning

Neural Networks. Lots of hidden layers (deep)

The network determines what features are useful

Source: Alexander Del Toro Barba : https://www.linkedin.com/pulse/how-artificial-intelligence-revolutionizing-finance-del-toro-barba

Courtesy: Keelin Murphy

@MurphyKeelin

Neural Network - Training

Dog

Cat

Penguin

0.3

0.4

0.3

0.0

1.0

0.0

Input

Layer

Hidden Layers

(3)

Output

Layer

Neuron / Perceptron

= Array of Weights

ERROR

TRUTH

Back-Propagation

Update weights to minimize errors

Courtesy: Keelin Murphy

@MurphyKeelin

Neural Network - Testing

Dog

Cat

Penguin

0.08

0.02

0.9

Input

Layer

Hidden Layers

(3)

Output

Layer

Neuron / Perceptron

= Array of Weights

Courtesy: Keelin Murphy

@MurphyKeelin

Inside a Neural Network

x1

x2

x3

x4

Input

Layer

w11

w12

w13

w14

w21

w23

w24

b1

b2

Hidden Layer

2 neurons

(w11)(x1) + (w12)(x2)

+ (w13)(x3) + (w14)(x4) + b1

(w21)(x1) + (w22)(x2) +

(w23)(x3) + (w24)(x4) + b2

Output of hidden layer

w22

Further Layers

Courtesy: Keelin Murphy

@MurphyKeelin

Inside a Neural Network

Matrix Notation

Input

Layer

b1

b2

Hidden Layer

2 neurons

x

(4x1)

Output of hidden layer

W2 (1x4)

W1 (1x4)

(W1)(X) + b1

(W2)(X) + b2

W

(2x4)

b

(2x1)

Wx + b

(2x1)

Courtesy: Keelin Murphy

@MurphyKeelin

The Activation Function

Input

Layer

Hidden Layer

3 neurons

x

Output of hidden layer

Wx + b

Hidden Layer

2 neurons

f(Wx + b)

Courtesy: Keelin Murphy

@MurphyKeelin

The Activation Function

g()

ReLU

Rectified Linear Unit

Softmax

Input

Layer

Hidden Layer

3 neurons

x

Hidden Layer

2 neurons

g(Wx + b)

Dog

Cat

Penguin

Output Layer

0.3

0.4

0.3

Softmax Function

5

9

5

\sigma(z)_i=\frac{e^{z_i}}{\sum_{k=1}^{K}{e^{z_k}}}

\sigma(z)_i=\frac{e^{z_i}}{\sum_{k=1}^{K}{e^{z_k}}}

Courtesy: Keelin Murphy

@MurphyKeelin

Back-Propagation

Input

Layer

Hidden Layer

3 neurons

x

Hidden Layer

2 neurons

g(Wx + b)

Dog

Cat

Penguin

Output Layer

0.3

0.4

0.3

Softmax Function

Dog

Cat

Penguin

0.0

1.0

0.0

ERROR

TRUTH

Back-Propagation

Update weights to minimize errors

Courtesy: Keelin Murphy

@MurphyKeelin

Back-Propagation

ERROR

Error is measured by the "Loss" (Cost) function

Want to update each weight to reduce the Loss

W_{1} = W_{1} -\alpha\frac{\partial L}{\partial W_{1}}

W_{1} = W_{1} -\alpha\frac{\partial L}{\partial W_{1}}

The Loss is a function of the network outputs, which are in turn functions of W and b values

W1

Loss (L)

Learning-Rate

Slope

Gradient Descent

Courtesy: Keelin Murphy

@MurphyKeelin

Back-Propagation

W1

W2

The "optimizer" decides how to move to reduce loss.

Optimizers are variants of basic gradient descent

"ADAM"

"Momentum"

"RMSprop"

Courtesy: Keelin Murphy

@MurphyKeelin

Tensorflow Tutorial

MNIST data

28

flatten

1

784

Output Layer

10 Neurons (classes)

Input Layer

batch-size =100

X (100 x 784)

W (784 x 10)

b (1 x 10)

XW + b

Softmax

(100 x 10)

Probabalistic Output

Truth

(100 x 10)

Error (Loss, Cost)

(Cross-Entropy)

Courtesy: Keelin Murphy

@MurphyKeelin

Machine Learning

Going Deeper