19AIE205

Sleep stage classification using different ML algorithms

Python for Machine Learning

Aadharsh Aadhithya - CB.EN.U4AIE20001
Anirudh Edpuganti - CB.EN.U4AIE20005

Madhav Kishore - CB.EN.U4AIE20033
Onteddu Chaitanya Reddy - CB.EN.U4AIE20045
Pillalamarri Akshaya - CB.EN.U4AIE20049

Team-1

Sleep stage classification using different ML algorithms

DataSet

Delta

Theta

Alpha

Beta

K-Complex

0.5-4

4-8

8-13

13-22

0.5-1.5

Hz

Continious Wavelet Transform

Time-Frequency Analysis Framework

Continious Wavelet Transform

Time-Frequency Analysis Framework

W_{a,b} = \int s(t) \psi_{a,b}(t) dt

\psi_{a,b}(t) = \frac{1}{\sqrt{a}} \psi(\frac{t-b}{a})

Continious Wavelet Transform

Time-Frequency Analysis Framework

W_{a,b} = \int s(t) \psi_{a,b}(t) dt

\psi_{a,b}(t) = \frac{1}{\sqrt{a}} \psi(\frac{t-b}{a})

Similarity

Measure

Mother

Wavelet

Continious Wavelet Transform

Time-Frequency Analysis Framework

W_{a,b} = \int s(t) \psi_{a,b}(t) dt

\psi_{a,b}(t) = \frac{1}{\sqrt{a}} \psi(\frac{t-b}{a})

Continious Wavelet Transform

Time-Frequency Analysis Framework

W_{a,b} = \int s(t) \psi_{a,b}(t) dt

\psi_{a,b}(t) = \frac{1}{\sqrt{a}} \psi(\frac{t-b}{a})

a \rightarrow \text{Streching or Shrinking}

b \rightarrow \text{translation along time}

Continious Wavelet Transform

Time-Frequency Analysis Framework

W_{a,b} = \int s(t) \psi_{a,b}(t) dt

\psi_{a,b}(t) = \frac{1}{\sqrt{a}} \psi(\frac{t-b}{a})

f = \frac{f_c}{a T_s}

Delta

Theta

Alpha

Beta1

K-Complex

0.5-4

4-8

8-13

13-22

0.5-1.5

Hz

Beta2

22-35

Hz

Sleep spindles

12-14

Hz

s(t)

CWT

\int s(t) \psi_{a,b}(t) dt

E = - \sum p_i log(p_i)

Delta

Theta

Alpha

Beta1

K-Complex

0.5-4

4-8

8-13

13-22

0.5-1.5

Hz

Beta2

22-35

Hz

Sleep spindles

12-14

Hz

s(t)

CWT

\int s(t) \psi_{a,b}(t) dt

E = - \sum p_i log(p_i)

Features

Random Forest

Dataset

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

idx	x1	x2	x3	y

3	1.1	3.1	2.8	3

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

idx	x1	x2	x3	y

3	1.1	3.1	2.8	3

.	.	.	.	.

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

idx	x1	x2	x3	y

3	1.1	3.1	2.8	3

.	.	.	.	.

2	1.5	2.3	4.1	1

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

2	1.5	2.3	4.1	1

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

2	1.5	2.3	4.1	1

2	1.5	2.3	4.1	1

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

2	1.5	2.3	4.1	1

Random Forest

Dataset

idx	x1	x2	x3	y

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

3	1.1	3.1	2.8	3

.	.	.	.	.

.	.	.	.	.

.	.	.	.	.

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

2	1.5	2.3	4.1	1

Data 1

Random Forest

Dataset

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

2	1.5	2.3	4.1	1

Data 1

Random Forest

Dataset

3	1.1	3.1	2.8	3

idx	x1	x2	x3	y

.	.	.	.	.

.	.	.	.	.

1	1.2	3.4	4.5	0

2	1.5	2.3	4.1	1

2	1.5	2.3	4.1	1

Data 1

Random Feature Selection

Random Forest

Dataset

Data 1

Random Forest

Dataset

Data 1

Data 2

Random Forest

Dataset

Data 1

Data 2

Data 3

Random Forest

Dataset

Data 1

Data 2

Data 3

Data 4

Random Forest

Dataset

Data 1

Data 2

Data 3

Data 4

Data 5

Random Forest

Dataset

Data 1

Data 2

Data 3

Data 4

Data 5

Bootstrapped Data

Random Forest

Dataset

Data 1

Data 2

Data 3

Data 4

Data 5

x_1,x_2

x_1,x_3

x_1,x_2

x_2,x_3

x_1,x_3

Random Forest

Dataset

Data 1

x_1,x_2

Decision Tree

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

Random Forest

Dataset

Data 1

x_1,x_2

x1	x2	x3

x1	x2
1.3	3.8

1.3	3.8	4.9

Predict

Random Forest

Dataset

Data 1

x_1,x_2

x1	x2	x3

1.3	3.8	4.9

Predict

x1	x2
1.3	3.8

Prediction

0

Random Forest

Dataset

x1	x2	x3

1.3	3.8	4.9

Predict

Prediction

0

Data 2

x_1,x_3

x1	x3
1.3	4.9

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

0

3

1

0

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

0

3

1

0

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

0

3

1

0

Majority Voting

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

0

3

1

0

Majority Voting

0

Random Forest

Dataset

Data 1

x_1,x_2

Data 2

Data 3

x_1,x_2

Data 4

x_2,x_3

Data 5

x_1,x_3

0

3

1

0

x1	x2	x3

1.3	3.8	4.9

Labelled as

y

0

Support Vector Machine

Text

Binary Classification

Support Vector Machine

Text

Binary Classification

Yes

No

Support Vector Machine

Text

Multi-Class Classification

Support Vector Machine

Text

Multi-Class Classification

Class 1

Class 2

Class 3

Support Vector Machine

Text

Multi-Class Classification

Class 1

Class 2

Class 3

Types

Support Vector Machine

Text

Multi-Class Classification

One vs Rest

One vs One

Support Vector Machine

Text

Multi-Class Classification

One vs Rest

One vs One

Support Vector Machine

Text

Multi-Class Classification

One vs One

Labels

1

2

3

Support Vector Machine

Text

Multi-Class Classification

One vs One

1,2

2,3

1,3

Support Vector Machine

Text

Multi-Class Classification

One vs One

1,2

2,3

1,3

One vs Rest

Support Vector Machine

Text

Multi-Class Classification

1\text{ vs } 2,3

One vs Rest

2\text{ vs } 1,3

3\text{ vs } 1,2

Support Vector Machine

Text

Multi-Class Classification

1\text{ vs } 2,3

One vs Rest

2\text{ vs } 1,3

3\text{ vs } 1,2

INTUITION

Support Vector Machine

Text

One vs Rest

INTUITION

x_1

x_2

1

2

3

Text

x_1

x_2

x_1

x_2

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

New data

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Ask

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Ask

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Ask

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

Ask

Text

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

x_1

x_2

K-Nearest Neighbours

x_1

x_2

K-Nearest Neighbours

New data

x_1

x_2

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=1

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=1

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=1

x_1

x_2

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=2

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=2

x_1

x_2

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=4

K-Nearest Neighbours

x_1

x_2

New data

Suppose K=4

x_1

x_2

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

g(z_{1})^1

g(z_{2})^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

g(z_{1})^2

\hat{y}

W^i = \begin{bmatrix} w^i_{11} & w^i_{12} & w^i_{13} & \cdots&w^i_{1n} \\ w^i_{21} & w^i_{22} & w^i_{23} & \cdots&w^i_{2n} \\ \vdots & \ddots& \ddots & \vdots \\ w^i_{m1} & w^i_{m2} & w^i_{m3} & \cdots&w^i_{mn} \\ \end{bmatrix}

m \rightarrow \text{no .of neurons in } i^{th} \text{ layer} \\ n \rightarrow \text{no .of neurons in } (i-1)^{th} \text{ layer}

X^i = \begin{bmatrix} x_1 \\ x_2\\ \vdots \\ x_n \end{bmatrix}

X^i \rightarrow \text{Output of } (i-1)^{th} \text{ layer} \\ n \rightarrow \text{no .of neurons in } (i-1)^{th} \text{ layer}

b^i = \begin{bmatrix} b_1^i \\ b_2^i \\ \vdots \\ b_m^i \end{bmatrix}

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

g(z_{1})^2

\hat{y}

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

L(y,\hat{y})

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

g(z_{1})^2

\hat{y}

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

L(y,\hat{y})

Forward Pass

z^i = W^i \cdot x^i + b^i \\ y^i = g(z^i)

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

g(z_{1})^2

\hat{y}

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

L(y,\hat{y})

Forward Pass

z^1 = W^1 \cdot x^1 + b^1 \\

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

g(z_{1})^2

\hat{y}

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

L(y,\hat{y})

Forward Pass

y^1 = g(z^1)

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

Forward Pass

\cdots \\ y^i = g(z^i)

g(z_{1})^1

g(z_{2})^1

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

Forward Pass

\cdots \\ y^i = g(z^i)

g(z_{1})^1

g(z_{2})^1

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

Multi Layer Network

x_1

w_{11}^1

x_3

x_2

w_{12}^1

w_{13}^1

w_{21}^1

w_{22}^1

w_{23}^1

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\cdots \\ y^i = g(z^i)

g(z_{1})^1

g(z_{2})^1

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

w_{j k}^i \rightarrow i^{th} \text{layer ,} j^{th} \text{neuron , mapping to } k^{th} \text{ input}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\text{changing } \hat{y} \text{ changes } L(y,\hat{y})

\frac{\partial L(y , \hat{y} )}{\partial \hat{y} } \frac{\partial \hat{y} }{\partial w^i_{jk}}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\text{changing } \hat{y} \text{ changes } L(y,\hat{y})

\frac{\partial L(y , \hat{y} )}{\partial \hat{y} } \frac{\partial \hat{y} }{\partial w^i_{jk}}

We can derive

To Be Computed

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\text{changing } \hat{y} \text{ changes } L(y,\hat{y})

\frac{\partial L(y , \hat{y} )}{\partial \hat{y} } \frac{\partial \hat{y} }{\partial w^i_{jk}}

We can derive

To Be Computed

Here , We Can Resort To Using The Chain Rule .

How G changes with x?

Changing x changes h(x)

\frac{d h(x)}{dx}

Changing h changes g

\frac{d g}{dh}

\frac{dg} {dx} = \frac{d h(x)}{dx} \cdot \frac{d g}{dh}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

w_{11}^2

w_{12}^2

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\text{How do we reach } w^i_{jk} \text{ From} L(y,\hat{y})

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\text{How do we reach } w^i_{jk} \text{ From } L(y,\hat{y}) ??

Follow The RED Path !

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } L(y , \hat{y} ) \text{ changes with } \hat{y}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^2

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1

\Sigma \\ z_2

g(z_2)

g(z_`)

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } \hat{y} \text{ changes with } (z_1)^{i+1}

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^{i+1}

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1^i

\Sigma \\ z_2^i

g(z_2)^i

g(z_1)^i

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } z_1^{i+1} \text{ changes with } g(z_i^i)

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^{i+1}

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1^i

\Sigma \\ z_2^i

g(z_2)^i

g(z_1)^i

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } g(z_1)^i \text{ changes with } z_1^i

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

\frac{\partial g(z_1)^i }{\partial z_1^i}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^{i+1}

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1^i

\Sigma \\ z_2^i

g(z_2)^i

g(z_1)^i

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } z_1^i \text{ changes with } w^i_{jk}

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

\frac{\partial g(z_1)^i }{\partial z_1^i}

\frac{\partial z_1^i}{\partial w^i_{jk}}

Multi Layer Network

x_1

x_3

x_2

w_{j k}^i

\hat{y}

L(y,\hat{y})

BackPropagation

\Sigma

\int

\Sigma\\ (z_1)^{i+1}

(g(z_1))^2

\Sigma

\int

\Sigma\\ z_1^i

\Sigma \\ z_2^i

g(z_2)^i

g(z_1)^i

w_{11}^1

w_{12}^1

w_{22}^1

w_{23}^1

\cdots

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}} =

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{How } L(y,\hat{y} ) \text{ changes with } w^i_{jk}

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

\frac{\partial g(z_1)^i }{\partial z_1^i}

\frac{\partial z_1^i}{\partial w^i_{jk}}

\frac{\partial g(z_1)^i }{\partial z_1^i}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

Multi Layer Network

BackPropagation

\frac{\partial L(y , \hat{y} )}{\partial w^i_{jk}} =

\frac{\partial z_1^i}{\partial w^i_{jk}}

\frac{\partial g(z_1)^i }{\partial z_1^i}

\frac{\partial z_1^{i+1} }{\partial g(z_i^i)}

\frac{\partial \hat{y}}{\partial (z_1)^{i+1}}

\frac{\partial L(y,\hat{y})}{\partial \hat{y}}

\text{ This is for one weights } w_{jk}^i \\ \text{for the matrix } W^i \text{ We Have a Detailed Derivation in the accompanying material}

For Softmax output layer and Sigmoid Activation function

Multi Layer Network

BackPropagation

Multi Layer Network

BackPropagation

Multi Layer Network

BackPropagation

Multi Layer Network

BackPropagation

So , now , we have all the vectorized components to build our chain

Multi Layer Network

Full Story

19AIE205

DataSet

DataSet

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Multi Layer Network

Thank you Mam!