Intro to KD

Arvin Liu - MiRA Training course

Knowledge Distillation

Please use the link: slides.com/arvinliu/kd

Missing info in data

Taxonomy Dependency

What one-hot label might hurt your model?
- GT will give a large penalty to your model and make it overfit.

GT

Human

Model

P(y=5) = 0.5 \\ P(y=3) = 0.5

P(y=5) = 0.5 \\ P(y=3) = 0.5

P(y=5) = 0.4 \\ P(y=3) = 0.6

P(y=5) = 0.4 \\ P(y=3) = 0.6

P(y=5) = 1.0 \\ P(y=3) = 0.3

P(y=5) = 1.0 \\ P(y=3) = 0.3

Loss

QQ

What data

looks like

Before KD

Entropy Regularization

Make your entropy( logits ) more higher!

Class A
Data

Class A

Class B

Class C

CE

CE

CE

CE

CE

CE

CE

CE

Loss = CE(y',y)

Loss = CE(y',y)

+ \frac{\lambda}{CNum} \sum_{c}^{CNum} CE(y', c)

+ \frac{\lambda}{CNum} \sum_{c}^{CNum} CE(y', c)

Can't we make regularization more informative?

Baseline KD

Recap Label Refinery:

Model 1

GT

Model 2

Model 3

CE

CE

CE

CE

CE

CE

What about Baseline KD?

Teacher

GT

Student

KL_{div} \text{ with temperature}

KL_{div} \text{ with temperature}

Loss = \alpha T^2 \times KL(softmax(\frac{\text{Teacher's Logits}}{T})|| softmax(\frac{\text{Studente's Logtis}}{T})) + \\ (1-\alpha) (\text{Original Loss})

Loss = \alpha T^2 \times KL(softmax(\frac{\text{Teacher's Logits}}{T})|| softmax(\frac{\text{Studente's Logtis}}{T})) + \\ (1-\alpha) (\text{Original Loss})

Baseline KD - Temperature

Why we need temperature?
- Well-trained model has very low entropy -> like one-hot.

Loss = \alpha T^2 \times KL(softmax(\frac{\text{Teacher's Logits}}{T})|| softmax(\frac{\text{Studente's Logtis}}{T})) + \\ (1-\alpha) (\text{Original Loss})

Loss = \alpha T^2 \times KL(softmax(\frac{\text{Teacher's Logits}}{T})|| softmax(\frac{\text{Studente's Logtis}}{T})) + \\ (1-\alpha) (\text{Original Loss})

logits_{T} = [100, 10, 1]

logits_{T} = [100, 10, 1]

softmax(logits_{T}) = [1, \simeq0, \simeq0]

softmax(logits_{T}) = [1, \simeq0, \simeq0]

softmax(\frac{logits_{T}}{100}) = [0.56, 0.23, 0.21]

softmax(\frac{logits_{T}}{100}) = [0.56, 0.23, 0.21]

Deep Mutual Learning

Logits

y_{1,t}

y_{1,t}

y_{2,t}

y_{2,t}

x

x

Step 1: Update Net1

y

y

CE

D_{KL}(y_{2,t}||y_{1,t})

D_{KL}(y_{2,t}||y_{1,t})

Loss_1 = D_{KL}(y_{2,t}||y_{1,t})+ \text{CrossEntropy}(y_{1,t},y)

Loss_1 = D_{KL}(y_{2,t}||y_{1,t})+ \text{CrossEntropy}(y_{1,t},y)

Network 1

Network 2

GT

Deep Mutual Learning

Logits

y_{1,t}

y_{1,t}

y_{2,t}

y_{2,t}

x

x

Step 2: Update Net2

D_{KL}(y_{1,t}||y_{2,t})

D_{KL}(y_{1,t}||y_{2,t})

Loss_2 = D_{KL}(y_{1,t}||y_{2,t})+ \text{CrossEntropy}(y_{2,t},y)

Loss_2 = D_{KL}(y_{1,t}||y_{2,t})+ \text{CrossEntropy}(y_{2,t},y)

Network 1

Network 2

y

y

CE

GT

Loss_1 = D_{KL}(y_{2,t}||y_{1,t})+ \text{CrossEntropy}(y_{1,t},y)

Loss_1 = D_{KL}(y_{2,t}||y_{1,t})+ \text{CrossEntropy}(y_{1,t},y)

Teacher-free KD

We know about label refinery...

Model 1

GT

Model 2

Model 3

CE

CE

CE

CE

CE

CE

Born again neural networks. (ICLR 2018)

Two models teach each other in one generation. (CVPR 2018)

Teacher 1

GT

Student 1

CE

CE

CE

CE

CE

CE

Student 2

What if ... learn by myself in one generation? (???)

Model A

GT

Model A

GT

Model A1

Model B1

Model A2