WiML: Unworkshop ICML 2020

A Review of Early Exit

Early Exits in Neural Network 

Ways to Introduce Early Exits in Training

  • Joint Training
    • Sums the loss and trains on that
    • Loss can be weighed, but not so effectively used
  • Layerwise Training
    • Trains until the first classification layer 
    • Freeze the f1 layer
    • Do the same with C2 and f2
  • Backbone Training
    • train the backbone with respect to the last exit
    • train classifiers
    • Advantage of this method is that the last exit is the best exit
    • Can be very useful, since it is powerful enough to convert an existing neural network to Early Exits

Inference

  • You only use early exits during training, so eliminate that during inference
  • The ensemble of all the predictions of the exits
  • Prevention of Overthinking: By applying early exits at an optimal depth, we can help increase accuracy

Applications of Early Exit : Edge Computing 

Improving Accuracy and reducing inference time can help implement these neural networks to edge devices 

Made with Slides.com