Early Exits in Neural Network 

Ways to Introduce Early Exits in Training

• Joint Training
  • Sums the loss and trains on that
  • Loss can be weighed, but not so effectively used
• Layerwise Training
  • Trains until the first classification layer 
  • Freeze the f1 layer
  • Do the same with C2 and f2
• Backbone Training
  • train the backbone with respect to the last exit
  • train classifiers
  • Advantage of this method is that the last exit is the best exit
  • Can be very useful, since it is powerful enough to convert an existing neural network to Early Exits

Inference

• You only use early exits during training, so eliminate that during inference
• The ensemble of all the predictions of the exits
• Prevention of Overthinking: By applying early exits at an optimal depth, we can help increase accuracy