Learning structured and non-redundant representations with deep neural networks

Amir Hossein Abiri
Sharif University of Technology

Outline

  • Review Deep Neural Network
  • Overfitting problem
  • Regularizers
  • SDC
  • Soft SDC
  • Applying on CNN
  • Results

Deep Neural Network

Deep Learning
Fall 97

Overfitting Problem

Deep Learning
Fall 97

Approaches to avoid over-fitting

  • L1 Regularization
  • L2 Regularization
  • Dropout
  • Early Stop
  • Pruning
  • ...
Deep Learning
Fall 97

Dropout Pros

  • Better Generalization
  • Lower Variance

Dropout Cons

  • Late convergence
  • High Bias

Higher drop rate causes better generalization but increases bias.

Lower drop rate decreases bias but affects generalization

Deep Learning
Fall 97

Structured Decorrelation Constraint (SDC)

To avoid overfitting problems, we must first find the causes.

It is believed that overfitting in neural networks is a result of same layer neurons' co-adaption. Now, we try to control the co-adaption in a single FC layer by adding a regularizer term to loss function.

Co-adaption can be determined by covariance matrix.

Deep Learning
Fall 97

SDC

Cross-entropy loss function:

Let H be the output matrix of FC layers

Now the covariance matrix would be defined as:

Where     denotes the tth sample response of the ith neuron

Deep Learning
Fall 97

Soft SDC

What if we need some (actually a few) neurons to be co-adapted?

Is it actually the best way to always minimize the covariance matrix?

 

Idea: Divide neurons into k  groups. Minimize the covariance matrix between these k  groups. This allows some co-adaption between neurons.

Deep Learning
Fall 97

Grouping Neurons

Deep Learning
Fall 97

Running on CNN

The structure of constraining the convolutional layer. During training, the convolutional layer (Conv) is mapped to a GAP layer, then we decorrelate the GAP layer, which in turn has an effect of decorrelation on the Conv layer. JG denotes the Group Co-adaption of the GAP layer, JY is the cross-entropy loss of the classification layer.

Deep Learning
Fall 97

Results

Deep Learning
Fall 97

References

  • Yang, Jihai, et al. "Learning structured and non-redundant representations with deep neural networks." Pattern Recognition 86 (2019): 224-235.
Deep Learning
Fall 97
Made with Slides.com