Learning structured and non-redundant representations with deep neural networks
Amir Hossein Abiri Sharif University of Technology
Outline
- Review Deep Neural Network
- Overfitting problem
- Regularizers
- SDC
- Soft SDC
- Applying on CNN
- Results
Deep Neural Network

Deep Learning
Fall 97
Overfitting Problem

Deep Learning
Fall 97
Approaches to avoid over-fitting
- L1 Regularization
- L2 Regularization
- Dropout
- Early Stop
- Pruning
- ...
Deep Learning
Fall 97
Dropout Pros
- Better Generalization
- Lower Variance
Dropout Cons
- Late convergence
- High Bias
Higher drop rate causes better generalization but increases bias.
Lower drop rate decreases bias but affects generalization
Deep Learning
Fall 97
Structured Decorrelation Constraint (SDC)
To avoid overfitting problems, we must first find the causes.
It is believed that overfitting in neural networks is a result of same layer neurons' co-adaption. Now, we try to control the co-adaption in a single FC layer by adding a regularizer term to loss function.
Co-adaption can be determined by covariance matrix.
Deep Learning
Fall 97
SDC
Cross-entropy loss function:

Let H be the output matrix of FC layers

Now the covariance matrix would be defined as:

Where denotes the



Deep Learning
Fall 97
Soft SDC
What if we need some (actually a few) neurons to be co-adapted?
Is it actually the best way to always minimize the covariance matrix?
Idea: Divide neurons into k groups. Minimize the covariance matrix between these k groups. This allows some co-adaption between neurons.
Deep Learning
Fall 97
Grouping Neurons




Deep Learning
Fall 97
Running on CNN

The structure of constraining the convolutional layer. During training, the convolutional layer (Conv) is mapped to a GAP layer, then we decorrelate the GAP layer, which in turn has an effect of decorrelation on the Conv layer. JG denotes the Group Co-adaption of the GAP layer, JY is the cross-entropy loss of the classification layer.
Deep Learning
Fall 97
Results


Deep Learning
Fall 97
References
- Yang, Jihai, et al. "Learning structured and non-redundant representations with deep neural networks." Pattern Recognition 86 (2019): 224-235.
Deep Learning
Fall 97
deck
By Amir Hossein Abiri
deck
- 456