Amir Hossein Abiri Sharif University of Technology
Deep Learning
Fall 97
Deep Learning
Fall 97
Deep Learning
Fall 97
Higher drop rate causes better generalization but increases bias.
Lower drop rate decreases bias but affects generalization
Deep Learning
Fall 97
To avoid overfitting problems, we must first find the causes.
It is believed that overfitting in neural networks is a result of same layer neurons' co-adaption. Now, we try to control the co-adaption in a single FC layer by adding a regularizer term to loss function.
Co-adaption can be determined by covariance matrix.
Deep Learning
Fall 97
Cross-entropy loss function:
Let H be the output matrix of FC layers
Now the covariance matrix would be defined as:
Where denotes the
Deep Learning
Fall 97
What if we need some (actually a few) neurons to be co-adapted?
Is it actually the best way to always minimize the covariance matrix?
Idea: Divide neurons into k groups. Minimize the covariance matrix between these k groups. This allows some co-adaption between neurons.
Deep Learning
Fall 97
Deep Learning
Fall 97
The structure of constraining the convolutional layer. During training, the convolutional layer (Conv) is mapped to a GAP layer, then we decorrelate the GAP layer, which in turn has an effect of decorrelation on the Conv layer. JG denotes the Group Co-adaption of the GAP layer, JY is the cross-entropy loss of the classification layer.
Deep Learning
Fall 97
Deep Learning
Fall 97
Deep Learning
Fall 97