A jamming transition from under- to over-parametrization
affects loss landscape and generalization

arxivs: 1809.09349, 1810.09665

Stefano Spigler, Mario Geiger

Stéphane d’Ascoli      Levent Sagun      Marco Baity-Jesi

Giulio Biroli      Matthieu Wyart

Deep learning: high-dimensional,

non convex problem

 

• What is the geometry of the landscape?
  
  
   
• How does it depend on the number of parameters?
  
  
   

• Why not stuck in local minima?

Set-up

• Dataset: \(\large \color{red}P\) points \(\mathbf{x}_1,\dots,\mathbf{x}_P\)
   
• Binary classification: each point \(\mathbf{x}_\mu\) has a label \(y_\mu=\pm1\)
   
• Fully-connected network, \(\large \color{red}N\) parameters \(\mathbf{W}\in\mathbb{R}^N\)

\(\overbrace{\phantom{x+x+x+x+x+x++2}}^{L}\)

\(\to f(\mathbf{x}; \mathbf{W})\)

\(\mathbf{x} \to\)

Training

• The goal of training is to fit all the points \(\mathbf{x}_\mu\):
  
  \[ y_\mu \cdot f(\mathbf{x}_\mu;\mathbf{W}) > 0\]
   
• \(\mathbf{W}\) is fixed by minimizing a loss function
  
  \[\mathcal{L}(\mathbf{W}) = \frac1P \sum_\mu \ell\left(y_\mu \cdot f(\mathbf{x}_\mu; \mathbf{W})\right)\]

number of patterns

under-parametrized

over-parametrized

\(\mathbf W\)

\(\mathbf W\)

\(\mathcal{L}\)

\(\mathcal{L}\)

under-parametrized

over-parametrized

(above upper bound)

Loss landscape

Jamming in spheres

• \(\large N =\) Number of degrees of freedom
• \(\large N_\Delta =\) Number of contacts (constraints)

\(N_\Delta/N = 0\)

\(N_\Delta/N \geq 1\)

Loss function

Typical choice:

• cross entropy
  
       \(\ell(yf) = \log(1+e^{yf})\)

\[\mathcal{L} = \frac1P \sum_\mu \ell\left(y_\mu f_\mu\right)\]

\(fy\)

Loss function

Let's use
 

• quadratic hinge
  
     \(\ell(yf) = \theta(\Delta) \frac12 \Delta^2\)

finite range!

same performance,

sharp jamming transition

\(fy\)

\(\Delta=1-yf\)

Dataset:
 

• \(\mathbf{x}_\mu=\) random points on the sphere,
   
• \(y_\mu=\) random labels \(\pm1\)

Location of the transition

\(P\)

 \({\large\color{red}N_\Delta} =\) number of unsatisfied constraints \(\Delta_\mu > 0\)

Sharpness of the transition

\[\mathcal{L} = \frac1{P} \sum_\mu  \theta(\Delta_\mu) \frac12 \Delta_\mu^2 \]

\(N \approx 8000\)

\(P/N\)

After minimization   \(\longrightarrow\)   net is in a minimum of \(\mathcal{L}(\mathbf{W})\)

Stability   \(\Longrightarrow\)   positive semi-definite Hessian \(\nabla^2 \mathcal{L}(\mathbf{W})\)

\[\mathcal{L} = \frac1{P} \sum_\mu \theta(\Delta_\mu)\; \frac12  \Delta_\mu^2\]

\[ \mathcal{H} = \frac1P \sum_\mu \theta(\Delta_\mu)\; \nabla\Delta_\mu\otimes\nabla\Delta_\mu + \frac1P \sum_\mu \theta(\Delta_\mu)\; \Delta_\mu\, \nabla^2 \Delta_\mu\]

\[\Delta_\mu = 1 - y_\mu f(\mathbf x_\mu; \mathbf W)\]

Spectrum of \(\mathcal{H}_p\)

\(N_- \approx N/2\)

eigenvalue

• Upper bound:   \(N < \frac{1}{C_0} \times P\)
   
• Discontinuous jump:   \(N_\Delta > C_0 \times N\)

• Stability   \(\Longrightarrow \ N_-\leq N_\Delta\leq P\)
   
• Assumption:  \(N_- = \left\{\begin{array}{ll}C_0\times N & \mathcal{L} > 0 \\ 0 & \mathcal{L} = 0\end{array} \right. \)

\(C_0 \approx 0.5\)   in all observed cases

\(\mathcal{L}=0\)

\(N_-=0\)

\(N_\Delta=0\)

\(\mathcal{L}>0\)

\(N_-= C_0\times N\)

\(N_\Delta>C_0\times N\)

\(\mathbf{x}_\mu = \) MNIST

\(y_\mu = \) parity

\(P\)

Spectrum of \(\mathcal{H}\)

Over-parametrized

\(\mathcal{H}_p=0\)

\(N_\Delta=0\)

eigenvalue

Conclusions

• Jamming transition delimits over- and under-parametrized networks
  
  
   
• In the over-parametrized phase only \(\mathcal{L}=0\) is stable

(perfect fitting)

Generalization for MNIST

\(P\)

Generalization

• There is a cusp at jamming
  
   
• Test error monotonically decreases in the over-parametrized phase

\(N\)

\(N/N_\mathrm{jamming}\)