Relative stability toward diffeomorphisms indicates performance in deep nets

Leonardo Petrini, Alessandro Favero, Mario Geiger, Matthieu Wyart

Physics of Complex Systems Lab, EPFL

= "cat"

Relative stability toward diffeomorphisms indicates performance in deep nets

Leonardo Petrini, Alessandro Favero, Mario Geiger, Matthieu Wyart

Physics of Complex Systems Lab, EPFL

Relative stability toward diffeomorphisms indicates performance in deep nets

Leonardo Petrini, Alessandro Favero, Mario Geiger, Matthieu Wyart

Physics of Complex Systems Lab, EPFL

Introduction

The puzzling success of deep learning

• Deep learning is incredibly successful in wide variety of tasks

• Curse of dimensionality when learning in high-dimension,
  in a generic setting

• Idea: there are directions in input space that the task is invariant to.
• An example is the space of image translations - that is small though.
• Space of smooth deformations is much larger - getting invariant to that can be key to escape the curse of dimensionality!
• Such invariance can be characterized by the stability:
  
   
• Hypothesis: nets perform better and better by becoming more and more stable to diffeo

Invariances give structure to data-space

1. Introduce a max-entropy distribution of diffeomorphisms of controlled magnitude

Our contribution:

Max-entropy model of diffeomorphisms

• Goal: deform images in a controlled way
   
• How to sample from a uniform distribution on all diffeomorphisms \( - \,\tau\, -\) that have the same norm \(\|\tau\|\)?
   
• Can be solved as a classical problem in physics where the norm takes the place of an energy, which can be controlled by introducing a temperature \(T\)

more deformed

1. Introduce a max-entropy distribution of diffeomorphisms of controlled magnitude
    
2. Define the relative stability toward diffeomorphisms with respect to that of a random transformation, \(R_f\)

Our contribution:

Stability to fixed-norm random transformations

Relative stability to diffeomorphisms

\(x\)       input image

\(\tau\)       smooth deformation

\(\eta\)       isotropic noise with \(\|\eta\| = \langle\|\tau x - x\|\rangle\)

\(f\)       network function

Observable that quantifies if a deep net is less sensitive to diffeomorphisms than to generic data transformations

1. Introduce a max-entropy distribution of diffeomorphisms of controlled magnitude
    
2. Define the relative stability toward diffeomorphisms with respect to that of a random transformation, \(R_f\)
    
3. Keys results: \(R_f\) is learned and indicates performance!

Our contribution:

Good deep nets learn to become relatively stable

$$R_f = \frac{\langle \|f(\tau x) - f(x)\|^2\rangle_{x, \tau}}{\langle \|f(x + \eta) - f(x)\|^2\rangle_{x, \eta}}$$

Results:

1. At initialization (shaded bars) \(R_f \approx 1\) - SOTA nets don't show stability to diffeo at initialization.
2. After training (full bars) \(R_f\) is reduced by one/two orders of magnitude consistently across datasets and SOTA architectures.
   
   
   
    
3. By contrast, (2.) doesn't hold true for fully connected and simple CNNs for which \(R_f \sim 1\) before and after training.

conclusions

future work