Uncertainty in Neural Networks
Methods and Applications

Berkeley Statistics and Machine Learning Forum

What uncertainties are we talking about?

A Motivating Example: Probabilistic Regression

From this excellent tutorial: https://medium.com/tensorflow/regression-with-probabilistic-layers-in-tensorflow-probability-e46ff5d37baf

  • Linear regression
     
  • Aleatoric Uncertainties
     
  • Epistemic Uncertainties
     
  • Epistemic+ Aleatoric Uncertainties
\hat{y} = a x
\hat{y} \sim \mathcal{N}(a x, \sigma^2)
\hat{y} = w x \quad w \sim p(w | \{x_i, y_i\})
\hat{y} \sim \mathcal{N}(w x, \sigma^2) \\ w, \sigma \sim p(w, \sigma | \{x_i, y_i\})

Quiz: Which uncertainty dominates?

Predicting cluster masses from velocity dispersion

 

Bayesian Neural Networks

As tools to model Epistemic Uncertainties

Bayes by Backprop

Weight Uncertainty in Neural Networks
Blundel et al. 2015

https://arxiv.org/abs/1505.05424

Beyond point estimates of weights 

Maximum likelihood:

Bayesian Posterior by Variational Inference:

Practical Algorithm

Test on MNIST

Regression Example

MC  Dropout

Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning
Gal & Ghahramani, 2015

https://arxiv.org/abs/1506.02142

https://arxiv.org/abs/1506.02157

What is dropout?

Hinton 2012, Srivastava 2014

The idea

Let's express the predictive probability of the model:

Parameterize q(w) in the following way:

Experiments

Example of applications

Image Segmentation: https://arxiv.org/abs/1703.04977

Regression from images: https://arxiv.org/abs/1708.08843
Time Series Classification: https://arxiv.org/abs/1901.06384

Some other methods