Uncertainty in Neural Networks
Methods and Applications

Berkeley Statistics and Machine Learning Forum

What uncertainties are we talking about?

A Motivating Example: Probabilistic Regression

  • Linear regression
  • Aleatoric Uncertainties
  • Epistemic Uncertainties
  • Epistemic+ Aleatoric Uncertainties
\hat{y} = a x
\hat{y} \sim \mathcal{N}(a x, \sigma^2)
\hat{y} = w x \quad w \sim p(w | \{x_i, y_i\})
\hat{y} \sim \mathcal{N}(w x, \sigma^2) \\ w, \sigma \sim p(w, \sigma | \{x_i, y_i\})

Quiz: Which uncertainty dominates?

Predicting cluster masses from velocity dispersion


Bayesian Neural Networks

As tools to model Epistemic Uncertainties

Bayes by Backprop

Weight Uncertainty in Neural Networks
Blundel et al. 2015


Beyond point estimates of weights 

Maximum likelihood:

Bayesian Posterior by Variational Inference:

Practical Algorithm

Test on MNIST

Regression Example

MC  Dropout

Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning
Gal & Ghahramani, 2015



What is dropout?

Hinton 2012, Srivastava 2014

The idea

Let's express the predictive probability of the model:

Parameterize q(w) in the following way:


Example of applications

Image Segmentation: https://arxiv.org/abs/1703.04977

Regression from images: https://arxiv.org/abs/1708.08843
Time Series Classification: https://arxiv.org/abs/1901.06384

Some other methods

Bayesian Neural Networks

By eiffl

Bayesian Neural Networks

Session on modeling uncertainties with neural networks, for the Berkeley Statistics and Machine Learning Forum

  • 885