Uncertainty in Neural Networks
Methods and Applications

Berkeley Statistics and Machine Learning Forum
What uncertainties are we talking about?
A Motivating Example: Probabilistic Regression
From this excellent tutorial: https://medium.com/tensorflow/regression-with-probabilistic-layers-in-tensorflow-probability-e46ff5d37baf




- Linear regression
- Aleatoric Uncertainties
- Epistemic Uncertainties
- Epistemic+ Aleatoric Uncertainties
\hat{y} = a x
\hat{y} \sim \mathcal{N}(a x, \sigma^2)
\hat{y} = w x \quad w \sim p(w | \{x_i, y_i\})
\hat{y} \sim \mathcal{N}(w x, \sigma^2) \\ w, \sigma \sim p(w, \sigma | \{x_i, y_i\})
Quiz: Which uncertainty dominates?
Predicting cluster masses from velocity dispersion

Bayesian Neural Networks
As tools to model Epistemic Uncertainties
Bayes by Backprop
Weight Uncertainty in Neural Networks
Blundel et al. 2015
Beyond point estimates of weights



Maximum likelihood:
Bayesian Posterior by Variational Inference:

Practical Algorithm

Test on MNIST


Regression Example

MC Dropout
Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning
Gal & Ghahramani, 2015
What is dropout?
Hinton 2012, Srivastava 2014


The idea

Let's express the predictive probability of the model:

Parameterize q(w) in the following way:

Experiments



Example of applications
Image Segmentation: https://arxiv.org/abs/1703.04977
Regression from images: https://arxiv.org/abs/1708.08843
Time Series Classification: https://arxiv.org/abs/1901.06384
Some other methods
- Noise contrastive priors: https://arxiv.org/abs/1807.09289

Bayesian Neural Networks
By eiffl
Bayesian Neural Networks
Session on modeling uncertainties with neural networks, for the Berkeley Statistics and Machine Learning Forum
- 1,083