Uncertainty in Neural Networks
Methods and Applications
Berkeley Statistics and Machine Learning Forum
What uncertainties are we talking about?
A Motivating Example: Probabilistic Regression
From this excellent tutorial: https://medium.com/tensorflow/regression-with-probabilistic-layers-in-tensorflow-probability-e46ff5d37baf
- Linear regression
- Aleatoric Uncertainties
- Epistemic Uncertainties
- Epistemic+ Aleatoric Uncertainties
y^=ax
\hat{y} = a x
y^∼N(ax,σ2)
\hat{y} \sim \mathcal{N}(a x, \sigma^2)
y^=wxw∼p(w∣{xi,yi})
\hat{y} = w x \quad w \sim p(w | \{x_i, y_i\})
y^∼N(wx,σ2)w,σ∼p(w,σ∣{xi,yi})
\hat{y} \sim \mathcal{N}(w x, \sigma^2) \\ w, \sigma \sim p(w, \sigma | \{x_i, y_i\})
Quiz: Which uncertainty dominates?
Predicting cluster masses from velocity dispersion
Bayesian Neural Networks
As tools to model Epistemic Uncertainties
Bayes by Backprop
Weight Uncertainty in Neural Networks
Blundel et al. 2015
Beyond point estimates of weights
Maximum likelihood:
Bayesian Posterior by Variational Inference:
Practical Algorithm
Test on MNIST
Regression Example
MC Dropout
Dropout as a Bayesian Approximation:
Representing Model Uncertainty in Deep Learning
Gal & Ghahramani, 2015
What is dropout?
Hinton 2012, Srivastava 2014
The idea
Let's express the predictive probability of the model:
Parameterize q(w) in the following way:
Experiments
Example of applications
Image Segmentation: https://arxiv.org/abs/1703.04977
Regression from images: https://arxiv.org/abs/1708.08843
Time Series Classification: https://arxiv.org/abs/1901.06384
Some other methods
- Noise contrastive priors: https://arxiv.org/abs/1807.09289
Uncertainty in Neural Networks Methods and Applications Berkeley Statistics and Machine Learning Forum