A common notion of uncertainty

A common notion of uncertainty

What does this measure of uncertainty tell us?

1. Regions of the reconstruction with higher variance

2. Range of where a new sample will be

3. How far the ground-truth image is

Our setting

1. The ground truth image

$$X \sim p(X)$$

2. The observation process

$$Y = X + v,~v \sim \mathcal{N}(0, \sigma^2 \mathbb{I})$$

3. The sampling procedure

$$Z = f(Y) \sim \mathcal{Q}_Y \approx p(X \mid Y)$$

Three sources of randomness

Our setting

original

perturbed

sampled

Question

Fix a pair $(x, y)$, and sample $m$ times from $f(y)$,

where will the $(m + 1)$-th sample fall?

Conformal predictions

Coverage [Shafer and Vovk, 2008][Angelopoulos and Bates, 2022]

$$\mathbb{P}\left[Y_{\text{test}} \in \mathcal{C}(X_{\text{test}})\right] \geq 1 - \alpha$$

In our case, entrywise coverage

$$\forall j \in [d],~\mathbb{P}\left[f(y)_j \in \mathcal{I}(y)_j\right] \geq 1 - \alpha$$

Theorem (Entrywise calibrated quantiles) Let $\hat{l}_{\alpha},\hat{u}_{\alpha}$ be the $\lfloor (m+1) \alpha/2 \rfloor / m$ and $\lceil (m+1)(1 - \alpha/2) \rceil / m$ empirical quantiles of $f(y)_j$ over $m$ i.i.d. samples. Then,

$$\mathcal{I}(y)_j = [\hat{l}_{\alpha}, \hat{u}_{\alpha}]$$

provides entrywise coverage

Entrywise calibrated quantiles

original

$\hat{l}_{\alpha}$

$\hat{u}_{\alpha}$

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

Entrywise calibrated quantiles

original

$\hat{l}_{\alpha}$

$\hat{u}_{\alpha}$

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

Comparison of standard deviation with entrywise calibrated quantiles

Calibrated quantiles distribution

$Q_1\\ \Delta_{\text{HU}} \leq 7$

$Q_2\\\Delta_{\text{HU}} \in (7, 8]$

$Q_3\\\Delta_{\text{HU}} \in (8, 55]$

$Q_4\\\Delta_{\text{HU}} > 55$

original

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

Calibrated quantiles distribution

original

$Q_1\\\Delta_{\text{HU}} \leq 7$

$Q_2\\\Delta_{\text{HU}} \in (7, 8]$

$Q_3\\\Delta_{\text{HU}} \in (8, 9]$

$Q_4\\\Delta_{\text{HU}} > 9$

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

Calibrated quantiles distribution

original

$Q_3\\\Delta_{\text{HU}} \in (8, 34]$

$Q_4\\\Delta_{\text{HU}} > 34$

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

$Q_1\\\Delta_{\text{HU}} \leq 7$

$Q_2\\\Delta_{\text{HU}} \in (7, 8]$

Calibrated quantiles distribution

original

$Q_3\\\Delta_{\text{HU}} \in (8, 44]$

$Q_4\\\Delta_{\text{HU}} > 44$

$\hat{u}_{\alpha} - \hat{l}_{\alpha}$

$Q_1\\\Delta_{\text{HU}} \leq 7$

$Q_2\\\Delta_{\text{HU}} \in (7, 8]$

Calibrated quantiles

What does this measure of uncertainty tell us?

1. Regions of the reconstruction with higher variance

2. Range of where a new sample will be

3. How far the ground-truth image is

Risk control [Bates, 21]

For a pair $(x, y)$ define

$$\ell(x, \mathcal{I}(y)) = \frac{\#\{j \in [d]:~x_j \notin \mathcal{I}(y)_j\}}{d}$$

Definition (Risk-Controlling Prediction Set) A random set predictor $\mathcal{I}:~\mathcal{Y} \to \mathcal{P}(\mathcal{X})$ is an $(\epsilon,~\delta)$-RCPS if

$$\mathbb{P}\left[\mathbb{E}_{(X, Y)}\left[\ell(X, \mathcal{I}(Y)\right] \leq \epsilon\right] \geq 1-\delta$$

Image-to-Image Regression [Angelopolous, 22]