Active Learning Tutorial
Ethan K. Gordon
June 26, 2026
What? Why? How?
Active Exploration
Choosing actions to take in the environment with the goal of gaining useful information.
Online Learning
Using data collected at test time to permanently update:
- Policy
- Model
- Reward Distribution, etc.
What is Active Learning?
+
Example: OL w/o AE
Any Passive Observation
Many Greedy RL / Bandit Algorithms
Out-of-scope: pre-training / offline learning; treated as a prior
Example: AE w/o OL
Policy with History \(\pi_\theta(a|s_{[t-H,t]})\)
Choose actions to create a "good" context, but don't update \(\pi\)
Today's Focus
Why Active Learning?
Just collect enough data for the model / policy / etc. to be perfect in advance!
- Out-of-distribution, covariate shift still exists (for now)
- Distributions are non-stationary: what if you train on the universe and then the universe expands?
Much stronger argument for Active Exploration specifically:
- Test-Time Epistemic Uncertainty is unavoidable, example
True Distribution:
70% Aggressive
30% Passive
(training learns this)
Aleatoric uncertainty at training time:
inherent to the system. No training will tell us about any specific driver.
Epistemic uncertainty at test time:
we can collect info about the driver (e.g. inch forward) to reduce/eliminate it.
Real Life is a POMDP
What is "information", formally?
AE's goal is to choose actions that gain information.
My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'
- Claude Shannon
(Shannon) Entropy:
\(H[X] := \mathbb{E}_{p(X)}[-\log p(X)]\)
Thinking About Entropy
It's tricky: entropy of distributions, differential entropy, entropy of messages, thermodynamic entropy, etc.
Opinion for AE:
It is best to think of entropy = information \(\approx\) uncertainty.
Consider the driving example again.
- p = 0.5, maximum uncertainty. You need 1 bit of information to tell you about a given driver.
- p = 0 or p=1, minimum uncertainty. You need 0 bits of information to tell you about a given driver.
Making samples more certain -> reducing entropy.
\(H[X] := \mathbb{E}_{p(X)}[-\log p(X)]\)
Exercise: Entropy of a Gaussian
\(p(x \sim \mathcal{N}(\mu, \Sigma)) = \frac{C}{\sqrt{\det\Sigma}}\exp(C||x-\mu||^2_{\Sigma^{-1}})\)
\(H[X] := \mathbb{E}_{p(X)}[-\log p(X)]\)
\(H[X] = C + \frac{1}{2}\log\det\Sigma\)
Entropy \(\approx\) Uncertainty
Computing Information Gain
Choose an action \(a\)
Collect Data \(\mathcal{D} \sim p(\mathcal{D}|a, \theta^*)\)
\(H[\Theta]\)
\(H[\Theta | \mathcal{D}]\)
Expected Information Gain: \(\mathbb{E}_{\mathcal{D}}(H[\Theta] - H[\Theta | \mathcal{D}]) := \mathbb{E}_{\mathcal{D}}I(\Theta; \mathcal{D})\)
Other names: Expected Mutual Information, "Bayesian Active Learning by Disagreement" (BALD)
Computing Information Gain
How do we actually compute this!?
For Gaussian's, it's not bad!
Gaussian Prior, Gaussian Likelihood, Gaussian Posterior
Everything has a log, so we can generally ignore the normalization constant over the dataset.
Dropping constants:
\(I = H[\Theta] - H[\Theta | \mathcal{D}] = \log\det\Sigma_\Theta - \log\det\Sigma_{\Theta|\mathcal{D}} = \log\det(\Sigma_\Theta\Sigma^{-1}_{\Theta|\mathcal{D}})\)
Note: for an "uninformative" prior, we can just try an minimize the entropy of the posterior.
Exercise: Linear Regression
\(p(\mathcal{D} = \{Y, X\} | \Theta) \propto \exp(||X\theta-Y||^2_2)\)
\(y = x^T\theta + \epsilon\)
\(\epsilon \sim \mathcal{N}(0, 1)\)
Uninformative Prior: MLE = MAP
\(\tilde\theta = (X^TX)^{-1}X^TY\)
Write out the posterior:
\(p(\Theta | \mathcal{D}) \propto \exp(||\theta - (X^TX)^{-1}X^TY||^2_{X^TX})\)
\(\Sigma = (X^TX)^{-1}\)
\(\min\log\det(X^TX)^{-1} \rightarrow \max\log\det(X^TX)\)
Fisher Information, Part 1
\(\Sigma = (X^TX)^{-1}\)
\(\min\log\det(X^TX)^{-1} \rightarrow \max\log\det(X^TX)\)
"Fisher Information Matrix" of a Gaussian: \(\Sigma^{-1}\)
Not to be confused with entropy. We want to maximize Fisher Info in order to minimize uncertainty.
Important point: we can control this!
Gaussian Hessian "Trick"
\(H[\Theta] \propto -\log\det H''[\theta^*]\)
(Hessian evaluated at the peak \(\theta^*\))
Upshot: we can do a Gaussian approximation of any distribution (about one mode)
Fisher Information Matrix, the Actual Definition
(Sorry, didn't finish my slides, will have to move to the whiteboard from here.)