Sarah Dean PRO
asst prof in CS at Cornell
Contextual Bandits
ML in Feedback Sys #13
Prof Sarah Dean
Reminders
- Final project proposal due October 7
- Upcoming paper presentations starting 10/24
- Meet with Atul in advance
model
ft:X→Y
observation
prediction
Prediction in a streaming world
xt
accumulate
{(xt,yt)}
y^t
policy
πt:X→A
observation
action
xt
accumulate
{(xt,at,rt)}
at
Action in a streaming world
Linear Contextual Bandits
- for t=1,2,...
- receive context xt
- take action at∈A
- receive reward E[rt]=θ⋆⊤φ(xt,at)
Linear Contextual Bandits
Linear Contextual Bandits
- for t=1,2,...
- receive action set At
- take action φt∈At
- receive reward E[rt]=θ⋆⊤φt
Related Goals:
- choose best action: maxφ∈Atθ⋆⊤φ
- predict reward of action: r^t≈θ⋆⊤φt
- estimate reward function: θ^≈θ⋆
The prediction error (θ^t−θ⋆)⊤φ=∑k=1tεk(Vt−1φk)⊤φ∼N(0,σ2∥φ∥Vt−12φ⊤Vt−1φ)
Prediction errors
How much should I trust my predicted reward θ^⊤φ if observed rewards are corrupted by Gaussian noise?
θ^t=Vt−1k=1∑tφkrk,Vt=k=1∑tφkφk⊤
With probability 1−δ, we have ∣(θ^t−θ⋆)⊤φ∣≤ σ2log(2/δ)∥φ∥Vt−1
Correction: last lecture, expressions like ∥Mx∥2 should have instead been x⊤Mx=∥M1/2x∥2=:∥M∥V
For symmetric matrices with non-negative eigenvalues (i.e. "positive semi-definite matrices"), we know that M=VΛV⊤
So the "square root" of a PSD matrix is M1/2=Λ1/2V⊤
For a diagonal matrix D1/2=diag(λ1,…,λn)
where notation M1/2 means a matrix such that (M1/2)⊤M1/2=M
Confidence Ellipsoids
How much should I trust the estimate θ^?
Define the confidence ellipsoid Ct={θ∈Rd∣∥θ−θ^t∥Vt2≤βt}
For the right choice of βt, it is possible to guarantee θ⋆∈Ct with high probability
Exercise: For a fixed action φ, show that θ∈Ctmaxθ⊤φ≤θ^⊤φ+βt∥φ∥Vt−1
θ∈Ctminθ⊤φ≥θ^⊤φ−βt∥φ∥Vt−1
example: K=2 and we've pulled the arms 2 and 1 times respectively
Vt=[21]
(μ^1,μ^2)
Confidence set
2(μ1−μ^1)2+(μ2−μ^2)2≤βt
Pulling arm 1: μ^1±βt/2
Pulling arm 2: μ^2±βt
{[10],[10],[01]}
example: d=2 linear bandits
Vt=[3113]
θ^
Trying action a=[0,1]:
- θ^⊤a±βta⊤Vt−1a
- = θ^2±3βt/8
{[11],[−11],[−1−1]}
Vt−1=81[3−1−13]
Exercise: For fixed φ, show that best/worst case elements of Ct are given by θ=±∥a∥Vt−1βtVt−1φ
Regularization
Now we have
Confidence ellipsoid takes the same form Ct={θ∈Rd∣∥θ−θ^t∥Vt2≤βt}
θ^t=Vt−1k=1∑tφkrk,Vt=λI+k=1∑tφkφk⊤
θ^t=argθmink=1∑t(θ⊤φk−rk)2+λ∥θ∥22
To handle cases where {φk} are not full rank, we consider regularized LS
From Estimation to Action
How does estimation error affect suboptimality?
- Define the optimal action φt⋆=argmaxφ∈Atθ⋆⊤φ
- Suppose we choose φ^t=argmaxφ∈Atθ^⊤φ
The suboptimality is θ⋆⊤φt⋆−θ⋆⊤φ^t
- ≤ maxθ∈Cθ⊤φt⋆−minθ∈Cθ⊤φ^t since we don't know θ⋆
- ≤ θ^⊤φt⋆+β∥φt⋆∥V−1−(θ^⊤φ^t−β∥φ^t∥V−1) from previous slide
- ≤ ≤0θ^⊤φt⋆−θ^⊤φ^t+β(∥φt⋆∥V−1+∥φ^t∥V−1) by choice of θ^
From Estimation to Action
The suboptimality is θ⋆⊤φt⋆−θ⋆⊤φ^t
- ≤β(∥φt⋆∥V−1+∥φ^t∥V−1)
- ≤2βmaxφ∈Atφ⊤V−1φ
This perspective motivates techniques in experiment design
- e.g. select {φk} to ensure that V has a large minimum eigenvalue (so V−1 will have small eigenvalues)
When {φk}k=1N are chosen at random from "nice" distribution
∥V−1∥=λmin(V)1≲Nd with high probability.
From Estimation to Action
Then with high probability, the suboptimality
- θ⋆⊤φ⋆−θ⋆⊤φ^≲2BβtNd
Informal Theorem: Let the norm of all φ∈At be bounded by B for all t. Suppose that {φk}k=1N are chosen at random from "nice" distribution.
Cumulative sub-optimality: regret
For a fixed interaction horizon T, how to trade-off exploration and exploitation?
Design algorithms with low regret R(T)=t=1∑Ta∈Amaxr(xt,a)−r(xt,at)
R(T)=t=1∑Tφ∈Atmaxθ⋆⊤φ−θ⋆⊤φt
Explore-then-commit
ETC
- For t=1,…,N
- play φt at random
- Estimate θ^ with least squares
- For t=N+1,…,T
- play φ^t=argmaxφ∈Atθ^⊤φ
The regret has two components
R(T)=R1t=1∑N φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt+R2t=N+1∑T φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt
Explore-then-commit
The regret has two components
R(T)=R1t=1∑N φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt+R2t=N+1∑T φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt
Suppose that B bounds the norm of φ and ∥θ⋆∥≤1.
- θ⋆⊤φ⋆−θ⋆⊤φ≤2B
Then we have R1≤2BN
Explore-then-commit
The regret has two components
R(T)=R1t=1∑N φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt+R2t=N+1∑T φ∈Atmaxθ⋆⊤φ−θ⋆⊤φt
Using sub-optimality result, with high probability R2≲(T−N)2BβNd
Suppose that maxtβt≤β.
Explore-then-commit
The regret is bounded with high probability by
R(T)≲ 2BN+ 2BTβNd
Choosing N=T2/3 leads to sublinear regret
- R(T)≲T2/3
Upper Confidence Bound
Adaptive perspective: optimism in the face of uncertainty
Instead of exploring randomly, focus exploration on promising actions
UCB
- Initialize V0=λI, b0=0
- For t=1,…,T
- play φt=argφ∈Atmaxθ∈Ct−1maxθ⊤φ
- update Vt=Vt−1+φtφt⊤
and bt=bt−1+rtφt
- θ^t=Vt−1bt
- Ct={∥θ−θ^∥Vt≤βt}
Sub-optimality of optimism
- Define the optimal action φt⋆=argmaxφ∈Atθ⋆⊤φ
- Suppose we choose optimistic φ^t=argmaxφ∈Atmaxθ∈Ctθ⊤φ
- let θ~t=argmaxθ∈Ctθ⊤φ^t
The suboptimality is θ⋆⊤φt⋆−θ⋆⊤φ^t
- ≤ θ~t⊤φ^t−θ⋆⊤φ^t by choice of φ^t
- = (θ~t−θ⋆)⊤φ^t
- =(Vt−11/2(θ~t−θ⋆))⊤(Vt−1−1/2φ^t)
- ≤∥θ~t−θ⋆∥Vt−1∥φ^t∥Vt−1−1 by Cauchy-Schwarz
- ≤2βt−1∥φ^t∥Vt−1−1 using definition of confidence interval
Upper Confidence Bound
Proof Sketch: R(T)=∑t=1Tθ⋆⊤φt⋆−θ⋆⊤φ^t
- ≤2β∑t=1T∥φ^t∥Vt−1−1 from previous slide
- ≤2Tβ∑t=1T∥φ^t∥Vt−1−12 by Cauchy-Schwarz
- ≲T following Lemma 19.4 in Bandit Algorithms
The regret is bounded with high probability by
R(T)≲ T
After fall break: action in a dynamical world (optimal control)
Recap
- Linear contextual bandits
- Confidence bounds
- Sub-optimality & regret
- ETC (T2/3) and UCB (T)
Reference: Ch 19-20 in Bandit Algorithms by Lattimore & Szepesvari
Contextual Bandits ML in Feedback Sys #13 Prof Sarah Dean
13 - Bandits - ML in Feedback Sys
By Sarah Dean
