Contextual Bandits

ML in Feedback Sys #11

Fall 2025, Prof Sarah Dean

policy

\(\pi_t:\mathcal X\to\mathcal A\)

observation

action

\(x_t\)

accumulate

\(\{(x_t, a_t, r_t)\}\)

\(a_{t}\)

Action in a streaming world

Contextual bandits

"What we do"

• Initialize a model \(\hat r(x, a) = \hat\theta_0^\top \varphi(x,a)\) and \(V_0=\lambda I\)
• For \(t=1,2,...\)
  • receive context \(x_t\)
  • take action \(a_t\) according to $$a_t = \arg\max_{a\in\mathcal A} \hat\theta_{t-1}^\top \varphi(x_t,a) + \sqrt{\beta_t}\|\varphi(x_t,a)\|_{V_{t-1}^{-1}} $$
  • receive reward \(r_t\)
  • update model \(\hat\theta_{t}\) and confidence parameters \(V_{t}\) according to least squares with \(((x_t,a_t),r_t)\)
• This algorithm is called LinUCB (Upper Confidence Bound)

Contextual bandits

"Why we do it"

• Fact 1: The linear reward model assumption can represent many settings $$\mathbb E[r_t|x_t,a_t] = r(x_t,a_t) = \theta_\star^\top \varphi(x_t,a_t)$$
• Fact 2: As long as rewards are bounded and linear, then with high probability \(\|\theta_\star-\hat\theta_t\|_{V_{t-1}}^2\leq \beta_t \) and  $$\hat\theta_t^\top \varphi(x_t,a) + \sqrt{\beta_t}\|\varphi(x_t,a)\|_{V_{t-1}^{-1}} = \max_{\|\theta-\hat\theta_t\|_{V_{t-1}}^2\leq \beta_t} \theta^\top \varphi(x_t,a)$$
• Fact 3: Under the assumptions above, the average reward of actions chosen by LinUCB converges to optimal for \(C\) depending only logarithmically on \(T\) $$ \frac{1}{T}\sum_{t=1}^T r(x_t,a_t) \geq \frac{1}{T}\sum_{t=1}^T \max_{a\in\mathcal A} r(x_t, a)  - \frac{C}{\sqrt{T}}$$

Linear reward model

Taking action \(a_t\in\mathcal A\) in context \(x_t\) yields reward $$r_t = \langle\theta_\star, \varphi(x_t, a_t)\rangle + \varepsilon_t,\qquad \mathbb E[\varepsilon_t]=0,~~\mathbb E[\varepsilon_t^2] = \sigma^2$$

Taking action \(a_t\in\mathcal A\) in context \(x_t\) yields reward $$r_t = \langle\theta_\star, \varphi(x_t, a_t)\rangle + \varepsilon_t,\qquad \mathbb E[\varepsilon_t]=0,~~\mathbb E[\varepsilon_t^2] = \sigma^2$$

Linear reward model

• \(K\) armed bandits
  • \(\mathbb E[r(a)] = \begin{bmatrix} \mu_1\\\vdots\\ \mu_K\end{bmatrix}^\top \begin{bmatrix} \mathbf 1\{a=1\}\\\vdots\\ \mathbf 1\{a=K\}\end{bmatrix} \)
    

Taking action \(a_t\in\mathcal A\) in context \(x_t\) yields reward $$r_t = \langle\theta_\star, \varphi(x_t, a_t)\rangle + \varepsilon_t$$

• \(K\) armed bandits
  • \(\varphi(a) = e_a\)
• \(K\) armed contextual linear bandits
  • \(\mathbb E[r(x, a)] = \begin{bmatrix} \theta_1 \\\vdots\\ \theta_K \end{bmatrix}^\top \begin{bmatrix} x\mathbf 1\{a=1\}\\\vdots\\ x\mathbf 1\{a=K\}\end{bmatrix} \)

Linear reward model

Taking action \(a_t\in\mathcal A\) in context \(x_t\) yields reward $$r_t = \langle\theta_\star, \varphi(x_t, a_t)\rangle + \varepsilon_t$$

• \(K\) armed bandits
  • \(\varphi(a) = e_a\)
• \(K\) armed contextual linear bandits
  • \(\varphi(x,a) = e_a \otimes x\)
• Linear bandits
  • \(\mathbb E[r(a)] = \theta_\star^\top a\)

Linear reward model

Taking action \(a_t\in\mathcal A\) in context \(x_t\) yields reward $$r_t = \langle\theta_\star, \varphi(x_t, a_t)\rangle + \varepsilon_t$$

• \(K\) armed bandits
  • \(\varphi(a) = e_a\)
• \(K\) armed contextual linear bandits
  • \(\varphi(x, a) = e_a \otimes x\)
• Linear bandits
  • \(\varphi(a)=a\)
• Linear contextual bandits
  •  \(\mathbb E[r(x, a)] = ( \Theta_\star x)^\top a\)
    • = \(\mathrm{vec}(\Theta_\star)^\top \mathrm{vec}(ax^\top)\) so \(\varphi(x, a)= \mathrm{vec}(ax^\top)\)

Linear reward model

Let \(\varphi_t = \varphi(x_t,a_t)\), then using data \(\{(\varphi_k, r_k)\}_{k=1}^t\)

Recap: least squares estimation

Confidence ellipsoid

Last lecture we discussed \(1-\delta\) confidence interval on prediction \(\theta_\star^\top\varphi\) is $$\Big[\hat\theta_t^\top \varphi \pm  \sqrt{\beta} \|\varphi\|_{V_t^{-1}}\Big]$$

example: \(K=2\) and we've pulled the arms 2 and 1 times respectively

\(\hat\theta = (\hat \mu_1,\hat\mu_2)\)

• Confidence ellipsoid $$\{ \|\theta-\hat\theta_t\|_{V_t}^2\leq \beta_t\} $$

  • \(2(\mu_1-\hat\mu_1)^2 + (\mu_2-\hat\mu_2)^2 \leq \beta_t\)

• Confidence interval $$\big[\hat\theta_t^\top \varphi \pm  \sqrt{\beta} \|\varphi\|_{V_t^{-1}}\big]$$
  • Pulling arm 1: \(\hat \mu_1 \pm \sqrt{\beta_t/2}\)
  • Pulling arm 2:
    • \(\hat \mu_2 \pm \sqrt{\beta_t}\)

Example:

example: \(d=2\) linear bandits

\(\hat \theta\)

$$ \left\{\begin{bmatrix} 1\\ 1\end{bmatrix}, \begin{bmatrix} -1\\ 1\end{bmatrix}, \begin{bmatrix} -1\\ -1\end{bmatrix}\right \} $$

Example:

• Confidence ellipsoid $$\{ \|\theta-\hat\theta_t\|_{V_t}^2\leq \beta_t\} $$
• Confidence interval $$\big[\hat\theta_t^\top \varphi \pm  \sqrt{\beta} \|\varphi\|_{V_t^{-1}}\big]$$

  • Trying action \(a=[0,1]\):

  • \(\hat \theta^\top a\pm \sqrt{\beta_t a^\top V_t^{-1}a}\)
    • = \(\hat \theta_2 \pm \sqrt{3\beta_t/8}\)

Regularization

$$\hat\theta_t = \arg\min_\theta \sum_{k=1}^t (\theta^\top \varphi_k - r_k)^2 + \lambda\|\theta\|_2^2$$

To handle cases where \(\{\varphi_k\}\) are not full rank, we consider regularized LS

Reference: Ch 19-20 in Bandit Algorithms by Lattimore & Szepesvari

Upper Confidence Bound

Adaptive perspective: optimism in the face of uncertainty

• focus exploration on promising actions

Sub-optimality of optimism

• Define the optimal action \(a_t^\star = \arg\max_{a\in\mathcal A} \theta_\star^\top \varphi(x_t,a)\)
• Suppose we choose optimistic \(\hat a_t = \arg\max_{a\in\mathcal A} \max_{\theta\in\mathcal C_t} \theta^\top \varphi(x_t,a)\)
  • let \(\tilde\theta_t = \arg\max_{\theta\in\mathcal C_t} \theta^\top \hat\varphi(x_t,a),\varphi_t^\star =\varphi(x_t,a^\star_t), \hat\varphi_t=\varphi(x_t,\hat a_t)\)

Upper Confidence Bound

Prediction vs. Action

• Data format: labels \(y\in\mathcal Y\) vs. actions \(a\in\mathcal A\) and rewards \(r\in\mathbb R\) $$ \{(x_t, y_t)\}_{t=1}^T  \quad \text{vs.}\quad \{(x_t, a_t, r_t )\}_{t=1}^T $$
• Observations \(x_t\) previously called features now called contexts
• Goal: predict \(\hat y\) with high accuracy vs. chose \(a_t\) for high reward $$\min_{\hat y \in \mathcal Y} \mathbb E[\ell(y, \hat y)\mid x]\quad \text{vs.} \quad \max_{a\in\mathcal A} \mathbb E[r_t\mid x_t, a_t]$$
• Key difference: dataset is no longer fixed, but depends on our actions
• Today's lecture: assumed that reward depends only on current context and action (not past or future)

Recap

• Linear contextual bandits
• Confidence bounds
• Sub-optimality & regret
• UCB Algorithm