Efficient and Optimal Fixed-Time Regret with Two Experts  

Research Proficiency Exam

Victor Sanches Portella

PhD Student in Computer Science - UBC

October, 2020

The Two-Experts' Problem

Prediction with Expert Advice

Player

Adversary

\(n\) Experts

0.5

0.1

0.3

0.1

1

0

0.5

0.3

Probabilities

Costs

x_t
\ell_t

Player's loss:

\langle \ell_t, x_t \rangle

Goal:  sublinear regret in the worst-case

\displaystyle \mathrm{Regret}(T) = \sum_{t = 1}^T \langle \ell_t, x_t \rangle - \min_{i = 1, \dotsc, n} \sum_{t = 1}^T \ell_t(i)

Known Results

Multiplicative Weights Update method:

\displaystyle \mathrm{Regret}(T) \leq \sqrt{\frac{T}{2} \ln n}

Optimal for \(n,T \to \infty\) !

If \(n\) is fixed, we can do better

\(n = 2\)

\(n = 3\)

\(n = 4\)

\sqrt{\frac{T}{2\pi}} + O(1)
\sqrt{\frac{8T}{9\pi}} + O(\ln T)
\sim \sqrt{\frac{T \pi}{8}}

Player knows \(T\) !

Minmax regret in some cases:

What if \(T\) is not known?

\displaystyle \frac{\gamma}{2} \sqrt{T}

Minmax regret

\(n = 2\)

[Harvey, Liaw, Perkins, Randhawa FOCS 2020]

They give an efficient algorithm!

\displaystyle \gamma \approx 1.307

Known Results

Multiplicative Weights Update method:

\displaystyle \mathrm{Regret}(T) \leq \sqrt{\frac{T}{2} \ln n}

Optimal for \(n,T \to \infty\) !

If \(n\) is fixed, we can do better

\sqrt{\frac{T}{2\pi}} + O(1)

Minmax regret for 2 experts:

[Harvey, Liaw, Perkins, Randhawa FOCS 2020]

\(O(1)\) time per round

[Cover '67]

Player knows \(T\) (fixed-time)

Player doesn't know \(T\) (anytime)

\displaystyle \frac{\gamma}{2} \sqrt{T}
\displaystyle \gamma \approx 1.307

\(O(T)\) time per round

Dynamic Programming

Stochastic Calculus

[Greenstreet]

Our results:

A complete theoretical analysis of the fixed-time algorithm

The case of 2 Experts

Player knows \(T\) (fixed-time)

Player doesn't know \(T\) (anytime)

\(O(1)\) time per round

\(O(T)\) time per round

Stochastic calculus and discretization techniques

Dynamic programming

?

[Greenstreet]

\(O(1)\) time per round

[Harvey et al. '20]

\frac{\gamma}{2} \sqrt{T}

minmax regret

\sqrt{\frac{T}{2\pi}} + O(1)

minmax regret

[Cover '67]

Our results:

An efficient and optimal algorithm for two experts

regret

\sqrt{\frac{T}{2\pi}} + O(1)

Gaps and Cover's Algorithm

Binary costs

We will consider only 0 or 1 costs (no fractional costs!) Enough for the worst case

1

0

0

1

0

0

1

1

Equal costs are a "waste of time", so we do not consider those

Cover's algorithm strongly relies on these assumptions

Gap between experts

Thought experiment: how much probability mass to put on each expert?

Cumulative Loss on round \(t\)

\(\frac{1}{2}\) is both cases seems reasonable!

Takeaway: player's decision could depend on the gap between experts

Gap = |42 - 20| = 22

Lagging Expert

Leading Expert

42

20

2

2

42

42

Cover's Dynamic Program

Path-independent player:

Choice on round \(t\) depends only on the gap \(g_{t-1}\) of round \(t-1\)

Choice doesn't depend on the specific past costs

Path-independent player \(\implies\)

 \(V_p[t,g]\) depends only on \(\ell_{t+1}, \dotsc, \ell_T\) and \(g_t, \dotsc, g_{T}\)

\displaystyle V_p[0, 0] =

Maximum regret of \(p\)

p(t, g_{t-1})

on the Lagging expert

1 - p(t, g_{t-1})

on the Leading expert

 We can compute \(V_p\) backwards in time!

We then choose \(p^*\) that minimizes \(V^*[0,0] = V_{p^*}[0,0]\)

\displaystyle V_p[t, g] =

Maximum regret-to-be-suffered on rounds \(t+1, \dotsc, T\) if gap at round \(t\) is \(g\)

Regret and Player in terms of the Gap

Path-independent player:

If

round \(t\) and gap \(g_{t-1}\) on round \(t-1\)

p(t, g_{t-1})
1 - p(t, g_{t-1})

on the Lagging expert

on the Leading expert

Choice doesn't depend on the specific past costs

p(t, 0) = 1/2

for all \(t\), then

\displaystyle \mathrm{Regret(T)} = \sum_{t = 1}^{T} p(t, g_{t-1})(g_t - g_{t-1})

gap on round \(t\)

A discrete analogue of a Riemann-Stieltjes integral

A formula for the regret

A Dynamic Programming View

\displaystyle V_p[t, g] =

Maximum regret-to-be-suffered on rounds \(t+1, \dotsc, T\) when gap on round \(t\) is \(g\)

Path-independent player \(\implies\) \(V_p[t,g]\) depends only on \(\ell_{t+1}, \dotsc, \ell_T\) and \(g_t, \dotsc, g_{T}\)

\displaystyle V_p[t, 0] = \max\{p(t+1,0), 1 - p(t+1,0)\} + V_p[t+1, 1]

Regret suffered on round \(t+1\)

Regret suffered on round \(t + 1\)

\displaystyle V_p[t, g] = \max \Bigg\{
\displaystyle V_p[t+1, g+1] + p(t + 1,g)
\displaystyle V_p[t+1, g-1] - p(t + 1,g)

A Dynamic Programming View

\displaystyle V_p[t, g] =

Maximum regret-to-be-suffered on rounds \(t+1, \dotsc, T\) if gap at round \(t\) is \(g\)

 We can compute \(V_p\) backwards in time!

Path-independent player \(\implies\)

 \(V_p[t,g]\) depends only on \(\ell_{t+1}, \dotsc, \ell_T\) and \(g_t, \dotsc, g_{T}\)

We then choose \(p^*\) that minimizes \(V^*[0,0] = V_{p^*}[0,0]\)

\displaystyle V_p[0, 0] =

Maximum regret of \(p\)

A Dynamic Programming View

For \(g > 0\)

\displaystyle p^*(t,g) = \frac{1}{2}(V_{p^*}[t, g-1] - V_{p^*}[t, g + 1])

Optimal player

\displaystyle p^*(t,0) = \frac{1}{2}

Optimal regret (\(V^* = V_{p^*}\))

\displaystyle V^*[t,g] = \frac{1}{2}(V^*[t+1, g-1] + V^*[t+1, g + 1])

For \(g = 0\)

\displaystyle V^*[t,0] = \frac{1}{2} + V^*[t+1, 1]

For \(g > 0\)

For \(g = 0\)

A Dynamic Programming View

Optimal regret (\(V^* = V_{p^*}\))

\displaystyle V^*[t,g] = \frac{1}{2}(V^*[t+1, g-1] + V^*[t+1, g + 1])
\displaystyle V^*(t,0) = \frac{1}{2} + V^*[t+1, 1]

For \(g > 0\)

For \(g = 0\)

g
t
4
3
2
1
0
0
0
0
0
0
0
0
\frac{1}{2}
0
0
0
0
0
0
0
0
0
0
0
0
0
1
2
3

Connection to Random Walks

\displaystyle V^*[0,0] =

Maximum regret of \(p^*\)

Theorem

\displaystyle V^*[0,0] =

Expected # of 0's of a Sym. Random Walk of Length \(T\)

\displaystyle \frac{1}{2}
\displaystyle \Big(
\displaystyle \Big)
\displaystyle [\tfrac{2}{5}, 1] + \sqrt{\frac{2T}{\pi}} =

Theorem

For any player, if the gaps are random and distributed like a reflected symmetric random walk,

\displaystyle \mathbb{E}[\mathrm{Regret}(T)] \geq

Expected # of 0's of a SRW of Length \(T - 1\)

\displaystyle \frac{1}{2}
\displaystyle \Big(
\displaystyle \Big)

Continuous Regret

A Probabilistic View of Regret Bounds

Formula for the regret based on the gaps

\displaystyle \mathrm{Regret(T)} = \sum_{t = 1}^{T} p(t, g_{t-1})(g_t - g_{t-1})

Discrete stochastic integral of \(p\) with respect to the reflected RW \(g\)

Moving to continuous time:

Random walk \(\longrightarrow\) Brownian Motion

Insight:

Regret bound \(\equiv\) almost sure bound on the integral

Gaps are on the support of a reflected random walk

A Probabilistic View of Regret Bounds

Formula for the regret based on the gaps

\displaystyle \mathrm{Regret(T)} = \sum_{t = 1}^{T} p(t, g_{t-1})(g_t - g_{t-1})

Random walk \(\longrightarrow\) Brownian Motion

\displaystyle \mathrm{ContRegret(p, T)} = \int_{0}^{T} p(t, |B_t|)\mathrm{d}|B_t|

Reflected Brownian motion

Conditions on the continuous player \(p\)

Continuous on \([0,T) \times \mathbb{R}\)

p(t,0) = \frac{1}{2}

for all \(t \geq 0\)

Stochastic Integrals and Itô's Formula

How to work with stochastic integrals?

\displaystyle f(T, |B_T|) - f(0, |B_0|) = \int_{0}^T \partial_g f(t, |B_t|) \mathrm{d}|B_t| + \int_{0}^T \overset{*}{\Delta} f(s, |B_s|) \mathrm{d}s

Itô's Formula:

\displaystyle \overset{*}{\Delta} f(s, |B_s|) = \partial_t f(s, |B_s|) + \frac{1}{2} \partial_{gg} f(s, |B_s|)
\displaystyle \mathrm{ContRegret}(\partial_g f, T)

Different from classic FTC!

\(\overset{*}{\Delta} f(t, g) = 0\) everywhere

 ContRegret doesn't depend on the path of \(B_t\)

\displaystyle \implies

Backwards Heat Equation

Goal:

Find a "potential function" \(R\) such that

(1) \(p = \partial_g R\) is a valid continuous player

(2) \(R\) satisfies the Backwards Heat Equation

A Solution Inspired by Cover's Algorithm

For Cover's algorithm, we can show

p^*(t,g) =

Lagging expert finishes leading

\mathbb{P} \Bigg(
\Bigg)

Gaps ~ Reflected RW

Law of Large Numbers:

\displaystyle = \mathbb{P}\Big(\mathcal{N}(0,T - t) > g\Big)
\displaystyle \Big\} \eqqcolon Q
= \mathbb{P}(S_T > 0 | S_t = -g)

Itô's Formula \(\implies\)

\displaystyle \mathrm{ContRegret}(Q, T) \leq
\displaystyle \sqrt{\frac{T}{2\pi}}
\displaystyle \approx \mathbb{P}(B_T > 0 | B_t = -g)

\(Q\) satisfies BHE

\(R(t,g)\) such that

\implies

Calculus trick

\displaystyle \Bigg\{

\(R\) satisfies BHE

\(\partial_g R = Q\)

\(R(t,g) \leq \sqrt{T/2\pi}\)

But we wanted a potential R satisfying BHE

Discretization

Discrete Derivatives

p^*(t,g) = \frac{1}{2} \big( V^*[t, g-1] - V^*[t, g+1]\big)
\approx \partial_g V^*[t,g]
\eqqcolon V_g^*[t,g]
V^*[t, g] - V^*[t-1, g]
= \frac{1}{2}( V^*[t, g-1] - V^*[t, g]) - \frac{1}{2}(V^*[t, g] - V^*[t,g+1])
\coloneqq V_t^*[t,g]
\coloneqq \frac{1}{2}V_{gg}^*[t,g]
\partial_t V^*[t,g] \approx
\approx \frac{1}{2}\partial_g V^*[t,g-1]
\approx \frac{1}{2} \partial_g V^*[t,g]
\approx \frac{1}{2} \partial_{gg} V^*[t,g]

Discrete Derivatives

p^*(t,g) = \frac{1}{2} \big( V^*[t, g-1] - V^*[t, g+1]\big)
\approx \partial_g V^*[t,g]
\eqqcolon V_g^*[t,g]
V^*[t, g] - V^*[t-1, g]
= \frac{1}{2}( V^*[t, g-1] + V^*[t, g+1] - 2V^*[t,g])
\coloneqq V_t^*[t,g]
\coloneqq \frac{1}{2} V_{gg}^*[t,g]

\(V^*\) satisfies the "discrete" Backwards Heat Equation!

Discretizatized player:

q(t,g) \coloneqq R_g(t,g)

Bound regret with a discrete analogue of Itô's Formula

Hopefully, \(R\) satisfies the discrete BHE

Bounding the Discretization Error

In the work of Harvey et al., they had

R_t(t,g) + \tfrac{1}{2} R_{gg}(t,g) \geq 0

In this fixed-time solution, we are not as lucky

Negative discretization error!

g
t
T = 1000

Bounding the Discretization Error

Main idea

\(R\) satisfies the continuous BHE

\implies
R_t(t,g) + \frac{1}{2} R_{gg}(t,g) \approx

Approximation error of the derivatives

\implies
\leq 0.74
\displaystyle \mathrm{Regret(T)} \leq \frac{1}{2} + \sqrt{\frac{T}{2\pi}} + \sum_{t = 1}^T O\Bigg( \frac{1}{(T - t)^{3/2}}\Bigg)

Lemma

\partial_{gg}R(t,g) - R_{gg}(t,g)
\partial_t R(t,g) - R_t(t,g)
\in O\Big( \frac{1}{(T - t)^{3/2}}\Big)
\Bigg\}

Efficient and Optimal Fixed-Time Regret with Two Experts  

Research Proficiency Exam

Victor Sanches Portella

PhD Student in Computer Science - UBC

October, 2020

RPE Presentation

By Victor Sanches Portella

RPE Presentation

  • 240