joint work with Nick Harvey and Christopher Liaw
Victor Sanches Portella
cs.ubc.ca/~victorsp
Player
Adversary
n Experts
0.5
0.1
0.3
0.1
Probabilities
1
-1
0.5
-0.3
Costs
Player's loss:
Adversary knows the strategy of the player
Loss of Best Expert
Player's Loss
Optimal!
For random ±1 costs
Multiplicative Weights Update:
(Hedge)
Boosting in ML
Understanding sequential prediction & online learning
Universal Optimization
TCS, Learning theory, SDPs...
Best Expert
Best Experts
ε-fraction
MWU:
Needs knowledge of ε
We design an algorith with Tln(1/ε) quantile regret
for all ε and best known leading constant
Loss of
top εn expert
ε-Quantile Regret
Algorithms design guided by
PDEs and (Stochastic) Calculus tools
Main Goal of this Talk: describe the main ideas of the
continuous time model and tools
Analysis often becomes clean
Sandbox for design of optimization algorithms
Gradient flow is useful for smooth optimization
Key Question: How to model non-smooth (online) optimization in continuous time?
Why go to ?
continuous time
Total loss of expert i:
Useful perspective: L(i) is a realization of a random walk
realization of a Brownian Motion
Probability 1 = Worst-case
Discrete Time
Continuous Time
Discrete time
Continuous time
Cummulative loss
Player's cummulative loss
Player's loss per round
[Freund '09]
Regret Vector
Regret
Potential based players
Multiplicative Weights Update
LogSumExp
NormalHedge
First algorithm for quantile regret
Very clean Continuous time analysis
[Freund '09]
Ito's Lemma
(Fundamental Theorem of Stochastic Calculus)
B(t) is very non-smooth ⟹ second-order terms matter
Ito's Lemma
Idea: Pick Φ as to make Ito's Lemma simpler for
Idea: Use stochastic calculus to guide the algorithm design
Potential based players
Smooth
Non-smooth
Potential based players
For all ε
Ito's Lemma suggests Φ that satisfy the Backwards Heat Equation
Using this potential*, we get
Best leading constant
Discrete time analysis is IDENTICAL to continuous time analysis
Discrete Ito's Lemma
*(with a slightly bigger cnst. in the BHE)
Question:
Are the minimax regret with and without knowledge of T different?
fixed-time
anytime
[Harvey, Liaw, Perkins, Randhawa '23]
n = 2
anytime
fixed-time
[Cover '67]
Back. Heat Eq.
Efficient version via SC
<
[Greenstreet, VSP, Harvey '20]
Heat Eq.
?
In Continuous Time, both are equal if Brownian Motions are independent.
[VSP, Liaw, Harvey '22]
Large n
Question:
What is the expected regret in the anytime setting
even without idependent experts?
[VSP, Liaw, Harvey '22]:
High expected regret ⟹ lower bound
In the language of martingales:
Nearly tight bounds.
asymptotically!
For a martingale Xt, find upper and lower bounds to
sup
is a stopping time
Evidence that
anytime = fixed-time
Player
Adversary
Unconstrained
Linear functions
Player's loss:
Loss of Fixed u
Player's Loss
Goal:
No knowledge of ∥u∥
Small regret if ∥gt∥ small
[Zhang, Yang, Cutkosky, Paschalidis '24]:
Parameter-free and Adaptive algorithm
Backwards Heat Equation
Parameter free and adaptive algorithms matching lower bounds
(even up to leading constant)
Pontential based player satisfying
+ refined discretization
Continuous Time Model for Experts and OLO
Thanks!
[VSP, Liaw, Harvey '22] Continuous prediction with experts' advice.
[Zhang, Yang, Cutkosky, Paschalidis '24] Improving adaptive online learning using refined discretization.
[Freund '09] A method for hedging in continuous time.
[Harvey, Liaw, Perkins, Randhawa '23] Optimal anytime regret with two experts.
[Greenstreet, VSP, Harvey '22] Efficient and Optimal Fixed-Time Regret with Two Experts
[Harvey, Liaw, VSP '22] On the Expected infinity-norm of High-dimensional Martingales
Improve LB for anytime experts? Or better upper-bounds?
?
High-dim continuous time OLO?
?
Hopefully this model can be helpful in more developments in OL and optimization!
Application to offline non-smooth optimization?
?
joint work with Nick Harvey and Christopher Liaw
Victor Sanches Portella
cs.ubc.ca/~victorsp
+ better anytime algorithms in continuous time
[Zhang, Yang, Cutkosky, Paschalidis '24]
Optimal anytime lower bound 2 experts + optimal algorithm
Best known algorithms for quantile regret
[Harvey, Liaw, Perkins, Randhawa '23]
Efficient optimal algorithms for fixed time 2 experts
[Greenstreet, VSP, Harvey '20]
Optimal parameter-free algorithms for online linear optimization
[VSP, Liaw, Harvey '22]
Simple continuous time analysis of NormalHedge
[Freund '09]
Potential based players
MWU!
Same regret bound as discrete time!
Idea: Use stochastic calculus to guide the algorithm design
LogSumExp
Regret bounds
when T is known
when T is not known
anytime
fixed-time
with prob. 1
Potential based players
Matches fixed-time!
Ito's Lemma suggests Φ that satisfy the Backwards Heat Equation
This new anytime algorithm has good regret!
Does not translate easily to discrete time
need correlation between experts
Take away: Anytime lower bounds for (continuous) experts
need dependent experts
Discrete Regret
Continuos Regret
Theorem:
If Φ satisfies the BHE and
Going to higher dim:
Continuous time analogue
of
Learn direction and scale separately
Use refined discretization
Discretizing:
Discrete time analysis is IDENTICAL to continuous time analysis
Discrete Ito's
Lemma
Improved anytime algorithms with bounds
quantile regret
Design guided by continuous time setting