Online Convex Optimization
Learning, Duality, and Algorithms
Victor Sanches Portella
Advisor: Marcel K. de Carli Silva
IME - USP
May, 2019
Online Convex Optimization
Online Convex Optimization (OCO)
At each round
Player chooses a point
Enemy chooses a function
Player suffers a loss
SIMULTANEOUSLY
Player
Enemy
!
!
x∈X
x \in X
f
f
f(x)
f(x)
CONVEX
Player and Enemy see
f and x
f~\text{and}~x
Formalizing Online Convex Optimization
An Online Convex Optimization Problem
C=(X,F)
\mathcal{C}
= (X, \mathcal{F})
X
X
convex set
F
\mathcal{F}
set of convex functions
Player
Enemy
Rounds
t=1,…,T
t = 1, \dotsc, T
xt∈X
x_t \in X
ft∈F
f_t \in \mathcal{F}
Expert's Problem
Player
Enemy
Experts
0.5
0.1
0.3
0.1
1
0
-1
1
f(p)=yTp=Ee∼p[ye]
f(p) = y^{T}p = \mathbb{E}_{e \sim p}[y_e]
Probabilities
Costs
p∈ΔE
p \in \Delta_E
y∈[−1,1]E
y \in [-1,1]^E
Online Regression
Online Linear Regression
Player
Enemy
rt
r_t
(xt,yt)
(x_t, y_t)
∣rt(xt)−yt∣
|r_t(x_t) - y_t|
rt(x)=⟨wt,x⟩
r_t(x) = \langle w_t,x \rangle
Regression Function
Query & Answer
Loss
wt
w_t
ft(w)=∣⟨w,xt⟩−yt∣
f_t(w) = |\langle w, x_t \rangle - y_t |
Regret
RegretT(u)=t=1∑Tft(xt)−t=1∑Tft(u)
\mathrm{Regret}_T( u)
= \displaystyle \sum_{t = 1}^T f_t(x_t) - \sum_{t = 1}^T f_t(u)
RegretT(U)=u∈UsupRegretT(u)
\mathrm{Regret}_T( U)
= \displaystyle \sup_{u \in U} \mathrm{Regret}_T( u)
Cost of always choosing
u
u
Goal: sublinear Regret
T→∞limT1RegretT(U)=0
\displaystyle \lim_{T \to \infty} \frac{1}{T}\mathrm{Regret}_T( U)
= 0
Player's Loss
Player Strategies
Sublinear regret under mild conditions
Focus of this talk: algorithms for the Player
Hupefully efficiently implementable
Algorithms We Shall See
AdaFTRL
\text{AdaFTRL}
AdaOMD
\text{AdaOMD}
AdaDA
\text{AdaDA}
AdaReg
\text{AdaReg}
FTRL
\text{FTRL}
EOMD
\text{EOMD}
LOMD
\text{LOMD}
Adaptive
FTRL
Cummulative Loss
Experts
0
0
1
1
0.5
0.5
1
1
t=1
t = 1
1
1
1.5
1.5
0.5
0.5
1
1
t=2
t = 2
1.5
1.5
2
2
1
1
1.5
1.5
t=3
t = 3
2.5
2.5
3
3
2
2
1.5
1.5
t=4
t = 4
Follow the Leader
Enemy
Player
f1
f_1
f2
f_2
f3
f_3
f4
f_4
xt+1=argmini=1∑tfi(x)
x_{t+1} = \displaystyle \mathrm{arg}\,\mathrm{min} \sum_{i = 1}^t f_{i}(x)
UNSTABLE!
x1
x_1
x2
x_2
x3
x_3
x4
x_4
x∈X
{}_{x \in X}
Adding Regularization
Enemy
Player
f1
f_1
x1
x_1
f2
f_2
x2
x_2
f3
f_3
x3
x_3
f4
f_4
x4
x_4
xt+1=argmini=1∑tfi(x)+R(x)
x_{t+1} = \displaystyle \mathrm{arg}\,\mathrm{min} \sum_{i = 1}^t f_{i}(x) + R(x)
R
R
FTRL
Fixed Regularizer
x∈E
{}_{x \in \mathbb{E}}
Adding Adaptive Regularization
At round use regularizer
Rt
R_t
t
t
xt+1=argmini=1∑tfi(x)+Rt+1(x)
x_{t+1} = \displaystyle \mathrm{arg}\,\mathrm{min} \sum_{i = 1}^t f_{i}(x) + R_{t+1}(x)
Rt
R_t
Rt+1
R_{t+1}
?
rt+1
r_{t+1}
Rt+1=Rt+rt+1
R_{t+1} = R_t + r_{t+1}
Rt+1=r1+r2+…+rt+1
R_{t+1} = r_1 + r_2 + \dotsc + r_{t+1}
Regularizer Increment
Convex Function
x∈E
{}_{x \in \mathbb{E}}
AdaFTRL
xt+1=argmini=1∑tfi(x)+Rt+1(x)
x_{t+1} = \displaystyle \mathrm{arg}\,\mathrm{min} \sum_{i = 1}^t f_{i}(x) + R_{t+1}(x)
i=1∑t+1ri(x)
\displaystyle
\sum_{i = 1}^{t+1} r_{i}(x)
Efficiently computable?
Not clear in general
x∈E
{}_{x \in \mathbb{E}}
Adaptive
Online Mirror Descent
Online (sub)Gradient Descent
−∇ft(xt)
- \nabla f_t(x_t)
X
X
xt
x_t
xt+1
x_{t+1}
Round
t
t
xt+1=ProjX(xt−∇ft(xt))
x_{t+1} = \mathrm{Proj}_X(x_t - \nabla f_t(x_t))
projection
Another Perspective
∇ft(xt)
\nabla f_t(x_t)
Representation of derivative
[Dft(xt)]( )=⟨∇ft(xt), ⟩
[Df_t(x_t)](~\;) =
\langle \nabla f_t(x_t), ~~\; \rangle
What is
?
direction
u
u
u
u
u
u
Online Gradient Descent Update
xt−∇ft(xt)
x_t - \nabla f_t(x_t)
xt−Dft(xt)
x_t - Df_t(x_t)
point
⟨xt,⋅⟩−Dft(xt)
\langle x_t, \cdot \rangle - Df_t(x_t)
functional
(Riesz Repr. Theorem)
functional
functional
Directional derivative of at
ft
f_t
xt
x_t
Avoiding Inner-Product
⟨xt,⋅⟩−Dft(xt)=DR(xt)−Dft(xt)
\langle x_t, \cdot \rangle - Df_t(x_t) = D R (x_t) - Df_t(x_t)
R(x)=21⟨x,x⟩
R(x) = \frac{1}{2} \lang x, x\rang
⟹
\implies
∇R(x)=x
\nabla R(x) = x
xt−η∇ft(xt)=∇R(xt)−∇ft(xt)
x_t - \eta \nabla f_t(x_t) = \nabla R (x_t) - \nabla f_t(x_t)
What if we make other choices for ?
R(x)
R(x)
21∥x∥22
\frac{1}{2}\lVert x\rVert_2^2
Mirror Maps
What if we make other choices for ?
R(x)
R(x)
R(x)
R(x)
(i)
(i)
strictly convex and differentiable on
int(domR)
\mathrm{int}(\mathrm{dom} R)
(ii)
(ii)
y=∇R( )
y = \nabla R(~~)
(iii)
(iii)
For every
y
y
there is
∈int(domR)
~~~ \in \mathrm{int}(\mathrm{dom} R)
such that
ΠXR(y)∈int(domR)
\Pi_X^R(y) \in \mathrm{int}(\mathrm{dom} R)
Bregman Projections onto attained by
int(domR)
\mathrm{int}(\mathrm{dom} R)
yˉ
\bar{y}
yˉ
\bar{y}
∇R−1(y)=
\nabla R^{-1}(y) =
yˉ
\bar{y}
⟹
\implies
X
X
∀y∈int(domR)
\forall y \in \mathrm{int}(\mathrm{dom} R)
∗
{}^*
Bregman Projector
Online Mirror Descent
xt
x_t
∇R(xt)
\nabla R(x_t)
−∇ft(xt)
- \nabla f_t(x_t)
yt+1
y_{t+1}
xt+1
x_{t+1}
∇R
\nabla R
∇R∗
\nabla R^*
ΠXR
\Pi_X^R
Bregman
Projection
Dual
Primal
X
X
int(domR)
\mathrm{int}(\mathrm{dom} R)
Adaptive?
t+1
{}_{t+1}
t+1
{}_{t+1}
t+1
{}_{t+1}
∇R (xt)
\nabla R_{~~~~~~}(x_t)
t+1
{}_{t+1}
Adaptive!
Adaptive Online Mirror Descent
First round
x1∈argmin R1(x)
x_1 \in \mathrm{arg}\,\mathrm{min}~R_1(x)
x∈X
x \in X
Round
t+1
t+1
for
t=1,…,T
t = 1, \dotsc, T
yt+1=∇Rt+1(xt)−∇ft(xt)
y_{t+1} = \nabla R_{t+1}(x_t) - \nabla f_t(x_t)
xt+1=ΠXRt+1(∇Rt+1∗(yt+1))
x_{t+1} = \Pi_X^{R_{t+1}} (\nabla R^*_{t+1}(y_{t+1}))
Rt+1=r1+…+rt+rt+1
R_{t+1} = r_1 + \dotsc + r_t + r_{t+1}
Rt+1=Rt+rt+1
R_{t+1} = R_t + r_{t+1}
Mirror Map Increments
Lazy Online Mirror Descent
xt
x_t
∇R(xt)
\nabla R(x_t)
−∇ft(xt)
- \nabla f_t(x_t)
yt+1
y_{t+1}
xt+1
x_{t+1}
∇R
\nabla R
∇R∗
\nabla R^*
ΠXR
\Pi_X^R
Bregman
Projection
X
X
int(domR)
\mathrm{int}(\mathrm{dom} R)
yt
y_{t}
Classic Online Mirror Descent
First round
x1∈argmin R(x)
x_1 \in \mathrm{arg}\,\mathrm{min}~R(x)
x∈X
x \in X
xt+1=ΠXR(∇R∗(yt+1))
x_{t+1} = \Pi_X^R (\nabla R^*(y_{t+1}))
First round
x1∈argmin R(x)
x_1 \in \mathrm{arg}\,\mathrm{min}~R(x)
x∈X
x \in X
For
t=1,…,T
t = 1, \dotsc, T
yt+1= −∇ft(xt)
y_{t+1} = ~~~~~~~ - \nabla f_t(x_t)
xt+1=ΠXR(∇R∗(yt+1))
x_{t+1} = \Pi_X^R (\nabla R^*(y_{t+1}))
For
t=1,…,T
t = 1, \dotsc, T
yt
y_t
yt+1= −∇ft(xt)
y_{t+1} = ~~~~~~~~~~~~~ - \nabla f_t(x_t)
∇R(xt)
\nabla R(x_t)
Eager
Lazy
LOMD as FTRL
yt+1= −∇ft(xt)
y_{t+1} = ~~~~~- \nabla f_t(x_t)
= −∇ft(xt)
= ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - \nabla f_t(x_t)
−i=1∑t∇fi(xi)
\displaystyle -\sum_{i = 1}^{t} \nabla f_i(x_i)
...
yt
y_t
yt−1−∇ft−1(xt−1)
y_{t-1} -\nabla f_{t-1}(x_{t-1})
=
=
xt+1=argmin i=1∑t⟨∇fi(xi),x⟩+RX(x)
\displaystyle x_{t+1} = \mathrm{arg}\,\mathrm{min}~ \sum_{i=1}^t \langle \nabla f_i(x_i), x \rangle + R_X(x)
RX=
R_X =
{
\{
R
R
inside
X
X
+∞
+ \infty
outside
FTRL
∇RX∗(yt+1)=ΠXR(∇R∗(yt+1))
\nabla R_X^*(y_{t+1}) =
\Pi_X^R(\nabla R^*(y_{t+1}))
x∈E
{}_{x \in \mathbb{E}}
EOMD as FTRL
yt+1= −∇ft(xt)
y_{t+1} = ~~~~~~~~~~~~~~~- \nabla f_t(x_t)
= −∇ft(xt)
= ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - \nabla f_t(x_t)
∇R(xt)
\nabla R(x_t)
yt−1−∇ft−1(xt−1)
y_{t-1} -\nabla f_{t-1}(x_{t-1})
RX=
R_X =
{
\{
R
R
inside
X
X
+∞
+ \infty
outside
∂RX(xt)=∇R(xt)+NX(xt)
\partial R_X(x_t)
= \nabla R(x_t) + N_X(x_t)
∇RX∗(yt+1)=ΠXR(∇R∗(yt+1))
\nabla R_X^*(y_{t+1}) =
\Pi_X^R(\nabla R^*(y_{t+1}))
NX(xt)
N_X(x_t)
X
X
xt
x_t
EOMD as FTRL
X
X
∇R(xt)
\nabla R(x_t)
R
R
∇R(xt)+NX(xt)
\nabla R(x_t) + N_X(x_t)
xt
x_t
NX(xt)=[0,+∞)
N_X(x_t) = [0, +\infty)
EOMD as FTRL
yt+1= −∇ft(xt)
y_{t+1} = ~~~~~~~~~~~~~~~- \nabla f_t(x_t)
∇R(xt)
\nabla R(x_t)
xt+1=argmin i=1∑t⟨∇fi(xi)+pi,x⟩+RX(x)
\displaystyle x_{t+1} = \mathrm{arg}\,\mathrm{min}~ \sum_{i=1}^t \langle \nabla f_i(x_i) + p_i, x \rangle + R_X(x)
RX=
R_X =
{
\{
R
R
inside
X
X
+∞
+ \infty
outside
FTRL
∂RX(xt)=∇R(xt)+NX(xt)
\partial R_X(x_t)
= \nabla R(x_t) + N_X(x_t)
∇RX∗(yt+1)=ΠXR(∇R∗(yt+1))
\nabla R_X^*(y_{t+1}) =
\Pi_X^R(\nabla R^*(y_{t+1}))
p1∈NX(x1),p2∈NX(x2),…,pt∈NX(xt)
p_1 \in N_X(x_1), p_2 \in N_X(x_2), \dotsc, p_t \in N_X(x_t)
x∈E
{}_{x \in \mathbb{E}}
EOMD vs LOMD
xt+1=argmin i=1∑t⟨∇fi(xi),x⟩+RX(x)
\displaystyle x_{t+1} = \mathrm{arg}\,\mathrm{min}~ \sum_{i=1}^t \langle \nabla f_i(x_i), x \rangle + R_X(x)
Eager = Lazy
xt+1=argmin i=1∑t⟨∇fi(xi)+ ,x⟩+RX(x)
\displaystyle x_{t+1} = \mathrm{arg}\,\mathrm{min}~ \sum_{i=1}^t \langle \nabla f_i(x_i) + ~~~~ , x \rangle + R_X(x)
pi
p_i
NX(z1)
N_X(z_1)
X
X
z1
z_1
z2
z_2
NX(z2)={0}
N_X(z_2) = \{0\}
xi∈int(dom R)
x_i \in \mathrm{int}(\mathrm{dom~R})
int(domR)⊆ri X
\mathrm{int}(\mathrm{dom R})
\subseteq \mathrm{ri}~X
⟹
\implies
pi
p_i
x∈E
{}_{x \in \mathbb{E}}
x∈E
{}_{x \in \mathbb{E}}
A Genealogy of
Algorithms
Connection Among the Main Algorithms
AdaReg
yt+1=xt− ∇ft(xt)
y_{t+1} = x_t - ~~~~~~~ \nabla f_t(x_t)
xt+1=Proj (yt+1)
x_{t+1} = \mathrm{Proj}_{~~~~~~~~}(y_{t+1}^{})
Ht+1
H_{t+1}
Ht+1−1
{}_{H_{t+1}^{-1}}
Ht+1≈Gt−1
\displaystyle H_{t+1} \approx G_t^{-1}
Ht+1≈Gt−21
\displaystyle H_{t+1} \approx G_t^{-\frac{1}{2}}
Second Order Algorithms?
Gt=i=1∑t∇fi(xi)∇fi(xi)⊺
\displaystyle G_{t} = \sum_{i = 1}^t \nabla f_i(x_i) \nabla f_i(x_i)^{\intercal}
A Bird's-eye View
Future Directions
Generalizations and Special Cases
Limited Feedback: Bandit, two-point Bandit feedback
Special Cases: Combinatorial, other specific settings
Player
Drop or Add Hypotheses: Convexity, adversarial enemies,
Hypercube
L2-Ball
Change Metric: Policy Regret, Raw Loss
side information
OCO in Other Areas
Quantum Computing
Approximately Maximum Flow
Robust Optimization
Competitive Analysis
Spectral Sparsification
SDP Solver
Oracle Boosting
Ideas
New Setting
Variational Perspective
Online Convex Optimization
Learning, Duality, and Algorithms
Victor Sanches Portella
Advisor: Marcel K. de Carli Silva
IME - USP
May, 2019
Online Convex Optimization Learning, Duality, and Algorithms Victor Sanches Portella Advisor: Marcel K. de Carli Silva IME - USP May, 2019