Approximate Linear Programming for Markov Decision Processes
report is made by
Pavel Temirchev
Deep RL
reading group
Motivation
- We want to model Interaction with Users
- User state - is the Environment state
- Our actions are adds, recommendations, etc.
- Myopic predictions - current approach
- We want to maximize return for Long-Term interactions
- And we want to use pretrained myopic models (such as Logistic Regression)
- State-Action space is very large, usually discrete and sparse!
Contents
- Background
- MDP, Factored MDP
- Approximate Linear Programming
- Logistic Regression
- Logistic MDP
- Factored Logistic MDP
- ALP for Logistic MDP
- Exact Sequential Approach
- Piece-Wise Constant Approximation
- Error Analysis
- Experiments
- Extensions
Some remarks
- Model-Based method
- We do not model Transition Dynamics, it is given from sky
- Not really RL
- Not really Deep
- Work is in the progress
Background: MDP
Text
V^{\pi}(x) = r_x^a + \gamma \sum_{x'} p(x'|x, a) V^{\pi}(x')
Vπ(x)=rxa+γ∑x′p(x′∣x,a)Vπ(x′)
Q^{\pi}(x,a) = r_x^a + \gamma \sum_{x'} p(x'|x, a) V^{\pi}(x')
Qπ(x,a)=rxa+γ∑x′p(x′∣x,a)Vπ(x′)
where a=π(x)
\pi^*(x) = \arg\max_a Q^*(x, a)
π∗(x)=argmaxaQ∗(x,a)
p(x'|x, a)
p(x′∣x,a)
is transition probabilities
Background: MDP
x \in X
x∈X
- is a finite discrete state space
a \in A
a∈A
- is a finite discrete action space
x_i \in Dom(X_i)
xi∈Dom(Xi)
- is a finite discrete domain on each feature in x
- is a finite discrete domain on each feature in a
a_i \in Dom(A_i)
ai∈Dom(Ai)
x, a are then onehot encoded
Background:
Linear Programming task for MDP
\min_{v} \sum_x \alpha(x)v(x)
minv∑xα(x)v(x)
s.t.
x \in X
x∈X
v(x) \geq Q^v(x, a) = r_x^a + \gamma \sum p(x'|x,a)v(x')
v(x)≥Qv(x,a)=rxa+γ∑p(x′∣x,a)v(x′)
\forall x \in X, \forall a \in A
∀x∈X,∀a∈A
We have LP solvers.
Is this task tractable?
1) Too many variables to minimize
2) Too many summation terms
3) Too many terms in expectations
4) We even can't store transition p-s
5) Exponential number of constraints
Background: Factored MDP
We need concise representation of the transition probabilities
p(x' | x, a) = \prod_i p(x_i'|x,a)
p(x′∣x,a)=∏ip(xi′∣x,a)
Let:
And further:
p(x' | x, a) = \prod_i p(x_i'|par_i)
p(x′∣x,a)=∏ip(xi′∣pari)
par_i = (x[Par_i], a[Par_i])
pari=(x[Pari],a[Pari])
Par_i \subseteq X \cup A
Pari⊆X∪A
Let's use Dynamical Bayesian Network representation
4) We even can't store transition probabilities
Background:
Approximate Linear Programming
\min_{v} \sum_x \alpha(x)v(x)
minv∑xα(x)v(x)
1) Too many variables to minimize
2) Too many summation terms
Let
v(x) := \sum_{i=0}^k w_i\beta_i(x)
v(x):=∑i=0kwiβi(x)
And let's denote
x[B_i] = b_i, B_i \subseteq X
x[Bi]=bi,Bi⊆X
So
v(x) := \sum_{i=0}^k w_i\beta_i(b_i)
v(x):=∑i=0kwiβi(bi)
Where
\beta_i
βi
- are some basis functions
Background:
Approximate Linear Programming
If we assume the same initial distribution factorization
\alpha(x) := \prod_{i=0}^k \alpha(b_i)
α(x):=∏i=0kα(bi)
We will get a new LP task:
\min_w \sum_{i=0}^k \sum_{b_i} \alpha(b_i)w_i \beta_i(b_i)
minw∑i=0k∑biα(bi)wiβi(bi)
Background:
Approximate Linear Programming
PROOF:
\sum_x \alpha(x)v(x) = \sum_x \alpha(x)\sum_iw_i\beta_i(x[B_i])=
∑xα(x)v(x)=∑xα(x)∑iwiβi(x[Bi])=
= \sum_{b_0,...,b_k} \big[\prod_{j=0}^k \alpha(b_j) \big]\sum_iw_i\beta_i(b_i) =
=∑b0,...,bk[∏j=0kα(bj)]∑iwiβi(bi)=
=\sum_i \sum_{b_0,...,b_k} \big[\prod_{j=0}^k \alpha(b_j) \big]w_i\beta_i(b_i) =
=∑i∑b0,...,bk[∏j=0kα(bj)]wiβi(bi)=
=\sum_i \sum_{b_i} \alpha(b_i)\big[\sum_{b_j:j\neq i} \prod_{j\neq i} \alpha(b_j) \big]w_i\beta_i(b_i)
=∑i∑biα(bi)[∑bj:j=i∏j=iα(bj)]wiβi(bi)
=\sum_i \sum_{b_i} \alpha(b_i)w_i\beta_i(b_i)
=∑i∑biα(bi)wiβi(bi)
Background: ALP + Factored MDP
v(x) \geq Q^v(x, a) = r_x^a +\gamma \sum p(x'|x,a)v(x')
v(x)≥Qv(x,a)=rxa+γ∑p(x′∣x,a)v(x′)
3) Too many terms in expectations
Constraints for the LP problem:
\sum_{i=0}^k w_i \big[\gamma g_i - \beta_i(b_i)\big] + r_x^a \leq 0
∑i=0kwi[γgi−βi(bi)]+rxa≤0
g_i = \sum_{b_i'}\beta_i(b_i')p(b_i'|par_{B_i})
gi=∑bi′βi(bi′)p(bi′∣parBi)
par_{B_i} = \cup_{i:X_i\in B_i} par_i
parBi=∪i:Xi∈Bipari
It may be rewriten as:
And even further decompose rewards as:
r_x^a = \sum_{j=0}^r \rho_j(x[R_j], a[R_j])
rxa=∑j=0rρj(x[Rj],a[Rj])
Background: ALP + Factored MDP
v(x) \geq r_x^a +\gamma \sum p(x'|x,a)v(x')
v(x)≥rxa+γ∑p(x′∣x,a)v(x′)
\sum_{i=0}^k w_i \big[\gamma g_i(par_{B_i}) - \beta_i(b_i)\big] + r_x^a \leq 0
∑i=0kwi[γgi(parBi)−βi(bi)]+rxa≤0
g_i = \sum_{b_i'}\beta_i(b_i')p(b_i'|par_{B_i})
gi=∑bi′βi(bi′)p(bi′∣parBi)
PROOF:
\sum_iw_i\beta_i(b_i) \geq r_x^a +
∑iwiβi(bi)≥rxa+
+ \gamma \sum_{b_0',...,b_k'} \prod_{j=0}^k p(b_j'|par_{B_j})\sum_iw_i \beta_i(b_i')
+γ∑b0′,...,bk′∏j=0kp(bj′∣parBj)∑iwiβi(bi′)
\sum_iw_i\beta_i(b_i) \geq r_x^a +
∑iwiβi(bi)≥rxa+
+ \gamma \sum_i \big[\sum_{b_j':j\neq i} \prod_{j\neq i} p(b_j'|par_{B_j})\big] \sum_{b_i'} p(b_i'|par_{B_i}) w_i \beta_i(b_i')
+γ∑i[∑bj′:j=i∏j=ip(bj′∣parBj)]∑bi′p(bi′∣parBi)wiβi(bi′)
\cdot
⋅
\cdot
⋅
\cdot
⋅
Background: Constraint Generation
5) Exponential number of constraints
- Solve master LP problem for a subset of constraints using GLOP. Get optimal w values
- Find a maximally violated constraint among ones, which was not added to master LP.
- Add it to the master LP if violation is positive, else Break
- Repeat
\max_{x, a} \sum_{i=0}^k w_i \big[\gamma g_i(par_{B_i}) - \beta_i(b_i)\big] + r_x^a
maxx,a∑i=0kwi[γgi(parBi)−βi(bi)]+rxa
MVC search:
Background: Logistic Regression
Text
p(\phi = 1 | x, a) = \sigma (x^T u_x + a^T u_a)
p(ϕ=1∣x,a)=σ(xTux+aTua)
\sigma(x) = \frac{1}{1 + \exp(-x)}
σ(x)=1+exp(−x)1
Need more? Try Google, it's free
Logistic Regression
\phi
ϕ
X_1
X1
X_n
Xn
A_1
A1
A_m
Am
STATE
ACT
ION
RESPONSE
X_1
X1
X_n
Xn
A_1
A1
A_m
Am
X_1
X1
X_n
Xn
A_1
A1
A_m
Am
t
t+1
MDP
Logistic Markov Decision Processes
\phi
ϕ
X_1
X1
X_n
Xn
A_1
A1
A_m
Am
X_1
X1
X_n
Xn
A_1
A1
A_m
Am
t
t+1
p(x^{t+1}| x^{t}, a^t) =
p(xt+1∣xt,at)=
=\mathbb{E}_{p(\phi|x,a)} p(x^{t+1}| x^{t}, a^t,\phi^t)
=Ep(ϕ∣x,a)p(xt+1∣xt,at,ϕt)
We allow response ϕt to influence user's state at t+1 timestep
Factored Logistic MDP
p(x^{t+1}|x^t, a^t) = \sum_{\phi\in\{0,1\}}\prod_ip(x^{t+1}_i|par_i,\phi)p(\phi|x^t,a^t)
p(xt+1∣xt,at)=∑ϕ∈{0,1}∏ip(xit+1∣pari,ϕ)p(ϕ∣xt,at)
Transition Dynamics:
Reward function:
r^x_a(\phi) = \sum_{j=0}^r \rho_j(x[R_j],a[R_j],\phi)
rax(ϕ)=∑j=0rρj(x[Rj],a[Rj],ϕ)
Hence, our backprojections gi now dependent on complete x and a vectors
Let's rewrite Q(x,a) as:
Q(x, a) = \mathbb{E}_{\phi} \big[ r^x_a(\phi) + \gamma\sum_iw_ig_i(par_{B_i}, \phi) \big]
Q(x,a)=Eϕ[rax(ϕ)+γ∑iwigi(parBi,ϕ)]
ALP for Logistic MDP
Let's denote:
h(x,a,\phi,w) = \sum_j^r \rho(x[R_j],a[R_j],\phi) +
h(x,a,ϕ,w)=∑jrρ(x[Rj],a[Rj],ϕ)+
+ \sum_i w_i\big(\gamma g_i (par_{B_i},\phi)-\beta_i(b_i) \big)
+∑iwi(γgi(parBi,ϕ)−βi(bi))
Then ALP task may be reformulated as:
\min_w \sum_i \sum_{b_i} w_i \alpha(b_i)\beta(b_i)
minw∑i∑biwiα(bi)β(bi)
0 \geq C(x,a,w)
0≥C(x,a,w)
\forall x \in X, \forall a \in A
∀x∈X,∀a∈A
Constraints are now nonlinear since p(ϕ∣x,a) is nonlinear. MCV search is not MIP problem now
s.t.
C(x,a,w) = \sum_{\phi \in \{0,1\}}p(\phi|x, a) h(x,a,\phi,w)
C(x,a,w)=∑ϕ∈{0,1}p(ϕ∣x,a)h(x,a,ϕ,w)
ALP for Logistic MDP
\max_{x,a} \sum_{\phi \in \{0,1\}}p(\phi|x, a) h(x,a,\phi,w)
maxx,a∑ϕ∈{0,1}p(ϕ∣x,a)h(x,a,ϕ,w)
We will denote
p(\phi|x,a) = \sigma(f(x,a))
p(ϕ∣x,a)=σ(f(x,a))
And
[f_l, f_u] = \{ (x,a) : f_l \leq f(x,a) \leq f_u \}
[fl,fu]={(x,a):fl≤f(x,a)≤fu}
\sigma_l = \sigma(f_l)
σl=σ(fl)
\sigma_u = \sigma(f_u)
σu=σ(fu)
Constant Approximation
\max_{x,a} \sigma^* h(x,a,\phi=1) +(1 - \sigma^*) h(x,a,\phi=0)
maxx,aσ∗h(x,a,ϕ=1)+(1−σ∗)h(x,a,ϕ=0)
H^+ = \{ (x,a): h(x,a,\phi=1) - h(x,a,\phi=0) \geq 0\}
H+={(x,a):h(x,a,ϕ=1)−h(x,a,ϕ=0)≥0}
Where σ∗ is some constant
We consider two subsets of possible (x,a) pairs:
where the constraint is non-decreasing with σ∗ growth
H^- = \{ (x,a): h(x,a,\phi=1) - h(x,a,\phi=0) < 0\}
H−={(x,a):h(x,a,ϕ=1)−h(x,a,ϕ=0)<0}
where the constraint is non-increasing with σ∗ growth
We denote by Uu the solution for
\max_{x,a} \sigma_u h(x,a,\phi=1) +(1 - \sigma_u) h(x,a,\phi=0)
maxx,aσuh(x,a,ϕ=1)+(1−σu)h(x,a,ϕ=0)
s.t.
(x,a)\in [f_l, f_u] \cap H^+
(x,a)∈[fl,fu]∩H+
And by Ul the solution with σl and H− instead of σu and H+
Constant Approximation
- Uu is an upper bound on the maximal constraint violation (CV) in the subset (x,a)∈[fl,fu]∩H+
- True CV in that point : Cu=C(xu,au,w) is a lower bound on maximal CV in this subset.
Same thing for the Ul and Cl in the subset (x,a)∈[fl,fu]∩H−
Hence,
U^* = max(U^l, U^u)
U∗=max(Ul,Uu)
C^* = max(C^l, C^u)
C∗=max(Cl,Cu)
is an upper bound on MCV in [fl,fu]
is a lower bound on MCV in [fl,fu]
Constant Approximation
CV
σ
\cdot
⋅
\cdot
⋅
\cdot
⋅
C(x(2),a(2),σ)
C(x(1),a(1),σ)
Uu
σl
σu
σ(f(x(2),a(2)))
σ(f(x(1),a(1)))
The degree of CV for two state-action pairs as a function of σ
MVC search in ALP-SEARCH
1) Solve two MIP tasks for some interval [fl,fu] :
\max_{x,a} \sigma_u h(x,a,\phi=1) +(1 - \sigma_u) h(x,a,\phi=0)
maxx,aσuh(x,a,ϕ=1)+(1−σu)h(x,a,ϕ=0)
s.t.
(x,a)\in [f_l, f_u] \cap H^+
(x,a)∈[fl,fu]∩H+
\max_{x,a} \sigma_l h(x,a,\phi=1) +(1 - \sigma_l) h(x,a,\phi=0)
maxx,aσlh(x,a,ϕ=1)+(1−σl)h(x,a,ϕ=0)
s.t.
(x,a)\in [f_l, f_u] \cap H^-
(x,a)∈[fl,fu]∩H−
2) If U∗<ϵ then there are no constrain violation in [fl,fu] and we terminate
3) If U∗−C∗<ϵ then we report that C∗ is a MCV in [fl,fu] and we terminate
3) If C′ in another interval is larger than C∗ then we terminate
If nothing from above holds we divide interval into two and recursively repeat
Piece-Wise Constant approximation
A piece-wise constant approximation of the sigmoid
MVC search in ALP-APPROX
\max_{x,a} \sigma_i h(x,a,\phi=1) +(1 - \sigma_i) h(x,a,\phi=0)
maxx,aσih(x,a,ϕ=1)+(1−σi)h(x,a,ϕ=0)
(x,a) \in [\delta_{i-1}, \delta_i]
(x,a)∈[δi−1,δi]
s.t.
where
\sigma_i = \sigma(f_i), f_i: \delta_{i-1} \leq f_i \leq \delta_i
σi=σ(fi),fi:δi−1≤fi≤δi
Then we calculate:
C^i = C(x^i, a^i , \sigma(x^i,a^i))
Ci=C(xi,ai,σ(xi,ai))
- true CV, probably not a maximal one in [δi−1,δi]
C^* = \max_iC^i
C∗=maxiCi
- estimation of the maximal CV in [fl,fu]
Approximation error in ALP-APPROX
THEOREM
A bounded log-relative error for the logistic regression (assuming features with finite domains) can be achieved with O(ϵ1∣∣u∣∣1) intervals in logit space, where u is the logistic regression weight vector
THEOREM
Given an interval [a,b] in logit space, the value σ(x) with x=ln1+ebea+b+eb minimizes the log-relative error over the interval.
Experiments
Text
- Advertising task
- Reward: 1 for click, 0 otherwise
- The aim is to maximize Cumulative Click-Through Rate (CCTR)
- Features are one-hot encoded
- Features are divided into three categories:
- User state (static or dynamic) - state variable
- Ad description - action variable
- User-Ad interaction - action variable
- Transition dynamics is simple: either identity function or Bernoulli distribution on moving to next bucket in feature domain
- Pretrained Logistic Regression on 300M of examples
Experiments
Text
Model sizes:
- Tiny:
- 2 state features (48 binarized)
- 1 action feature (7 binarized)
-
Small:
- 6 state features (71 binarized)
- 4 action feature (15 binarized)
-
Medium:
- 11 state features (251 binarized)
- 8 action feature (170 binarized)
- Large:
- 12 state features (2630 binarized)
- 11 action features (224 binarized)
Experiments
Experiments
Experiments
Extensions
Text
- Relaxation of the CG optimization
- Cross-product features
- Multiple response variables
- Partition-free CG
- Non-linear response model
Partition-free CG
\max_{x,a} \sigma(f(x,a) )h(x,a,\phi=1) +\sigma(-f(x,a)) h(x,a,\phi=0)
maxx,aσ(f(x,a))h(x,a,ϕ=1)+σ(−f(x,a))h(x,a,ϕ=0)
which is equivalent to
\max_{x,a,y} \sigma(y)h(x,a,\phi=1) +\sigma(-y) h(x,a,\phi=0)
maxx,a,yσ(y)h(x,a,ϕ=1)+σ(−y)h(x,a,ϕ=0)
y=f(x,a)
y=f(x,a)
s.t.
The simple idea is to iteratively alternate between two steps:
- maximize over x,a using MIP solver
- choose y=f(x,a)
But it will be stuck in local optima almost surely
Partition-free CG
\min_{\lambda}\max_{x,a,y} \sigma(y)h(x,a,\phi=1) +\sigma(-y) h(x,a,\phi=0) -
minλmaxx,a,yσ(y)h(x,a,ϕ=1)+σ(−y)h(x,a,ϕ=0)−
Another approach is to consider Lagrangian relaxation:
- \lambda f(x,a) + \lambda y
−λf(x,a)+λy
Primal-Dual alternating optimization:
Initialize
\lambda, x, a, y
λ,x,a,y
for
t:= 1,...,T
t:=1,...,T
do:
end for
1)
1)
2)
2)
y^{(t+1)} = y^{(t)} + \eta_t \nabla^{(t)}_y
y(t+1)=y(t)+ηt∇y(t)
(x,a)^{(t+1)}=\arg\max_{x,a} \sigma(y^{(t+1)})h(x,a,\phi=1)+
(x,a)(t+1)=argmaxx,aσ(y(t+1))h(x,a,ϕ=1)+
+\sigma(-y^{(t+1)})h(x,a,\phi=0)
+σ(−y(t+1))h(x,a,ϕ=0)
\lambda^{(t+1)} = \lambda^{(t)} - \hat{\eta}_t \big[
y^{(t+1)} - f((x,a)^{(t+1)}) \big]
λ(t+1)=λ(t)−η^t[y(t+1)−f((x,a)(t+1))]
3)
3)
4)
4)
5)
5)
6)
6)
Non-linear Response Model
Text
We consider wide-n-deep response model:
- some features from x and a are used as an input to the final logistic output unit
- some features are passed through DNN with several layers and several non-linear units
If we can express non-linearity in DNN in such way that the input to the final logistic output formulates as a linear-like function of (x,a) then CG optimization will be the same as for Logistic Regression response model.
ReLu non-linearity may be expressed in such way using just one or two indicator functions per unit
Thanks for your
attention!
Approximate Linear Programming for Markov Decision Processes report is made by Pavel Temirchev Deep RL reading group
Approximate Linear Programming for Markov Decision Processes
By cydoroga
Approximate Linear Programming for Markov Decision Processes
- 583
