Differentiable RL
Stream of Consciousness....
Policy Search Problem
\begin{aligned}
V(x_t,\alpha)\quad & \coloneqq \mathbb{E}_{w_{t:\infty}\sim p}\left[\sum^\infty_{t'=t} \gamma^{t'-t} r(x_{t'},u_{t'})\right] \\
\text{s.t.}\quad & x_{t+1}=f(x_t,u_t),\; u_t = \pi(x_t,\alpha) + w_t
\end{aligned}
Value Function
\begin{aligned}
\min_\alpha \mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right]
\end{aligned}
Initial distribution of rewards
Gaussian Policy
Deterministic & Differentiable Dynamics
Policy Optimization given Critic
Model Value Expansion
\begin{aligned}
V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H V_\theta(x_H)}\right] \\
\end{aligned}
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
\begin{aligned}
H=\infty
\end{aligned}
\begin{aligned}
H=0
\end{aligned}
DDPG
BPTT
\begin{aligned}
u_H = \pi(x_H,\alpha) + w_H
\end{aligned}
Policy Optimization given Critic
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
\begin{aligned}
u_H = \pi(x_H,\alpha) + w_H
\end{aligned}
Model-Based Component
Model-Free Component
\begin{aligned}
H=\infty
\end{aligned}
\begin{aligned}
H=0
\end{aligned}
DDPG
BPTT
Bias-Variance Tradeoff
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
- Unbiased in Expectation
- Potentially High Variance
- Biased Estimate
- Low Variance
Bias-Variance Tradeoff
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
- Unbiased in Expectation
- Potentially High Variance
- Biased Estimate
- Low Variance
Q1: How do we choose the horizon for a given system?
Horizon Selection
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
- Unbiased in Expectation
- Potentially High Variance
- Biased Estimate
- Low Variance
Consider a simple rule:
\begin{aligned}
\min_H \quad & \textbf{Bias}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \\
\text{s.t.} \quad & \textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \leq C
\end{aligned}
Horizon Selection
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
- Unbiased in Expectation
- Potentially High Variance
- Biased Estimate
- Low Variance
Consider a simple rule:
\begin{aligned}
\min_H \quad & \textbf{Bias}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \\
\text{s.t.} \quad & \textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \leq C
\end{aligned}
Maximize horizon to minimize bias
Until the variance is too difficult to handle
Horizon Selection
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
- Unbiased in Expectation
- Potentially High Variance
- Biased Estimate
- Low Variance
Consider a simple rule:
\begin{aligned}
\max_H \quad & H \\
\text{s.t.} \quad & \textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \leq C
\end{aligned}
Maximize horizon to minimize bias
Until the variance is too difficult to handle
Controlling Model Variance
Variance Reduction
Can use empirical variance, but tend to be difficult to estimate.
\begin{aligned}
\textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] & = \textbf{Var}\left[\frac{\partial}{\partial\alpha} \sum_{t=0}^{H-1} \gamma^t r_t\right]\\
& = \mathcal{O}(H^4 {\color{red} L_f^{4H}}L_\pi^{4H}/M)
\end{aligned}
Bound on First-order Gradient Variance given Lipschitz Constants
Controlling Dynamic Stiffness Alleviates Variance!
Controlling Model Variance
Variance Reduction
Can use empirical variance, but tend to be difficult to estimate.
\begin{aligned}
\textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] & = \textbf{Var}\left[\frac{\partial}{\partial\alpha} \sum_{t=0}^{H-1} \gamma^t r_t\right]\\
& = \mathcal{O}(H^4 {\color{red} L_f^{4H}}L_\pi^{4H}/M)
\end{aligned}
Bound on First-order Gradient Variance given Lipschitz Constants
Controlling Dynamic Stiffness Alleviates Variance!
\begin{aligned}
\max_H \quad & H \\
\text{s.t.} \quad & \textbf{Var}\left[\frac{\partial}{\partial\alpha} V^H_\theta(x_0,\alpha)\right] \leq C
\end{aligned}
Less Stiff Dynamics Leads to Longer Model-Based Horizon
Dynamics Smoothing
\begin{aligned}
q_1 = {\color{blue} \mathbb{E}_{w\sim \rho}}\left[f^\mathrm{u}(q_0,u_0 + {\color{blue}w})\right]
\end{aligned}
\begin{aligned}
q_1 = f^\mathrm{u}(q_0,u_0)
\end{aligned}
Non-smooth Contact Dynamics
Smooth Surrogate Dynamics
\begin{aligned}
q^\mathrm{u}_1
\end{aligned}
\begin{aligned}
u_0
\end{aligned}
No Contact
Contact
\begin{aligned}
u_0
\end{aligned}
\begin{aligned}
q^\mathrm{u}_1
\end{aligned}
Averaged
Differentiable Simulator Comes with a Knob on Smoothness
Bias-Variance Tradeoff
Horizon Expansion
Dynamic Smoothing
High Gradient Estimation Variance
Low Gradient Estimation Bias
(Due to value estimation)
Low Gradient Estimation Variance
High Gradient Estimation Bias
(due to wrong dynamics)
Bias-Variance Tradeoff
Horizon Expansion
Dynamic Smoothing
High Dynamics Bias
Low Policy Gradient Variance
High Gradient Estimation Variance
Low Gradient Estimation Bias
(Due to value estimation)
Low Gradient Estimation Variance
High Gradient Estimation Bias
(due to wrong dynamics)
High Function Estimation Bias
Low Policy Gradient Variance
"The Path of Manageable Variance"
Bias-Variance Tradeoff
Horizon Expansion
Dynamic Smoothing
High Dynamics Bias
Low Policy Gradient Variance
High Gradient Estimation Variance
Low Gradient Estimation Bias
(Due to value estimation)
Low Gradient Estimation Variance
High Gradient Estimation Bias
(due to wrong dynamics)
High Function Estimation Bias
Low Policy Gradient Variance
"The Path of Manageable Variance"
But how does annealing help?
If we only cared about low variance, might as well just run H=0 DDPG!
Does annealing help with value function estimation?
\begin{aligned}
q^\mathrm{u}_0
\end{aligned}
\begin{aligned}
q^\mathrm{a}_0
\end{aligned}
\begin{aligned}
q^\mathrm{u}_{goal}
\end{aligned}
\begin{aligned}
u
\end{aligned}
\begin{aligned}
\min_{u_0} \quad & \frac{1}{2}{\color{red} \mathbb{E}_{w\sim \rho}}\|f^\mathrm{u}(q_0,u_0 {\color{red} + w}) - q^\mathrm{u}_{goal}\|^2
\end{aligned}
Reinforcement Learning
Cost
Contact
No Contact
Averaged
\begin{aligned}
u_0
\end{aligned}
Dynamic Smoothing
\begin{aligned}
\min_{u_0} \quad & \frac{1}{2}\|{\color{red} \mathbb{E}_{w\sim \rho}}\left[f^\mathrm{u}(q_0,u_0 {\color{red} + w})\right] - q^\mathrm{u}_{goal}\|^2
\end{aligned}
Averaged
Contact
No Contact
No Contact
Previous analysis doesn't capture
dynamic smoothing benefits...
Dynamic Smoothing vs. Diffusion?
Score-based Generative Modeling
Policy Optimization through Contact
True data distribution lives on manifold, gradient is too stiff.
True contact dynamics is too stiff
Value Function Estimation
Model Value Expansion
Suppose we have a estimate of our value function / Q-function.
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\color{red}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H)}\right] \\
\end{aligned}
Model-Based Component
Model-Free Component
Q2: What's the right way to estimate the value,
1. Given that we will be using gradients,
2. Given that we chose some horizon H?
Bounding Gradient Bias
\begin{aligned}
\min_\alpha \mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right]
\end{aligned}
Original Problem (True Value)
Surrogate Problem
\begin{aligned}
\min_\alpha \mathbb{E}_{x_1\sim\eta}\left[V^H_\theta(x_1,\alpha)\right]
\end{aligned}
\begin{aligned}
\textbf{Bias}(V^H_\theta)\coloneqq \left\|\frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right] - \frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V^H_\theta(x_1,\alpha)\right]\right\|^2
\end{aligned}
Gradient Bias (Quantity of Interest)
Bounding Gradient Bias
\begin{aligned}
\textbf{Bias}(V^H_\theta)\coloneqq \left\|\frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right] - \frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V^H_\theta(x_1,\alpha)\right]\right\|^2
\end{aligned}
Gradient Bias (Quantity of Interest)
\begin{aligned}
\frac{\partial}{\partial\alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\color{red}\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial \alpha}\mathbb{E}_{w_H}Q_\theta(x_H,u_H,\alpha)}\right] \\
\end{aligned}
\begin{aligned}
\frac{\partial}{\partial \alpha} V(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\color{red}\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r(x_t,u_t)} + {\color{blue}\gamma^H \frac{\partial}{\partial\alpha}\mathbb{E}_{w_H}Q(x_H,u_H,\alpha)}\right] \\
\end{aligned}
This part is unbiased
Bounding Gradient Bias
\begin{aligned}
\textbf{Bias}(V^H_\theta)\coloneqq \left\|\frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right] - \frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V^H_\theta(x_1,\alpha)\right]\right\|^2
\end{aligned}
Gradient Bias (Quantity of Interest)
\begin{aligned}
\textbf{Bias}(V_\theta^H) & = \gamma^{2H}\left\|\mathbb{E}_{x_H\sim \mu_H}\mathbb{E}_{w_H\sim p}\left[\frac{\partial}{\partial\alpha}Q(x_H,u_H) - \frac{\partial}{\partial \alpha}Q_\theta(x_H,u_H)\right]\right\|^2 \\
\end{aligned}
\begin{aligned}
\textbf{Bias}(V_\theta^H) & = \gamma^{2H}\left\|\mathbb{E}_{x_H\sim \mu_H}\left[\frac{\partial}{\partial\alpha}V(x_H) - \frac{\partial}{\partial \alpha}V_\theta(x_H)\right]\right\|^2 \\
\end{aligned}
State Value Estimation
State-Action Value Estimation
Gradients need to match well over distribution of states at time t.
Push-forward measure from initial distribution under closed-loop dynamics
\begin{aligned}
u_H = \pi(x_H,\alpha) + w_H
\end{aligned}
Bounding Gradient Bias
Gradient Bias (Quantity of Interest)
\begin{aligned}
\textbf{Bias}(V_\theta^H) & = \gamma^{2H}\left\|\mathbb{E}_{x_H\sim \mu_H}\left[\frac{\partial}{\partial\alpha}V(x_H) - \frac{\partial}{\partial \alpha}V_\theta(x_H)\right]\right\|^2 \\
\end{aligned}
Usual Regression for Value Function Estimation
\begin{aligned}
\min_\theta \mathbb{E}_{(x,x',r)\sim\beta}\left[(V_\theta(x) - \left(r + \gamma V_\theta(x')\right)^2\right]
\end{aligned}
\begin{aligned}
\min_\theta \mathbb{E}_{(x,u,r,x',u')\sim\beta}\left[(Q_\theta(x,u) - \left(r + \gamma Q_\theta(x',u')\right)^2\right]
\end{aligned}
Sources of Bias
1. Distribution of data is from different behavior policy.
2. Accurate modeling of value does not guarantee modeling of gradients.
3. Bootstrapping bias.
Value Function Estimation
Finite-error modeling of value does not guarantee good gradients.
DDPG is equally questionable?
Value Gradient Boostrapping
\begin{aligned}
\textbf{Bias}(V^H_\theta)\coloneqq \left\|\frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V(x_1,\alpha)\right] - \frac{\partial}{\partial \alpha}\mathbb{E}_{x_1\sim\eta}\left[V^H_\theta(x_1,\alpha)\right]\right\|^2
\end{aligned}
Gradient Bias (Quantity of Interest)
\begin{aligned}
\textbf{Bias}(V_\theta^H) & = \gamma^{2H}\left\|\mathbb{E}_{x_H\sim \mu_H}\left[\frac{\partial}{\partial\alpha}V(x_H) - \frac{\partial}{\partial \alpha}V_\theta(x_H)\right]\right\|^2 \\
& = \gamma^{2H}\left\|\mathbb{E}_{x_H\sim \mu_H}\left[{\color{red}\frac{\partial V}{\partial x_H}}\frac{\partial x_H}{\partial\alpha} - {\color{red}\frac{\partial V_\theta}{\partial x_H}}\frac{\partial x_H}{\partial \alpha}\right]\right\|^2 \\
\end{aligned}
State Value Estimation
In known dynamics case, suffices to estimate dV/dx well.
Value Gradient Boostrapping
\begin{aligned}
V(x_t,\alpha) = \mathbb{E}_{w_t}\left[r_t + \gamma V(x_{t+1},\alpha)\right]
\end{aligned}
Bellman Equation
Consider the boostrapping equation for TD(0)
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|V_\theta(x_t) - \mathbb{E}_{w_t}\left[r_t + \gamma V_\theta(x_{t+1})\right]\right\|^2
\end{aligned}
Value Gradient Boostrapping
\begin{aligned}
V(x_t,\alpha) = \mathbb{E}_{w_t}\left[r_t + \gamma V(x_{t+1},\alpha)\right]
\end{aligned}
Bellman Equation
Consider the boostrapping equation for TD(0)
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|V_\theta(x_t) - \mathbb{E}_{w_t}\left[r_t + \gamma V_\theta(x_{t+1})\right]\right\|^2
\end{aligned}
\begin{aligned}
{\color{red}\frac{\partial V}{\partial x_t}} = \mathbb{E}_{w_t}\left[\frac{\partial r_t}{\partial x_t} + \frac{\partial r_t}{\partial u_t}\frac{\partial u_t}{\partial x_t} + \gamma{\color{red}\frac{\partial V}{\partial x_{t+1}}}\left[\frac{\partial x_{t+1}}{\partial x_t} + \frac{\partial x_{t+1}}{\partial u_t}\frac{\partial u_t}{\partial x_t}\right]\right]
\end{aligned}
Bellman Gradient Equation
We can boostrap the gradient directly!
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t)} - \mathbb{E}_{w_t}\left[\frac{\partial r_t}{\partial x_t} + \frac{\partial r_t}{\partial u_t}\frac{\partial u_t}{\partial x_t} + \gamma{\color{red} z_\theta(x_{t+1})}\left[\frac{\partial x_{t+1}}{\partial x_t} + \frac{\partial x_{t+1}}{\partial u_t}\frac{\partial u_t}{\partial x_t}\right]\right]\right\|^2
\end{aligned}
TD(k) Bootstrapping
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t)} - \mathbb{E}_{w_{t:t+H}\sim p}\left[\frac{\mathrm{d}}{\mathrm{d} x_t}\sum_{t'=t}^{t+H-1} \gamma^{t'-t}r_{t'} + \gamma^{H}{\color{red} z_\theta(x_{t+H})}\left[\frac{\mathrm{d} x_{t+H}}{\mathrm{d} x_t}\right]\right]\right\|^2
\end{aligned}
Actor Loss
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & \coloneqq \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r_t} + \gamma^H\frac{\partial}{\partial\alpha}V_\theta(x_H)\right] \\
& = \mathbb{E}_{w_{0:H-1}\sim p}\left[{\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r_t} + \gamma^H{\color{red} \frac{\partial V}{\partial x_H}}\frac{\partial x_H}{\partial \alpha}\right] \\
& = \mathbb{E}_{w_{0:H-1}\sim p}\left[\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r_t + \gamma^H {\color{red} z_\theta(x_{H})}\frac{\partial x_{H}}{\partial\alpha}\right]
\end{aligned}
TD(k) Gradient Boostrapping (Critic Loss), can be weighted for TD Lambda
TD(k) Bootstrapping
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t)} - \mathbb{E}_{w_{t:t+H}\sim p}\left[\frac{\mathrm{d}}{\mathrm{d} x_t}\sum_{t'=t}^{t+H-1} \gamma^{t'-t}r_{t'} + \gamma^{H}{\color{red} z_\theta(x_{t+H})}\left[\frac{\mathrm{d} x_{t+H}}{\mathrm{d} x_t}\right]\right]\right\|^2
\end{aligned}
TD(k) Gradient Boostrapping (Critic Loss), can be weighted for TD Lambda
Actor Loss
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & = \mathbb{E}_{w_{0:H-1}\sim p}\left[\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r_t + \gamma^H {\color{red} z_\theta(x_{H})}\frac{\partial x_{H}}{\partial\alpha}\right]
\end{aligned}
Critic H and Actor H does need not be the same!
Hypothesis: Actually it's beneficial to use the same H
1. Distribution of data is from different behavior policy.
2. Accurate modeling of value does not guarantee modeling of gradients.
3. Bootstrapping bias.
TD(k) Bootstrapping
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t)} - \mathbb{E}_{w_{t:t+H}\sim p}\left[\frac{\mathrm{d}}{\mathrm{d} x_t}\sum_{t'=t}^{t+H-1} \gamma^{t'-t}r_{t'} + \gamma^{H}{\color{red} z_\theta(x_{t+H})}\left[\frac{\mathrm{d} x_{t+H}}{\mathrm{d} x_t}\right]\right]\right\|^2
\end{aligned}
Sources of Bias
TD(k) Gradient Boostrapping (Critic Loss), can be weighted for TD Lambda
Gradient Boostrapping Performance
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t)} - \mathbb{E}_{w_{t:t+H}\sim p}\left[\frac{\mathrm{d}}{\mathrm{d} x_t}\sum_{t'=t}^{t+H-1} \gamma^{t'-t}r_{t'} + \gamma^{H}{\color{red} z_\theta(x_{t+H})}\left[\frac{\mathrm{d} x_{t+H}}{\mathrm{d} x_t}\right]\right]\right\|^2
\end{aligned}
Relies on Gradients from Models
High does dynamic stiffness affect estimation error?
TD(k) Gradient Boostrapping (Critic Loss), can be weighted for TD Lambda
Variance for long H and stiff dynamics tells us that
we should use less H for stiff dynamics!
Smoothing Conditioned Value Gradient
Critic Learning
Smoothing Conditioned Value Gradient
\begin{aligned}
\min_\theta \mathbb{E}_{x_t,u_t,r_t,x_{t+1}\sim\mathcal{D}}\left\|{\color{red}z_\theta(x_t,\rho)} - \mathbb{E}_{w_{t:t+H}\sim p}\left[\frac{\mathrm{d}}{\mathrm{d} x_t}\sum_{t'=t}^{t+H-1} \gamma^{t'-t}r_{t'} + \gamma^{H}{\color{red} z_\theta(x_{t+H},\rho)}\left[\frac{\mathrm{d} x_{t+H}}{\mathrm{d} x_t}\right]\right]\right\|^2
\end{aligned}
\begin{aligned}
z_\theta(x,{\color{red}\rho}) = \frac{\partial}{\partial x}V(x,{\color{red}\rho})
\end{aligned}
Actor Learning
\begin{aligned}
\frac{\partial}{\partial \alpha}V^H_\theta(x_1,\alpha) & = \mathbb{E}_{w_{0:H-1}\sim p}\left[\frac{\partial}{\partial\alpha}\sum^{H-1}_{t=0} \gamma^t r_t + \gamma^H {\color{red} z_\theta(x_{H},\rho)}\frac{\partial x_{H}}{\partial\alpha}\right]
\end{aligned}
Anneal H and rho simultaneously
(Noise Conditioned) Score Function under the
Control as Inference Interpretation
Differentiable Simulators
TinyDiffSim
gradSim
dFlex
Warp
Brax
Have my own preference on how I would implement it,
but not willing to go through the giant engineering time.
Mujoco FD
Personally have more trust in these,
but don't like soft contact models.
DiffRL
By Terry Suh
