russtedrake PRO
Roboticist at MIT and TRI
Reinforcement Learning
(part 1)
MIT 6.881: Robotic Manipulation
Fall 2020, Lecture 17
Follow live at https://slides.com/russtedrake/fall20-lec17/live
(or later at https://slides.com/russtedrake/fall20-lec17)
OpenAI - Learning Dexterity
Recipe:
- Make the simulator
- Write cost function
- Deep policy gradient
From plans to policies
From planning to policies
The MIT Leg Lab Hopping Robots
http://www.ai.mit.edu/projects/leglab/robots/robots.html
Visuomotor policies
Levine*, Finn*, Darrel, Abbeel, JMLR 2016
this slide is from Sean Meyn's talk at Simon's RL Bootcamp, Sept 2020
this slide is from Sean Meyn's talk at Simon's RL Bootcamp, Sept 2020
this slide is from Sean Meyn's talk at Simon's RL Bootcamp, Sept 2020
Ex: OpenAI Gym
import gym
from gym import error, spaces, utils
from gym.utils import seeding
class FooEnv(gym.Env):
metadata = {'render.modes': ['human']}
def __init__(self):
...
def step(self, action):
...
def reset(self):
...
def render(self, mode='human'):
...
def close(self):
...http://gym.openai.com/
import pydrake.all
builder = DiagramBuilder()
....
diagram = builder.Build()
simulator = Simulator(diagram)
simulator.AdvanceTo(...)
observation = sensor_output_port->Eval(context)
reward = reward_output_port->Eval(context)
context = diagram.CreateDefaultContext()
meshcat.Publish(context)
"Domain Randomization"
(Image source: Tobin et al, 2017)
CMA-ES
https://en.wikipedia.org/wiki/CMA-ES
Keypoints for picking up a plate (in 2D)
Why/when does gradient-based policy search work?
OpenAI - Learning Dexterity
"PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance."
https://openai.com/blog/openai-baselines-ppo/
Should we expect this to work?
- Do we need the over-parameterization of deep policies?
- Is there a comparable story to interpolating solutions in high-dimensional policy space?
Lessons from Control
(better controller parameterizations?)
\min \int_0^\infty dt \left[x'Qx + u'Ru\right], \qquad \dot{x} = Ax + Bu
u^* = -Kx
Parameterizations for LQR
But there are also cases where it will not work...
{\bf A} = \begin{bmatrix} 0 & 0 & 2 \\ 1 & 0 & 0 \\ 0 & 1 & 0\end{bmatrix}, \quad {\bf B} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \quad {\bf C} = \begin{bmatrix} 1 & 1 & 3 \end{bmatrix},
\dot{x} = {\bf A}x + {\bf B} u, \quad y = {\bf C}x,
u = -ky.
A simple counter-example from static output feedback:
http://underactuated.mit.edu/policy_search.html
The set of stabilizing \(k\) is a disconnected set.
|
k |
Maximum real closed-loop eigenvalue |
|---|---|
| 0.9 | -0.035 |
| 1.5 | 0.032 |
| 2.1 | -0.009 |
- Youla parameterization (disturbance-based feedback)
- LMI formulations
- Convex formulations are for linear systems, but the benefits are likely more general.
For LQG, \(H_\infty\), etc., we know convex parameterizations.
Youla parameters (time-domain, no-noise)
http://underactuated.csail.mit.edu/lqr.html
Disturbance-based output feedback
HSCC, 2020
Simple example: for linear Gaussian, this will recover the (unrolled) LQG controller.
(Kalman gains + LQR, up to the similarity transform)
- Just because you can search over \(u = -Kx\) directly, does not mean that you should!
- Convergence rates??
- Set of stabilizing \(K\) is nontrivial
- Example: Search over \(Q\) and \(R\), or convex parameterization.
ADPRL, 2012
- There is still a big gap between these results and control for contact. (Contact => non-linear)
-
Maybe it is the distribution over tasks?
- Control has traditionally studied algorithms that must work for all A,B,C. Maybe the world never gives us the hard ones?
- Optimizing simultaneously over diverse tasks might be easier than optimizing over one task.
Lecture 17: Reinforcement Learning (part 1)
By russtedrake
Lecture 17: Reinforcement Learning (part 1)
MIT Robotic Manipulation Fall 2020 http://manipulation.csail.mit.edu
