Reinforcement Learning

(part 1)

MIT 6.881: Robotic Manipulation

Fall 2020, Lecture 17

Follow live at https://slides.com/russtedrake/fall20-lec17/live

(or later at https://slides.com/russtedrake/fall20-lec17)

OpenAI - Learning Dexterity

Recipe:

Make the simulator
Write cost function
Deep policy gradient

From plans to policies

From planning to policies

The MIT Leg Lab Hopping Robots

http://www.ai.mit.edu/projects/leglab/robots/robots.html

Visuomotor policies

Levine*, Finn*, Darrel, Abbeel, JMLR 2016

this slide is from Sean Meyn's talk at Simon's RL Bootcamp, Sept 2020

Ex: OpenAI Gym

import gym
from gym import error, spaces, utils
from gym.utils import seeding

class FooEnv(gym.Env):
  metadata = {'render.modes': ['human']}

  def __init__(self):
    ...
  def step(self, action):
    ...
  def reset(self):
    ...
  def render(self, mode='human'):
    ...
  def close(self):
    ...

http://gym.openai.com/

import pydrake.all


builder = DiagramBuilder()
....
diagram = builder.Build()
simulator = Simulator(diagram)


simulator.AdvanceTo(...)
observation = sensor_output_port->Eval(context)
reward = reward_output_port->Eval(context)


context = diagram.CreateDefaultContext()


meshcat.Publish(context)

"Domain Randomization"

(Image source: Tobin et al, 2017)

CMA-ES

https://en.wikipedia.org/wiki/CMA-ES

Keypoints for picking up a plate (in 2D)

Why/when does gradient-based policy search work?

OpenAI - Learning Dexterity

"PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance."

https://openai.com/blog/openai-baselines-ppo/

Should we expect this to work?

Do we need the over-parameterization of deep policies?
- Is there a comparable story to interpolating solutions in high-dimensional policy space?

Lessons from Control

(better controller parameterizations?)

\min \int_0^\infty dt \left[x'Qx + u'Ru\right], \qquad \dot{x} = Ax + Bu

u^* = -Kx

Parameterizations for LQR

But there are also cases where it will not work...

{\bf A} = \begin{bmatrix} 0 & 0 & 2 \\ 1 & 0 & 0 \\ 0 & 1 & 0\end{bmatrix}, \quad {\bf B} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \quad {\bf C} = \begin{bmatrix} 1 & 1 & 3 \end{bmatrix},

\dot{x} = {\bf A}x + {\bf B} u, \quad y = {\bf C}x,

u = -ky.

A simple counter-example from static output feedback:

http://underactuated.mit.edu/policy_search.html

The set of stabilizing \(k\) is a disconnected set.

k	Maximum real closed-loop eigenvalue
0.9	-0.035
1.5	0.032
2.1	-0.009

Youla parameterization (disturbance-based feedback)
LMI formulations

Convex formulations are for linear systems, but the benefits are likely more general.

For LQG, \(H_\infty\), etc., we know convex parameterizations.

Youla parameters (time-domain, no-noise)

http://underactuated.csail.mit.edu/lqr.html

Disturbance-based output feedback

HSCC, 2020

arXiv:2001.09254

Simple example: for linear Gaussian, this will recover the (unrolled) LQG controller.

(Kalman gains + LQR, up to the similarity transform)

Just because you can search over \(u = -Kx\) directly, does not mean that you should!
- Convergence rates??
- Set of stabilizing \(K\) is nontrivial
Example: Search over \(Q\) and \(R\), or convex parameterization.

ADPRL, 2012

paper link

There is still a big gap between these results and control for contact. (Contact => non-linear)

Maybe it is the distribution over tasks?
- Control has traditionally studied algorithms that must work for all A,B,C. Maybe the world never gives us the hard ones?
- Optimizing simultaneously over diverse tasks might be easier than optimizing over one task.

https://forms.gle/BwRmUsFQ51Z5bzeT9

Lecture 17: Reinforcement Learning (part 1)