Russ Meeting

Thesis Topic

- Efficient

- Model-based

- Generalizable

- Learning

- Contact-Rich

- Dexterous

- Manipulation

- Differentiable

1. Do Differentiable Simulators Give Better Policy Gradients?

2. Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasidynamic Contact Models

3. Motion Sets for Dexterous Contact-Rich Manipulation

4. Value Gradient Learning for Long-Horizon Manipulation

5. Generalizing beyond demonstrations with models

Efficient and Generalizable Dexterous Contact-Rich Manipulation

Project 1. Model-Based Imitation Learning

\mathcal{D}=\{(x,u,x')\}

\mathcal{D}=\{(x,u,x')\}

Demonstration data

Overall goal: Imitation Learning provides a dense reward to model the long-horizon behavior, while we can do local reward-driven refinements that allow slight generalization

Example: Off-line / Off-Policy RL

\max_\pi J(\pi)

\max_\pi J(\pi)

Off-line RL

"When off-line data is of 'good quality' RL can improve much faster."

Project 1. Model-Based Imitation Learning

\mathcal{D}=\{(x,u,x')\}

\mathcal{D}=\{(x,u,x')\}

Demonstration data

Behavior Cloning

\pi(u|x)

\pi(u|x)

- Only utilizes what action is taken given state

- But potentially more things we can learn from the data (e.g. dynamics)

Model-Based Imitation Learning

\begin{aligned} \max_{x_t,u_t}\quad& \sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

\begin{aligned} \max_{x_t,u_t}\quad& \sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

- Learns dynamics from demonstration data

- Planning to stay near demonstrations

- Stochastic Planning naturally multi-modal

Project 1. Model-Based Imitation Learning

Reward Weighting

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \alpha\sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \alpha\sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

Terminal Value Formulation

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \log p(x_T) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \log p(x_T) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

- Attempts to use additional rewards to guide local refinements when environment dynamics are known, on-line adaptation is faster than RL.

- Cannot recover if the change in the environment requires significantly different demonstrations

Project 2. What should we imitate?

Imitating slightly higher-level actions can allow us to generalize to different embodiments / objects

\mathcal{D}=\{(x,u,x')\}

\mathcal{D}=\{(x,u,x')\}

Demonstration

Track higher-level actions with e.g. inverse dynamics

Watch

Understand

Extract higher-level actions

(contact points, forces, etc.)

Do

Project 2. What should we imitate?

\mathcal{D}=\{(x,u,x')\}

\mathcal{D}=\{(x,u,x')\}

Demonstration

Track higher-level actions with e.g. inverse dynamics

Watch

Understand

Extract higher-level actions

(contact points, forces, etc.)

Do

1. How much do we need to instrument humans / build tools to understand what they did?

2. What is a good intermediate action that we can imitate?

Meeting with Russ