Meeting with Russ

Thesis Topic

- Efficient

- Model-based 

- Generalizable

- Learning

- Contact-Rich

- Dexterous

- Manipulation

- Differentiable 

1. Do Differentiable Simulators Give Better Policy Gradients?

2. Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasidynamic Contact Models

3. Motion Sets for Dexterous Contact-Rich Manipulation

4. Value Gradient Learning for Long-Horizon Manipulation

5. Generalizing beyond demonstrations with models

Efficient and Generalizable Dexterous Contact-Rich Manipulation

Graduation Timeline

June

Feb

May

Sep

Feb

June 20th

First Committee Meeting

Oct/ Nov

Second Committee Meeting

Jan/ Feb

Defense

Imitation Learning

- Foundation Models, ChatGPT for Robotics, big data, etc.

- Robotics has scarcity of data

- Co-training with sim can increase data efficiency, we can augment?

- How can we do better when we know model structure, etc.?

What is the extent to which we can generalize the same demonstration data to different settings?

Can we use more efficient use of data when we know the environment dynamics?

Project 1. Model-Based Imitation Learning

\mathcal{D}=\{(x,u,x')\}

Demonstration data

Overall goal: Imitation Learning provides a dense reward to model the long-horizon behavior, while we can do local reward-driven refinements that allow slight generalization

Example: Off-line / Off-Policy RL

\max_\pi J(\pi)

Off-line RL

"When off-line data is of 'good quality' RL can improve much faster."

Project 1. Model-Based Imitation Learning

\mathcal{D}=\{(x,u,x')\}

Demonstration data

Behavior Cloning

\pi(u|x)

- Only utilizes what action is taken given state

- But potentially more things we can learn from the data (e.g. dynamics)

Model-Based Imitation Learning

\begin{aligned} \max_{x_t,u_t}\quad& \sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

- Learns dynamics from demonstration data

- Planning to stay near demonstrations

- Stochastic Planning naturally multi-modal

Project 1. Model-Based Imitation Learning

Reward Weighting

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \alpha\sum^T_{t=1} \log p(x_t,u_t) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

Terminal Value Formulation

\begin{aligned} \max_{x_t,u_t}\quad& {\color{red}\sum^T_{t=1} r(x_t,u_t)} + \log p(x_T) \\ \text{s.t.}\quad& x_{t+1}=f(x_t,u_t) \end{aligned}

- Attempts to use additional rewards to guide local refinements when environment dynamics are known, on-line adaptation is faster than RL.

- Cannot recover if the change in the environment requires significantly different demonstrations

Project 2. What should we imitate?

Imitating slightly higher-level actions can allow us to generalize to different embodiments / objects

\mathcal{D}=\{(x,u,x')\}

Demonstration

Track higher-level actions with e.g. inverse dynamics

Watch

Understand

Extract higher-level actions 

(contact points, forces, etc.)

Do

Project 2. What should we imitate?

\mathcal{D}=\{(x,u,x')\}

Demonstration

Track higher-level actions with e.g. inverse dynamics

Watch

Understand

Extract higher-level actions 

(contact points, forces, etc.)

Do

1. How much do we need to instrument humans / build tools to understand what they did?

2. What is a good intermediate action that we can imitate?

Project 2. What should we imitate?

Version 1. Imitating Contact Points

From videos / tactile gloves + mocap, extract where the human made contact

Watch

Understand

Find out contact forces and points so that the observed understanding of the scene is physically consistent

Do

Re-target contact forces and points to a physical robot so that the same object motion can be achieved.