Diffusion Policies and

Graphs of Convex Sets

Russ Tedrake

USC CPS Seminar

October 10, 2023

A golden age for robotics

https://code-as-policies.github.io/

"Dexterous Manipulation" Team

(founded in 2016)

For the next challenge:

Good control when we don't have useful models?

For the next challenge:

Good control when we don't have useful models?

• Rules out:
  • Simulation
  • Reinforcement learning (in practice)
  • State Estimation / Model-based control
• Two natural choices:
  • Learn a dynamics model
  • Behavior cloning (imitation learning)

Levine*, Finn*, Darrel, Abbeel, JMLR 2016 

Visuomotor policies

perception network

(often pre-trained)

other robot sensors

From this morning...

Diffusion models (generative AI)

Image source: Ho et al. 2020 

Learns a distribution (score function) over actions

e.g. to deal with "multi-modal demonstrations"

ResNet-18 (pretrained on ImageNet)
110M-150M Parameters
Training Time: 3-6 GPU Days ($150-$300)

Why Denoising Diffusion Models?

• High capacity + great performance
• Small number of demonstrations (typically ~50)
• Multi-modal (non-expert) demonstrations
• Training stability and consistency
  • no hyper-parameter tuning
• Generates high-dimension continuous outputs
  • vs categorical distributions (e.g. RT-1, RT-2)
  • Action-chunking transformers (ACT)
• Solid mathematical foundations (score functions)
• Reduces nicely to the simple cases (e.g. LQG / Youla)

Enabling technologies

Haptic Teleop Interface

Excellent system identification / robot control

Visuotactile sensing

with TRI's Soft Bubble Gripper

Open source:

https://punyo.tech/

Scaling Up

• I've discussed training one skill
• Wanted: few shot generalization to new skills
  • multitask, language-conditioned policies
  • connects beautifully to internet-scale data

• Big Questions:
  • How do we feed the data flywheel?
  • What are the scaling laws?

• I don't see any immediate ceiling

There is so much more to understand/do!

(manipulation is not solved)

Advanced contact simulation

http://drake.mit.edu

• Hydroelastic contact models
• Fast/stable contact solvers
• New superfast soft-body (FEM) solver
• (Note: we run to convergence)

Even in simulation (e.g. given a model), I cannot reliably achieve this level of performance w/o a teacher;

Motion Planning around Obstacles with Convex Optimization.

Tobia Marcucci, Mark Petersen, David von Wrangel, Russ Tedrake.

Available at: https://arxiv.org/abs/2205.04422​

Accepted for publication in Science Robotics

A new approach to motion planning

Claims:

• Find better plans faster than sampling-based planners
• Avoid local minima from trajectory optimization
• Can guarantee paths are collision-free
• Naturally handles dynamic limits/constraints
• Scales to big problems (e.g. multiple arms)

Default playback at .25x

Scaling

• ~10k regions in 3D
   
• 20k vertices and 400k edges.
   
• Online planning takes 0.3s

by Tobia Marcucci in collaboration w/ Stephen Boyd

A rich intellectual framework

• GCS Trajectory optimization (Collision-free motion planning at the dynamic limits)
• The bigger view of GCS:
  • Planning on manifolds / planning through contact.
  • Task and Motion Planning / Multi-modal motion planning.
    • GCS \(\gg\) shortest path problems (permutahedron, etc).
  • Dynamics.  LQR \(\Rightarrow\) PWAQR; Footstep planning.
  • GCS as a policy / value function.
  • Sequential composition of skills (e.g. LQR-trees + GCS).
  • Output feedback.

GCS Trajectory Optimization

Sampling-based motion planning

The Probabilistic Roadmap (PRM)
from Choset, Howie M., et al. Principles of robot motion: theory, algorithms, and implementation. MIT press, 2005.

• PRM: A roadmap of points (vertices) connected by line segments
• GCS: A roadmap of convex sets (vertices) of continuous curves connected by continuity constraints

Key ingredients

• Efficient algorithms for solving shortest paths on "Graphs of Convex Sets" (aka "GCS")
• Transcription of the trajectory optimization problem into GCS
  • (Approximate) convex decomposition of collision-free configuration space.  (aka "IRIS")
  • Convex optimization over continuous curves w/ time scaling (\(\Rightarrow\) "GCS Trajectory Optimization")

Graphs of Convex Sets

• For each \(i \in V:\)
  • Compact convex set \(X_i \subset \R^d\)
  • A point \(x_i \in X_i \) 
• Edge length given by a convex function \[ \ell(x_i, x_j) \]

Graphs of Convex Sets

Mixed-integer formulation with a very tight convex relaxation

• Efficient branch and bound, or
• In practice we only solve the convex relaxation and round

IRIS for Configuration-space regions

• IRIS (Fast approximate convex segmentation).  Deits and Tedrake, 2014
• New versions work in configuration space:
  • Nonlinear optimization for speed. IRIS-NP
  • Sums-of-squares for rigorous certification. C-IRIS

Convex decomposition of configuration space

• Insight: Cliques in the visibility graph (almost) correspond to convex sets in configuration space
• Approach: (Approximate) minimum clique cover

(Approximate) Minimum clique cover

GCS Trajectory Optimization

GCS Trajectory Optimization

Transitioning from basic research to real use cases

Adoption

Adoption

Dave Johnson (CEO): "wow -- GCS (left) is a LOT better! ... This is a pretty special upgrade which is going to become the gold standard for motion planning."

Planning Through Contact with GCS

Going beyond collision-free motion planning...

Formulation

Using GCS for Planning Through Contact

• The convex relaxation doesn't have to be perfect
• \( \implies \) Only needs to contain enough information to take the correct high-level decisions
• Then we can refine the details as a final step!

or

Start

Goal

Planar pushing (Push T)

More general manipulation

Task and motion planning

Task and motion planning with GCS

Prelimary results by Savva Morozov

As a motion planning tool

​This is version 0.1 of a new framework.

• Already competitive (better paths faster; higher DOF; supports differential constraints)
• We've provided a mature implementation

A rich intellectual framework

• GCS Trajectory optimization (Collision-free motion planning at the dynamic limits)
• The bigger view of GCS:
  • Planning on manifolds / planning through contact.
  • Task and Motion Planning / Multi-modal motion planning.
    • GCS \(\gg\) shortest path problems (permutahedron, etc).
  • Dynamics.  LQR \(\Rightarrow\) PWAQR; Footstep planning.
  • GCS as a policy / value function.
  • Sequential composition of skills (e.g. LQR-trees + GCS).
  • Output feedback.

Summary

• Dexterous manipulation is still unsolved, but progress is fast
• Visuomotor diffusion policies
  • via imitation learning from humans
  • via advanced simulation + planning and control

• Much of our code is open-source:

drake.mit.edu

Online classes (videos + lecture notes + code)

drake.mit.edu