Goal: Foundation Models for Manipulation

TRI has a special role to play

• Expertise across robotics, ML, and software
• Resources to train large models and do rigorous evaluation
• Resources to build large high-quality datasets
• Ability to advance robot hardware

• Our charter is basic research ("invent and prove") 
• Strong tradition of open source

"Dexterous Manipulation" Team

(founded in 2016)

For the next challenge:

Good control when we don't have useful models?

For the next challenge:

Good control when we don't have useful models?

• Rules out:
  • (Multibody) Simulation
  • Simulation-based reinforcement learning (RL)
  • State estimation / model-based control
• My top choices:
  • Learn a dynamics model
  • Behavior cloning (imitation learning)

Levine*, Finn*, Darrel, Abbeel, JMLR 2016 

Key advance: visuomotor policies

perception network

(often pre-trained)

other robot sensors

learned state representation

x history

I was forced to reflect on my core beliefs...

• The value of using RGB (at control rates) as a sensor is undeniable.  I must not ignore this going forward.
   
• I don't love imitation learning (decision making $\gg$ mimcry), but it's an awfully clever way to explore the space of policy representations
  • Don't need a model
  • Don't need an explicit state representation
    • (Not even to specify the objective!)

We've been exploring, and found something good in...

Diffusion models (e.g. for image generation)

Image source: Ho et al. 2020 

great tutorial: https://chenyang.co/diffusion.html

Image backbone: ResNet-18 (pretrained on ImageNet)
Total: 110M-150M Parameters
Training Time: 3-6 GPU Days ($150-$300)

a few new skills...

Why (Denoising) Diffusion Models?

• High capacity + great performance
• Small number of demonstrations (typically ~50-100)
• Multi-modal (non-expert) demonstrations

Learns a distribution (score function) over actions

e.g. to deal with "multi-modal demonstrations"

Why (Denoising) Diffusion Models?

• High capacity + great performance
• Small number of demonstrations (typically ~50)
• Multi-modal (non-expert) demonstrations
• Training stability and consistency
  • no hyper-parameter tuning
• Generates high-dimension continuous outputs
  • vs categorical distributions (e.g. RT-1, RT-2)
  • CVAE in "action-chunking transformers" (ACT)
• Solid mathematical foundations (score functions)
• Reduces nicely to the simple cases (e.g. LQG / Youla)

Diffusion Policy for Dexterous HANDs?

Enabling technologies

Haptic Teleop Interface

Excellent system identification / robot control

Visuotactile sensing

with TRI's Soft Bubble Gripper

Open source:

https://punyo.tech/

But there are definitely limits to the single-task models

Scaling Up

LLMs $\Rightarrow$ VLMs $\Rightarrow$ LBMs

large language models

visually-conditioned language models

large behavior models

$\sim$ VLA (vision-language-action)

$\sim$ EFM (embodied foundation model)

Q: Is predicting actions fundamentally different?

should we expect similar generalization / scaling-laws?

Prediction actions is different

• We don't have internet scale action data (yet)
• We still need rigorous/scalable "Eval"

Prediction actions is different

• We don't have internet scale action data (yet)
• We need rigorous/scalable "Eval"

The Robot Data Diet

(the "transfer learning bet")

Action prediction as representation learning

In both ACT and Diffusion Policy, predicting sequences of actions seems very important

Cumulative Number of Skills Collected Over Time

The (bimanual, dexterous) TRI CAM dataset

CAM data collect

The DROID dataset  

w/ Chelsea Finn and Sergey Levine

The Robot Data Diet

Big data

Big transfer

Small data

No transfer

(the "transfer learning bet")

simulation rollouts

w/ Shuran Song

The Robot Data Diet

Big data

Big transfer

Small data

No transfer

(the "transfer learning bet")

simulation rollouts

Prismatic VLMs

w/ Dorsa Sadigh

Fine-grained evaluation suite across a number of different visual reasoning tasks

Video Diffusion

w/ Carl Vondrick

This is just Phase 1

The AlphaGo Playbook

• Step 1: Behavior Cloning
  • from human expert games
• Step 2: Self-play
  • Policy network
  • Value network
  • Monte Carlo tree search (MCTS)

Scaling Monte-Carlo Tree Search

"Graphs of Convex Sets" (GCS)

Prediction actions is different

• We don't have internet scale action data (yet)
• We need rigorous/scalable "Eval"

Eval with real robots (it's hard!)

Example: we asked the robot to make a salad...

Eval with real robots

Rigorous hardware eval (Blind, randomized testing, etc)

Simulation Eval / Benchmark

drake.mit.edu

"Hydroelastic contact" as implemented in Drake

"Hydroelastic contact" as implemented in Drake

Material Point Method 

w/ Chenfanfu Jiang

Partnership with Amazon Robotics and NVidia

NVIDIA is starting to support Drake (and MuJoCo):

• Drake OpenUSD parser
• RTX rendering
• potentially make Drake available in IsaacSym/Omniverse

Wrap-up

A foundation model for manipulation, because...

• start the data flywheel for general purpose robots
• unlock the new science of visuomotor "intelligence" (with aspects that can only be studied at scale)

Dexterous Manipulation at TRI

https://www.tri.global/careers

We are hiring! (a little)

tri.global/careers