Goal: Foundation Models for Manipulation

LLMs $\Rightarrow$ VLMs $\Rightarrow$ LBMs

large language models

visually-conditioned language models

large behavior models

$\sim$ VLA (vision-language-action)

$\sim$ EFM (embodied foundation model)

Q: Is predicting actions fundamentally different?

should we expect similar generalization / scaling-laws?

The Robot Data Diet

(the "transfer learning bet")

World-class simulator for robotics

drake.mit.edu

NVIDIA selected Drake and MuJoCo

(for potential inclusion in Omniverse)

"Hydroelastic contact" as implemented in Drake

The AlphaGo Playbook

• Step 1: Behavior Cloning
  • from human expert games
• Step 2: Self-play
  • Policy network
  • Value network
  • Monte Carlo tree search (MCTS)

Scaling Monte-Carlo Tree Search

"Graphs of Convex Sets" (GCS)

Online classes (videos + lecture notes + code)