Robotics and
Generative AI

Russ Tedrake

VP, Robotics Research

Dexterous Manipulation at TRI

My story

  • Undergrad at UMich
  • PhD at MIT
  • Professor at MIT since 2005.


  • in 2016, I helped start TRI

DARPA Robotics Competition, 2015

Robots are dancing and doing parkour.


Now computer vision is really starting  to work...

can they load the dishwasher?

The newest Machine Learning Revolution: Generative AI

for robotics; in a few slides

ImageNet: 14 Million labeled images

Released in 2009

"Self-supervised" learning

Example: Text completion

No extra "labeling" of the data required!

GPT-4 is "just" doing next-word prediction

But it's trained on the entire internet...

And it's a really big network

Generative AI for Images

Humans have also put lots of captioned images on the web


Dall-E 2. Tested in Sept, 2022

"A painting of a professor giving a talk at a robotics competition kickoff"



Dall-E 3.  Sept, 2023

"a painting of a handsome MIT professor giving a talk about robotics and generative AI at brimmer and may school in newton, ma"



An image is just a list of numbers (pixel values)

Is Dall-E just next pixel prediction?

"Diffusion" models

Generative AI for robots?

Generative AI + data

+ very careful engineering

Our engineering design process

+ rigorous thinking

Haptic Teleop Interface

Excellent robot control

Cameras in the hands!

Open source:

LLMs \(\Rightarrow\) VLMs \(\Rightarrow\) LBMs

large language models

visually-conditioned language models

large behavior models

\(\sim\) VLA (vision-language-action)

\(\sim\) EFM (embodied foundation model)

Q: Is predicting actions fundamentally different?

Why actions (for dexterous manipulation) could be different:

  • Actions are continuous (language tokens are discrete)
  • Have to obey physics, deal with stochasticity
  • Feedback / stability
  • ...

should we expect similar generalization / scaling-laws?

Success in (single-task) behavior cloning suggests that these are not blockers

Prediction actions is different

  • We don't have internet scale action data (yet)
  • We still need rigorous/scalable "Eval"

The Robot Data Diet

Big data

Big transfer

Small data

No transfer

 robot teleop

(the "transfer learning bet")


simulation rollouts

novel devices

Cumulative Number of Skills Collected Over Time

The (bimanual, dexterous) TRI CAM dataset

CAM data collect

TRI has a special role to play

  • Expertise across robotics, ML, and software
  • Resources to train large models and do rigorous evaluation
  • Resources to build large high-quality datasets
  • Ability to advance robot hardware


  • Our charter is basic research ("invent and prove") 
  • Strong tradition of open source

+ Amazing university partners

Just one piece of TRI's robotics portfolio...

Online classes (videos + lecture notes + code)

BiR visits TRI Cambridge

By russtedrake

BiR visits TRI Cambridge

  • 161