Robotics and Generative AI

Russ Tedrake

 

Image credit: Boston Dynamics

What does a professor do at a place like MIT?

  • Teaching; but only one class per term.
  • Research.
  • Service (organizing conferences, reviewing papers, writing recommendation letters, ...)

 

  • Sometimes we try to start bigger initiatives
    • in 2016, I helped start TRI
    • So I'm also a Vice President at TRI

Q: Autonomous or Driver-controlled?

DARPA Robotics Competition

2015

Robots are dancing and starting to do parkour, but...


what about something more useful, like loading the dishwasher?

The Machine Learning Revolution

(for robotics; in a few slides)

What's my rule?

What's my rule?

Input

Neural Network

ImageNet: 14 Million labeled images

Released in 2009

Transfer learning

Something we couldn't have expected...

 

(Pre-)Training on ImageNet makes it easier to "learn" to recognize other objects

A sample annotated image from the COCO dataset

"Self-supervised" learning

Example: Text completion

No extra "labeling" of the data required!

GPT-4 is "just" doing next-word prediction

But it's trained on the entire internet...

And it's a really big network

Generative AI for Images

Humans have also put lots of captioned images on the web

...

Dall-E 2. Tested in Sept, 2022

"A painting of a professor giving a talk at a robotics competition kickoff"

Input:

Output:

Dall-E 3.  Tested last night!

"a painting of a handsome MIT professor giving a talk about robotics and generative AI at brimmer and may private school in newton, ma"

Input:

Output:

An image is just a list of numbers (pixel values)

Is Dall-E just next pixel prediction?

"Diffusion" models

Generative AI for robots?

Earlier this week...

Generative AI + data

+ very careful engineering

Our engineering design process

+ rigorous thinking

Haptic Teleop Interface

Excellent robot control

Cameras in the hands!

Open source:

https://punyo.tech/

Scaling Up

Going out into the real world...

Do you love robotics?

What can you do right now?

  • Programming => Software engineering
  • Physics
  • Math (it's extremely important!)
    • Calculus
    • Linear Algebra (the foundations of machine learning)
  • Machine learning tutorials online are becoming very accessible
  • Robotics club!

Online classes (videos + lecture notes + code)

http://manipulation.mit.edu

http://underactuated.mit.edu