Robotics and Generative AI

Slides forked from Russ Tedrake

Image credit: Boston Dynamics

Speaker: Shen Shen

March 2, 2025

HackMIT Blueprint

What do I do?

  • Teaching (right now, mostly machine learning classes)
  • Research (educational tech, optimization, control, robotics)
  • Service (reviewing papers, writing recommendation letters, ...)
  • Hacking/coding for fun

 

 

What do I typically use genAI for?

  • Documentation!
  • Boilerplate code!
  • Learning new programming languages -- syntax
  • Writing scripts -- conjuring/hallucinating contrived story arc
  • Brainstorming hack/project ideas

Physics + optimization 

DARPA Robotics Competition

2015

Robots are dancing and starting to do parkour, but...


what about something more useful, like loading the dishwasher?

The Machine Learning Revolution

(for robotics; in a few slides)

What's my rule?

What's my rule?

Input

Neural Network

ImageNet: 14 Million labeled images

Released in 2009

Transfer learning

Something we couldn't have expected...

 

(Pre-)Training on ImageNet makes it easier to "learn" to recognize other objects

A sample annotated image from the COCO dataset

"Self-supervised" learning

Example: Text completion

No extra "labeling" of the data required!

GPT-4 is "just" doing next-word prediction

But it's trained on the entire internet...

And it's a really big network

Generative AI for Images

Humans have also put lots of captioned images on the web

...

Dall-E 2. Tested in Sept, 2022

"A painting of a professor giving a talk at a robotics competition kickoff"

Input:

Output:

Dall-E 3.  Tested last night!

"A painting of a cool MIT professor delivering a lecture on robotics and generative AI at the Harvard-MIT Mathematics Tournament (HMMT) inside the Stata Center."

Input:

Output:

An image is just a list of numbers (pixel values)

Is Dall-E just next pixel prediction?

"Diffusion" models

Generative AI for robots?

Earlier this week...

Generative AI + data

+ very careful engineering

Our engineering design process

+ rigorous thinking

Haptic Teleop Interface

Excellent robot control

Cameras in the hands!

Open source:

https://punyo.tech/

Scaling Up

Going out into the real world...

Do you love robotics?

What can you do right now?

  • Programming => Software engineering
  • Physics
  • Math (it's extremely important!)
    • Calculus
    • Linear Algebra (the foundations of machine learning)
  • Machine learning tutorials online are becoming very accessible
  • Robotics club!

Online classes (videos + lecture notes + code)

https://introml.mit.edu

https://slides.com/shensquared

Online classes (videos + lecture notes + code)

http://manipulation.mit.edu

http://underactuated.mit.edu