Robotics and Generative AI
Slides forked from Russ Tedrake
Image credit: Boston Dynamics
Speaker: Shen Shen
Feb 16, 2025
Harvard-MIT Mathematics Tournament (HMMT)
What do I do?
- Teaching (right now, mostly machine learning classes)
- Research (educational tech, optimization, control, robotics)
- Service (reviewing papers, writing recommendation letters, ...)
- Hacking/coding for fun
What does a professor do at a place like MIT?
- Teaching; but only one class per term.
- Research.
- Service (organizing conferences, reviewing papers, writing recommendation letters, ...)
- Sometimes we try to start bigger initiatives
- in 2016, I helped start TRI
- So I'm also a Vice President at TRI
Physics + optimization
DARPA Robotics Competition
2015
Robots are dancing and starting to do parkour, but...
what about something more useful, like loading the dishwasher?
The Machine Learning Revolution
(for robotics; in a few slides)
What's my rule?
What's my rule?
Input
Neural Network
ImageNet: 14 Million labeled images
Released in 2009
Transfer learning
Something we couldn't have expected...
(Pre-)Training on ImageNet makes it easier to "learn" to recognize other objects
A sample annotated image from the COCO dataset
"Self-supervised" learning
Example: Text completion
No extra "labeling" of the data required!
GPT-4 is "just" doing next-word prediction
But it's trained on the entire internet...
And it's a really big network
Generative AI for Images
Humans have also put lots of captioned images on the web
...
Dall-E 2. Tested in Sept, 2022
"A painting of a professor giving a talk at a robotics competition kickoff"
Input:
Output:
Dall-E 3. Tested last night!
"A painting of a cool MIT professor delivering a lecture on robotics and generative AI at the Harvard-MIT Mathematics Tournament (HMMT) inside the Stata Center."
Input:
Output:
An image is just a list of numbers (pixel values)
Is Dall-E just next pixel prediction?
"Diffusion" models
Generative AI for robots?
Earlier this week...
Generative AI + data
+ very careful engineering
Our engineering design process
+ rigorous thinking
Haptic Teleop Interface
Excellent robot control
Cameras in the hands!
Open source:
Scaling Up
Going out into the real world...
Do you love robotics?
What can you do right now?
- Programming => Software engineering
- Physics
- Math (it's extremely important!)
- Calculus
- Linear Algebra (the foundations of machine learning)
- Machine learning tutorials online are becoming very accessible
- Robotics club!
Online classes (videos + lecture notes + code)
https://introml.mit.edu
Online classes (videos + lecture notes + code)
http://manipulation.mit.edu
http://underactuated.mit.edu
Robotics and Generative AI
By Shen Shen
Robotics and Generative AI
talk at Harvard-MIT Mathematics Tournament (HMMT)
