Grand Challenges in Robotics: Dexterity

Russ Tedrake

Follow live at

(or later at

Allegro Hand by Wonik Robotics

Let's Discuss Control (for Dexterous Manipulation)

How do we synthesize "visuomotor policies" (aka pixels to torques)?

  • Behavior Cloning (imitation learning)
  • Reinforcement Learning: Online / Offline
  • Structured algorithms (\(\gg\) Stochastic Gradient Descent)

Manipulation should be easy...!

Big questions/challenges:

  • Feedback from pixels; feedback through contact
  • The "open world": Intuitive physics / state representations (what is the state of my salad, or my shirt)?
  • Fleet learning / lifelong learning

Let's Discuss Simulation (for Dexterous Manipulation)

What do we need to simulate:

  • Physics (e.g. rigid bodies, soft bodies, cloth, ...)
  • Sensors (e.g. photorealistic rendering, touch, sound, ...)
  • Diverse tasks and environments
  • Perception, planning, control, networking, ...

Big Questions/Challenges:

  • Sim2Real (Simulation fidelity / coverage)
  • Real2Sim
  • Support for rich perception, planning, and control queries
  • Levels of detail / abstraction, uncertainty quantification, testing (verification / finding rare events), ...
pip install drake
sudo apt install drake

Online lectures / course notes:

Dexterity Panel

By russtedrake

Dexterity Panel

  • 254