How do we synthesize "visuomotor policies" (aka pixels to torques)?

• Behavior Cloning (imitation learning)
• Reinforcement Learning: Online / Offline
• Structured algorithms (\(\gg\) Stochastic Gradient Descent)

Manipulation should be easy...!

Big questions/challenges:

• Feedback from pixels; feedback through contact
• The "open world": Intuitive physics / state representations (what is the state of my salad, or my shirt)?
• Fleet learning / lifelong learning

What do we need to simulate:

• Physics (e.g. rigid bodies, soft bodies, cloth, ...)
• Sensors (e.g. photorealistic rendering, touch, sound, ...)
• Diverse tasks and environments
• Perception, planning, control, networking, ...

Big Questions/Challenges:

• Sim2Real (Simulation fidelity / coverage)
• Real2Sim
• Support for rich perception, planning, and control queries
• Levels of detail / abstraction, uncertainty quantification, testing (verification / finding rare events), ...