Michael Posa

Dynamic Autonomy and Intelligent Robotics Lab

September 12, 2025

If we cannot fully rely on memorized expertise, need to augment with online reasoning

Some steps toward contact-rich learning and control

Contact-rich
model learning

High-performance
hybrid MPC

Vision + Physics = Vysics

Vision-Based

Visible Geometry

Object Poses

Masked RGBD video

Tracking and
Reconstruction

(BundleSDF)

Bianchini*, Zhu*, et al. "Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics." RSS. 2025.

Bibit Bianchini

Minghan Zhu

Physics-Based

“Physible” Geometry

Robot Proprioception

Model Learning

Inertia

Integrated Geometry

Object URDF

  • Integrate with planning and control
  • Leverage unlabeled or unstructured robot or human video
  • As an intermediate representation to bridge to VLMs
Method bakingbox bottle egg milk oatly styro. toble. all
BundleSDF 3.84 2.65 3.70 3.17 2.45 2.55 2.44 2.98
3DSGrasp 3.83 2.80 3.78 3.15 2.51 2.66 2.77 3.06
IPoD 3.25 1.80 2.16 2.37 2.73 1.93 1.97 2.47
V-PRISM 3.52 2.47 2.31 3.33 2.30 2.54 2.48 2.80
OctMAE 3.11 2.22 1.52 2.93 2.13 2.00 2.36 2.45
Vysics (ours) 1.83 1.36 1.05 1.53 1.25 1.45 1.02 1.45

High-performance
hybrid MPC

Contact-rich MPC

Desiderata

  • Online decision making for novel tasks,
  • Autonomous, non-trivial mode selection and timing,
  • Naturally expression of task objectives
\begin{align*} \min_{x, u} \quad & \sum g(x_k, u_k) \\ \text{s.t.} \quad & x_{k+1} = f(x_k, u_k) \end{align*}

Real-time control to simultaneously plan continuous motions and contact schedules

[Yang and P. Dynamic On-Palm Manipulation via Controlled Sliding.  RSS, 2024. Outstanding Student Paper Award.]

Dynamic sliding and forceful dexterity

William Yang

Reliable and precise real-time control that repeatedly achieves arbitrary pose targets given only a 3D object model

Push Anything

object-ground

object-object

Planned Forces

end effector-object

1x

The controller plans to pivot the Letter S using contact with the book.

  • 25 objects
  • 700/701 success
  • Avg. time to goal:   31 sec

10x

2 objects

  • 100/102
  • Avg. 1.6 min

15x

3 objects at a time

20x

4 objects at a time

Clutter at the limits of reasoning

Aileen Liao

Reduce problem space by pruning
for task relevancy

Clutter at the limits of reasoning

Aileen Liao

Also works with VLA's: \(\pi_{0.5}\)

The DAIR Lab

Towards "pretty good, quickly"

  • Contact-rich control has come a long way in
    the last few years
  • Learning and AI as key components in the pipeline
  • Online reasoning shouldn't be tabula rasa.

    • Vision and language

    • Physics

    • Planning and control architectures

    • Pre-trained policies, value functions, etc.

Challenge: Seek out challenges where you have unique insight to contribute