Michael Posa
Dynamic Autonomy and Intelligent Robotics Lab
September 12, 2025
If we cannot fully rely on memorized expertise, need to augment with online reasoning
Some steps toward contact-rich learning and control
Contact-rich
model learning
High-performance
hybrid MPC
Vision + Physics = Vysics
Vision-Based
Visible Geometry
Object Poses
Masked RGBD video
Tracking and
Reconstruction
(BundleSDF)
Bianchini*, Zhu*, et al. "Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics." RSS. 2025.
Bibit Bianchini
Minghan Zhu
Physics-Based
“Physible” Geometry
Robot Proprioception
Model Learning
Inertia
Integrated Geometry
Object URDF
- Integrate with planning and control
- Leverage unlabeled or unstructured robot or human video
- As an intermediate representation to bridge to VLMs
| Method | bakingbox | bottle | egg | milk | oatly | styro. | toble. | all |
|---|---|---|---|---|---|---|---|---|
| BundleSDF | 3.84 | 2.65 | 3.70 | 3.17 | 2.45 | 2.55 | 2.44 | 2.98 |
| 3DSGrasp | 3.83 | 2.80 | 3.78 | 3.15 | 2.51 | 2.66 | 2.77 | 3.06 |
| IPoD | 3.25 | 1.80 | 2.16 | 2.37 | 2.73 | 1.93 | 1.97 | 2.47 |
| V-PRISM | 3.52 | 2.47 | 2.31 | 3.33 | 2.30 | 2.54 | 2.48 | 2.80 |
| OctMAE | 3.11 | 2.22 | 1.52 | 2.93 | 2.13 | 2.00 | 2.36 | 2.45 |
| Vysics (ours) | 1.83 | 1.36 | 1.05 | 1.53 | 1.25 | 1.45 | 1.02 | 1.45 |
High-performance
hybrid MPC
Contact-rich MPC
Desiderata
- Online decision making for novel tasks,
- Autonomous, non-trivial mode selection and timing,
- Naturally expression of task objectives
\begin{align*} \min_{x, u} \quad & \sum g(x_k, u_k) \\ \text{s.t.} \quad & x_{k+1} = f(x_k, u_k) \end{align*}
Real-time control to simultaneously plan continuous motions and contact schedules
[Yang and P. Dynamic On-Palm Manipulation via Controlled Sliding. RSS, 2024. Outstanding Student Paper Award.]
Dynamic sliding and forceful dexterity
William Yang
Reliable and precise real-time control that repeatedly achieves arbitrary pose targets given only a 3D object model
Push Anything
object-ground
object-object
Planned Forces
end effector-object
1x
The controller plans to pivot the Letter S using contact with the book.
- 25 objects
- 700/701 success
- Avg. time to goal: 31 sec
10x
2 objects
- 100/102
- Avg. 1.6 min
15x
3 objects at a time
20x
4 objects at a time
Clutter at the limits of reasoning
Aileen Liao
Reduce problem space by pruning
for task relevancy
Clutter at the limits of reasoning
Aileen Liao
Also works with VLA's: \(\pi_{0.5}\)
The DAIR Lab
Towards "pretty good, quickly"
- Contact-rich control has come a long way in
the last few years - Learning and AI as key components in the pipeline
- Online reasoning shouldn't be tabula rasa.
-
Vision and language
-
Physics
-
Planning and control architectures
-
Pre-trained policies, value functions, etc.
-
Challenge: Seek out challenges where you have unique insight to contribute
DAIR Overview
By Michael Posa
