July 22, 2025
Michael Posa
University of Pennsylvania
Predicting the future
- Pretty good, quickly vs. really good eventually
-
Online reasoning shouldn't be tabula rasa. What priors can you bring?
- Vision and language
- Physics
- Planning and control architectures
- Pre-trained policies, value functions, etc.
If we cannot fully rely on memorized expertise, need to augment with online reasoning
Some steps toward contact-rich learning and control
Contact-rich
model learning
High-performance
hybrid MPC
Vision + Physics = Vysics
Vision-Based
Visible Geometry
Object Poses
Masked RGBD video
Tracking and
Reconstruction
(BundleSDF)
Bianchini*, Zhu*, et al. "Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics." RSS. 2025.
Bibit Bianchini
Minghan Zhu
Physics-Based
“Physible” Geometry
Robot Proprioception
Model Learning
Inertia
Integrated Geometry
Object URDF
- Integrate with planning and control
- Leverage unlabeled or unstructured robot or human video
- As an intermediate representation to bridge to VLMs
| Method | bakingbox | bottle | egg | milk | oatly | styro. | toble. | all |
|---|---|---|---|---|---|---|---|---|
| BundleSDF | 3.84 | 2.65 | 3.70 | 3.17 | 2.45 | 2.55 | 2.44 | 2.98 |
| 3DSGrasp | 3.83 | 2.80 | 3.78 | 3.15 | 2.51 | 2.66 | 2.77 | 3.06 |
| IPoD | 3.25 | 1.80 | 2.16 | 2.37 | 2.73 | 1.93 | 1.97 | 2.47 |
| V-PRISM | 3.52 | 2.47 | 2.31 | 3.33 | 2.30 | 2.54 | 2.48 | 2.80 |
| OctMAE | 3.11 | 2.22 | 1.52 | 2.93 | 2.13 | 2.00 | 2.36 | 2.45 |
| Vysics (ours) | 1.83 | 1.36 | 1.05 | 1.53 | 1.25 | 1.45 | 1.02 | 1.45 |
- What's next: Vision + Physics + Data
- General object priors in 2D and 3D generative models.
- Takeaway:
- We need physically grounded robot vision for robots.
- Contact dynamics is a critical component.
- Robot vision needs roboticists!
- We need physically grounded robot vision for robots.
High-performance
hybrid MPC
Contact-rich MPC
Desiderata
- Online decision making for novel tasks,
- Autonomous, non-trivial mode selection and timing,
- Naturally expression of task objectives
\begin{align*} \min_{x, u} \quad & \sum g(x_k, u_k) \\ \text{s.t.} \quad & x_{k+1} = f(x_k, u_k) \end{align*}
Real-time control to simultaneously plan continuous motions and contact schedules
[Yang and P. Dynamic On-Palm Manipulation via Controlled Sliding. RSS, 2024. Outstanding Student Paper Award.]
Dynamic sliding and forceful dexterity
William Yang
Reliable and precise real-time control that repeatedly achieves arbitrary pose targets given only a 3D object model
DAIR Meta
By Michael Posa
