July 22, 2025

 

 

 

 

Michael Posa

University of Pennsylvania

 

 

 

 

​

Predicting the future

  • Pretty good, quickly vs. really good eventually
  • Online reasoning shouldn't be tabula rasa. What priors can you bring?
    • Vision and language
    • Physics
    • Planning and control architectures
    • Pre-trained policies, value functions, etc.

If we cannot fully rely on memorized expertise, need to augment with online reasoning

Some steps toward contact-rich learning and control

Contact-rich
model learning

High-performance
hybrid MPC

Vision + Physics = Vysics

Vision-Based

Visible Geometry

Object Poses

Masked RGBD video

Tracking and
Reconstruction

(BundleSDF)

Bianchini*, Zhu*, et al. "Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics." RSS. 2025.

Bibit Bianchini

Minghan Zhu

Physics-Based

“Physible” Geometry

Robot Proprioception

Model Learning

Inertia

Integrated Geometry

Object URDF

  • Integrate with planning and control
  • Leverage unlabeled or unstructured robot or human video
  • As an intermediate representation to bridge to VLMs
Method bakingbox bottle egg milk oatly styro. toble. all
BundleSDF 3.84 2.65 3.70 3.17 2.45 2.55 2.44 2.98
3DSGrasp 3.83 2.80 3.78 3.15 2.51 2.66 2.77 3.06
IPoD 3.25 1.80 2.16 2.37 2.73 1.93 1.97 2.47
V-PRISM 3.52 2.47 2.31 3.33 2.30 2.54 2.48 2.80
OctMAE 3.11 2.22 1.52 2.93 2.13 2.00 2.36 2.45
Vysics (ours) 1.83 1.36 1.05 1.53 1.25 1.45 1.02 1.45
  • What's next: Vision + Physics + Data
    • General object priors in 2D and 3D generative models.
  • Takeaway:
    • We need  physically grounded robot vision for robots.
      • Contact dynamics is a critical component. 
    • Robot vision needs roboticists!

High-performance
hybrid MPC

Contact-rich MPC

Desiderata

  • Online decision making for novel tasks,
  • Autonomous, non-trivial mode selection and timing,
  • Naturally expression of task objectives
\begin{align*} \min_{x, u} \quad & \sum g(x_k, u_k) \\ \text{s.t.} \quad & x_{k+1} = f(x_k, u_k) \end{align*}

Real-time control to simultaneously plan continuous motions and contact schedules

[Yang and P. Dynamic On-Palm Manipulation via Controlled Sliding.  RSS, 2024. Outstanding Student Paper Award.]

Dynamic sliding and forceful dexterity

William Yang

Reliable and precise real-time control that repeatedly achieves arbitrary pose targets given only a 3D object model