Active Tactile Exploration for Rigid Body Pose and Shape Estimation

Ethan K. Gordon, Bruke Baraki, Hien Bui, Michael Posa

Only tactile data is used to find the pose and geometry of an arbitrary dynamic convex object.

Project Website:

dairlab.github.io/activetactile

Learning a physical model online can improve data efficiency, predictability, and reuse between tasks.
Previous Tactile System Identification Work: Static Objects and/or Strong Shape Priors and/or 2D
Learning: apply ContactNets-style violation-implicit learning (Pfommer, CoRL 2020) to pose estimation, avoiding the numerical stiffness inherent in rigid-body contact dynamics.
We learn cuboid and convex polyhedra with less than 10s of randomly collected data.
Exploration: maximizing Expected Info Gain (EIG) leads to significantly faster learning.

Challenge: Use only tactile data to find the pose and geometry of an arbitrary dynamic convex object.

Modified Trifinger Robot
Densetact 2.0 (Do, 2023)
Contact Boolean \(c_t\) and Normal \(\hat{n}_t\) are computed with Optical Flow and a Helmholz-Hodge Decomposition
FoundationPose for Ground-Truth only

Each action: 1s motion towards estimated object state
Random Baseline: Randomize approach direction
EIG (Ours): Choose the approach to maximize EIG, sampling-based optimization with Gaussian Cross-Entropy Method

(Above) Example learning curves. Observed information increases with more data.
(Below) Real robot results

Conclusion

Identity Jacobian EIG showed significant improvement over object-directed actions from random approach directions.
Next Steps: expand experiment to future work (marginalize + sample) EIG formulation.

Difficulty: gradients of dynamics \(f\) are near-0 or near-\(\infty\), which is not amenable to learning.
Solution: add inner optimization over contact forces \(\lambda\). Trade-off: solving a (fast) QP each gradient step for better gradients.

\(\mathcal{L} = -\log p(m_t | \theta, x_{0\ldots T}) =\sum_t -\log p(m_t | \theta, x_t) + ||x_t - f_\theta(x_{t-1}, \lambda)||^2\)

Where \(\lambda = \min g_\theta(x, \lambda)\)

\(\mathcal{L} = \sum_t \min_\lambda -\log p(m_t | \theta, x_t) + ||x_t - f_\theta(x_{t-1})||^2 + g_\theta(x, \lambda)\)

Physics Constrained MLE with Trajectory Optimization:

Violation-Implicit (VIMP) Loss:

\(\mathcal{I} = \sum_{m_t} \left(\nabla_{\theta, x_T}\log p(m_t|\theta, x_T)\right)^2\)

\(\mathcal{F} = \mathbb{E}_{m_t}\left[\mathcal{I}(m_t, \theta, x_{H>T})\right]\)

Sample future actions, Simulate to estimate \(\mathcal{F}\), then maximize Expected Information Gain (EIG)

\(EIG := \log\det\left(\mathcal{F}\mathcal{I}^{-1} + \mathbf{I}\right)\)

w.r.t. Current Time T

w.r.t. Future Time H

Challenge: Compute \(\nabla \log p(m_t | x_T)\), i.e. sensitivity of past measurements to future states

Rejected: Backwards Simulation

\(\ldots =\nabla \log p(m_t|x_t(x_T))\)

Ill-defined for frictional contact.

Ours: Identity Jacobian

\(\nabla_{x_T}x_t = \nabla_{x_t}x_T = \mathbb{I}\)

Note: Treats object as quasi-static

Marginalize + Sample

\(\ldots \approx \nabla \log \sum_{x_t}p(m_t|x_t)p(x_t|x_T)\)

Sample \(x_t\) with MCMC

Use vimp loss \(\mathcal{L}\)

\(\approx softmax_{x_t}(\log p(m_t|x_t)) \cdot \nabla\mathcal{L}\)

Fast sampling with learned trajectory

Future Work