BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

Motivation

Problem: Partial Observability in Contact-rich Manipulation settings

Goal: Improve Partial Observability using Contact information

* Robot data

Missing information \( {}^{*} \) 

Robot/Env Occlusions \( {}^{*} \) 

Ambiguous geometry

Motivation

Problem: Partial Observability in Contact-rich Manipulation settings

Goal: Improve Partial Observability using Contact information

Insertion attempt\({}^+\) at FoundationPose pose estimate

Insertion attempt\({}^+\) at ICP\({}^*\) pose estimate

3x

3x

* Best of 10 initialisations, \({}^+\) All insertions are performed with the same controller

BayesContact

Key Insight: Geometry conditioned Force Likelihoods improve pose estimate for fine manipulation

Insertion attempt\({}^+\) at pose estimate from Vision

Insertion attempt\({}^+\) at pose estimate from Vision and Contact

\({}^+\) All insertions are performed with the same controller

System Overview

Depth Likelihood

Depth, RGB

Pixelwise Likelihood

Sampled Image

Ground Truth

Vision Particle

Depth Likelihood

Pixelwise Likelihood

Sampled Image

Robust to missing information from noisy depth sensing

Robust to environment and robot induced occlusion

Geometry-Conditioning of F/T signals

Simulation-based Contact Likelihood

Contact Particle

Evaluate contact probe locations for \( N \) particles per measurement.

Evaluate  \( K \) contact measurements sequentially, extracting local contact information.

Results

Improved Pose Estimates \( \rightarrow \) Improved Insertion

Insertion attempts\({}^+\) with FoundationPose (FP), ICP, BayesContact Vision (BCv)

Robust to missing information from noisy depth sensing

Insertion attempts\({}^+\) with BayesContact

FP

BCv

ICP

BCv

\({}^+\) All insertions are performed with the same controller