Deep Perception

Part 2:
Category-level representations

MIT 6.800/6.843:

Robotic Manipulation

Fall 2021, Lecture 12

Follow live at https://slides.com/d/CeVqzU8/live

(or later at https://slides.com/russtedrake/fall21-lec12)

TRI has "procedural dishes"!

TRI has "procedural dishes"

SE(3) pose is difficult to generalize across a category

https://keypointnet.github.io/

https://nanonets.com/blog/human-pose-estimation-2d-guide/

3D Keypoints provide rich, class-general semantics

Constraints & Cost on Keypoints

... and robust performance in practice

Lucas Manuelli*, Wei Gao*, Peter R. Florence and Russ Tedrake. kPAM: KeyPoint Affordances for Category Level Manipulation. ISRR 2019

Dense reconstruction + annotation GUI

Example: Keypoints for boxes

box example figures from Greg Izatt

Procedural synthetic training data

Mask-RCNN

Keypoints are not a sufficient representation

Keypoint "semantics" + dense 3D geometry 

Sample of results

(shoes on rack)

+ shape completion network (kPAM-SC)

to include collision-avoidance constraints

+ force control?

So far, keypoints are geometric and semantic 

(mug handle, front toe of shoe), but required human labels

Dense Object Nets

Core technology: dense correspondences

(built on Schmidt, Newcombe, Fox, RA-L 2017)

Peter R. Florence*, Lucas Manuelli*, and Russ Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. CoRL, 2018.

Dense Object Nets

dense 3D reconstruction

+ pixelwise contrastive loss

Good training/loss functions sharpen correspondences

Descriptors as dense self-supervised keypoints

Dense descriptors for boxes

H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, Jun. 2019, pp. 2637–2646, doi: 10.1109/CVPR.2019.00275.

Category-level Pose Estimation

"NOCS"