Category-Level Manipulation

MIT 6.881: Robotic Manipulation

Fall 2020, Lecture 14

Follow live at https://slides.com/russtedrake/fall20-lec14/live

(or later at https://slides.com/russtedrake/fall20-lec14)

"Category-Level" Manipulation

A useful first step...

TRI has "procedural dishes"!

TRI has "procedural dishes"

SE(3) pose is difficult to generalize across a category

So how do we even specify the task?

What's the cost function?

(Images of mugs on the rack?)

H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, Jun. 2019, pp. 2637–2646, doi: 10.1109/CVPR.2019.00275.

"Category-level Pose Estimation"

kPAM: KeyPoint Affordances for Manipulation

Lucas Manuelli*, Wei Gao*, Peter R. Florence and Russ Tedrake. kPAM: KeyPoint Affordances for Category Level Manipulation. ISRR 2019

Problem Statement

Manipulate potentially unknown rigid objects from a category (e.g. mugs, shoes) into desired target configurations.

https://keypointnet.github.io/

https://nanonets.com/blog/human-pose-estimation-2d-guide/

3D Keypoints provide rich, class-general semantics

Constraints & Cost on Keypoints

... and robust performance in practice

Lucas Manuelli*, Wei Gao*, Peter R. Florence and Russ Tedrake. kPAM: KeyPoint Affordances for Category Level Manipulation. ISRR 2019

Inverse kinematics as an optimization

\min_q | q-q_{desired}|

subject to:

  • rich end-effector constraints
  • joint limits
  • collision avoidance
  • "gaze constraints"
  • "feet stay put"
  • balance (center of mass)
  • ...

Keypoints are not a sufficient representation

Keypoint "semantics" + dense 3D geometry 

kPAM pipeline

No template model nor pose appears in this pipeline.

Custom annotation

tool

RGBD image w/ instance segmentation

Grasp

Planner

3D Keypoint Detection Network

Image

3D

Inverse Kinematics Planner

Keypoint network

Architecture based on Sun, Xiao, et al. "Integral human pose regression." ECCV, 2018

Custom annotation tool

Sample of results

(shoes on rack)

# train objects 10
# test objects 20
# trials 100
placed on shelf 98%
heel error (cm) 1.09 ± (1.29)
toe error (cm) 4.34 ± (3.05)

+ shape completion network (kPAM-SC)

to include collision-avoidance constraints

+ force control?

So far, keypoints are geometric and semantic 

(mug handle, front toe of shoe), but required human labels

If we forgo semantics, can we self-supervise?

Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese, “KETO: Learning Keypoint Representations for Tool Manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp. 7278–7285

Dense Object Nets

Core technology: dense correspondences

(built on Schmidt, Newcombe, Fox, RA-L 2017)

Peter R. Florence*, Lucas Manuelli*, and Russ Tedrake. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation. CoRL, 2018.

Dense Object Nets

dense 3D reconstruction

+ pixelwise contrastive loss

New loss function sharpens correspondences

Now 3D correspondences, trained with multiview

Dense descriptors as self-supervised keypoints

Correspondences alone are sufficient to specify some tasks

Dense Object Nets in Visuomotor Imitation Learning

Peter R. Florence, Lucas Manuelli, and Russ Tedrake. Self-Supervised Correspondence in Visuomotor Policy Learning. RA-L, April 2020

Visuomotor policies

Levine*, Finn*, Darrel, Abbeel, JMLR 2016 

Training the visual front-end (prior work)

  • Pose-based  auxiliary loss

 

  • Auto-encoding

 

 

  • End-to-end

Estimate object/hand pose

(but hard for category-level)

Idea: use dense descriptors

Idea: Use small set of dense descriptors

Imitation learning setup

from hand-coded policies in sim

and teleop on the real robot

Standard "behavior-cloning" objective + data augmentation

Simulation experiments

"push box"

"flip box"

Policy is a small LSTM network (~100 LSTMs)

< demonstrations?

> generalization?

Dense Object Nets for Model-Predictive Control (MPC)

Learn descriptor keypoint dynamics + trajectory MPC

Learn descriptor keypoint dynamics + trajectory MPC

My take-aways:

  • Can train a network to predict keypoint dynamics even for complex tasks ("hat on rack").
  • But planning/control is still hard.

Lucas at his defense: "Perception doesn't feel like the bottleneck anymore; it fees like the bottleneck is control."

Deep models

f=ma

Summary

Wanted: State representations for "category-level" dexterous manipulation

  • Keypoints can represent category-level geometry
    • ​Combine with local sensing / dense reconstruction
    • Can learn keypoint dynamics.
  • For task specification
    • Annotated keypoints with semantics
    • Self-supervised descriptors with demonstrations
  • Keypoints aren't everything.  They won't tell you when the eggs are done cooking!

 

State representations

  • Keypoints help us generalize from specific objects to object categories
  • ... but what about that onion?

The Onion Problem

H.J. Terry Suh and Russ Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. To appear in Workshop on the Algorithmic Foundations of Robotics (WAFR), 2020

The Onion Problem

Target Set

Carrot

 

The big question:

When is feedback "from pixels to torques" easy?

Problem Statement

Find a single policy (pixels to torques) that is invariant to the number of carrot pieces.

 

  • Objective: Move all carrot pieces to a “target set” on the cutting board.
  • Inputs (actions):  Discrete push, parameterized from (x1, y1) to (x2, y2)
  • Outputs (observations):  Images, I[n], or vectorized images y[n]

Try images as the state representation.

A control-Lyapunov function in image coordinates

Requires a forward model...

Crazy:  Try a linear model (for each discretized action)

A^* = \argmin_{A} \| Y_{k+1} - AY_k \|_F \\ \text{subject to} \quad A_{ij} \ge 0
Y_{k+1}
Y_{k}
A
=

Each row of     is an image, the "receptive field" of

A
y_{k+1}

My take-aways:

 

  • Not saying that linear models are better than deep.
  • This won't work for most problems.

 

  • When is pixels-to-torques easy?
    • One answer: When each pixel moves independently
    • Image acts like a probability density, dynamics as a transition matrix.

Many fundamental challenges

  • Dexterity / feedback with contact
  • Tactile sensing / contact mechanics
  • Distributional robustness (over tasks/environments)
  • Composition and hierarchy

 

  • State representations
  • Feedback from pixels (cameras as a sensor)

Today:

System

..., u_{-1}, u_0, u_1, ...
..., y_{-1}, y_0, y_1, ...

Auto-regressive (eg. ARMAX)

Lagrangian mechanics,

Recurrent neural networks (e.g. LSTM), ...

Feed-forward networks (e.g. \(y_n\)= image)

input

output

State-space

x_{n+1} = f(n, x_n, u_n, w_n, \theta) \\ \quad y_n = g(n, x_n, u_n, w_n, \theta)
y_{n+1} = f(n, u_n, u_{n-1}, ..., \\ \qquad \qquad y_n, y_{n-1}, ..., \\ \qquad \qquad w_n, w_{n-1}, ..., \theta)

Deep vs "physics-based" models

The failings of our physics-based models are mostly due to the unreasonable burden of estimating the "Lagrangian state" and parameters.

For e.g. onions, laundry, peanut butter, ...

The failings of our deep models are mostly due to our inability to due efficient/reliable planning, control design and analysis.

I want the next Newton to come around and to work on onions, laundry, peanut butter...

This fall: my manipulation class at MIT will be online.