Towards In-hand Manipulation from Vision and Touch

Roberto Calandra

Facebook AI Research

ICRA 2020 ViTac workshop - 31 May 2020

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Importance of Touch (in Robots)

Challenges

Tactile Sensors in Robotics

Many many sensors in the literature:

• Most are prototypes
• Just a few are commercially available
  or can be manufactured easily

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

Elastomer Robustness

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

DIGIT for manipulation

• Problem: Integrating tactile sensors in the control scheme is challenging, especially for rich tactile measurements.

DIGITs mounted on a Wonik's Allegro hand

• Task: Demonstrate fine manipulation skills from touch.

Learning to Manipulate a Marble

Model-based Reinforcement Learning

Marble Manipulation Results

Marble Manipulation Trajectories

Human Collaborators

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020

Summary

References

• Yuan, W.; Dong, S. & Adelson, E. H.
  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
  Sensors, 2017
• Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
  More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
  IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
• Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
  Integration of vision, force and tactile sensing for grasping
  Int. J. Intelligent Machines, 1999, 4, 129-149
• Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
  Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
  International Conference on Intelligent Robots and Systems (IROS), 2016
• Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
  Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
  BioRob, 2012
• Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
  Assessing grasp stability based on learning and haptic data
  Transactions on Robotics, 2011, 27
• Sommer, N. & Billard, A.
  Multi-contact haptic exploration and grasping with tactile sensors
  Robotics and autonomous systems, 2016, 85, 48-61