Towards In-hand Manipulation from Vision and Touch

Roberto Calandra

Facebook AI Research

ICRA 2020 ViTac workshop - 31 May 2020

The Importance of Touch (in Humans)

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Importance of Touch (in Robots)

Challenges

Difficult to integrate/process touch in traditional analytical control scheme

Lack of good available tactile sensors

 

Tactile Sensors in Robotics

Important factors:

  • Availability
  • Cost
  • Form factor
  • Capabilities (e.g., resolution)
  • Reliability

 

Many many sensors in the literature:

  • Most are prototypes
  • Just a few are commercially available
    or can be manufactured easily

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~10,000 $

Cost

~15 $

Resolution

29
contact points

307,200
contact points

Mounted on multi-finger hands

Open-sourced

https://digit.ml

Elastomer Robustness

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

DIGIT for manipulation

  • Problem: Integrating tactile sensors in the control scheme is challenging, especially for rich tactile measurements.

DIGITs mounted on a Wonik's Allegro hand

  • Task: Demonstrate fine manipulation skills from touch.
  • Approach: Integrating rich tactile sensors through                                   .

machine learning

Learning to Manipulate a Marble

Model-based Reinforcement Learning

Marble Manipulation Results

Marble Manipulation Trajectories

Human Collaborators

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020

Summary

Grand Goal:
Towards Human-level Manipulation using Vision and Touch

  • Touch is a key sense for in-hand manipulation
    • Introduced           , the first mass-producible, low-cost, high-resolution, compact tactile sensor.
    • Demonstrated robot marble manipulation using DIGIT
    • Open-source at

DIGIT

https://digit.ml

DIGIT: @ICRA on Wednesday - Slack: #wec01_6
OmniTact:  @ICRA on Monday - Slack: #moa15_3

References

  • Yuan, W.; Dong, S. & Adelson, E. H.
    GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
    Sensors, 2017
  • Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
    More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
    IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
  • Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
    Integration of vision, force and tactile sensing for grasping
    Int. J. Intelligent Machines, 1999, 4, 129-149
  • Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
    Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
    International Conference on Intelligent Robots and Systems (IROS), 2016
  • Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
    Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
    BioRob, 2012
  • Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
    Assessing grasp stability based on learning and haptic data
    Transactions on Robotics, 2011, 27
  • Sommer, N. & Billard, A.
    Multi-contact haptic exploration and grasping with tactile sensors
    Robotics and autonomous systems, 2016, 85, 48-61