Robots and the Sense of Touch

Roberto Calandra

Facebook AI Research

Re-Work Deep Reinforcement Learning Summit - 21 June 2019

State-of-the-art in Robotics

From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo

Learning to Interact with the World

  • Interactions with the world happen through forces (Newton's laws of motion)
    • Current robots are almost "blind" w.r.t. forces
    • Can we use tactile sensors to measure contact forces (like in humans) ?
  • As humans, we create complex models of the world
    • These models are useful to predict the effects of our actions
    • Can robots learn similar models from complex multi-modal inputs?
    • Can these learned models be used for control?

The Importance of Touch (in Robots)

The Importance of Touch (in Humans)

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

Challenges and Goal

  • Opportunity: Tactile sensors allow to accurately measure contact forces, and enable real feedback control during manipulation.
  • Goal: Improve current robot manipulation capabilities by integrating raw tactile sensors (through machine learning).
  • Problem: Integrating tactile sensors in the control scheme is challenging, and requires a lot of engineering and expertise.

Previous Literature

[Allen et al.  1999]

[Chebotar et al. 2016]

[Bekiroglu et al. 2011]

[Sommer and Billard 2016]

[Schill et al. 2012]

GelSight Sensor

[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, 2017]

  • Optical tactile sensor
  • Highly sensitive
  • High resolution
    (1280x960 @30Hz)
  • Grid structure

Examples of GelSight Measurements

Self-supervised Data Collection

  • 7-DOF Sawyer arm, Weiss WSG-50 Parallel gripper,  and one GelSight on each finger
  • Two RGB-D cameras in front and on top of the working space
  • (Almost) fully autonomous data collection:
    • Estimates the object position using depth, and perform a random grasp of the object.
    • Labels automatically generated by looking at the presence of contacts after the attempted lift

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Gentle Grasping

  • Since our model considers forces, we can select grasps that are effective, but gentle
  • We can halve the amount of force used with no significant loss in grasp success

Lessons Learned and Future Directions

  • Engineering is super-important!
  • No post lift-off control
    • e.g., slip prediction, dynamics during loading phase
  • Parallel gripper is very limiting
    • multi-finger hands
  • Single object
    • cluttered environments
  • Only consider macro-actions
    • Moving towards continuous control

Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019

Human Collaborators

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.

More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Andrew

Owens

Dinesh
Jayaraman

Wenzhen
Yuan

Justin
Lin

Jitendra
Malik

Sergey
Levine

Edward H.
Adelson

Summary

  • Touch is a key sensor modality for interacting with the world
  • Integrating high-resolution tactile sensors in traditional robot controllers is challenging
  • Deep Reinforcement Learning can help!
  • We presented a system that learns through self-supervised learning to grasp and re-grasp
  • This system can generalize to new object with high success rate

Vision:
Towards active feedback control from multi-modal sensing

Thank you for your attention

References

  • Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
    More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
    IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
  • Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
    Integration of vision, force and tactile sensing for grasping
    Int. J. Intelligent Machines, 1999, 4, 129-149
  • Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
    Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
    International Conference on Intelligent Robots and Systems (IROS), 2016
  • Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
    Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
    BioRob, 2012
  • Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
    Assessing grasp stability based on learning and haptic data
    Transactions on Robotics, 2011, 27
  • Sommer, N. & Billard, A.
    Multi-contact haptic exploration and grasping with tactile sensors
    Robotics and autonomous systems, 2016, 85, 48-61

Failure Case

Understanding the Learned Model

Understanding the Learned Model

Understanding the Learned Model

Additional Slide: Tactile MPC

Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019

Additional Slide: Full Grasping Results