Towards a Science of
Touch Processing

Roberto Calandra

Facebook AI Research

Workshop on Visuo-tactile Sensors for Robust Manipulation - 13 July 2020

Touch as a Commodity

Making Touch Sensing Ubiquitous

Hardware Requirements

• Readily available (e.g., mass-produced)
• Inexpensive
• Reliable
• Form factor suitable for the tasks of interests
• Rich in measurements w.r.t. the tasks of interests (e.g., high-resolution)
• Open-source

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

Marble Manipulation Trajectories

Making Touch Sensing Ubiquitous

Simulators Requirement

• Fast, fast, fast... (>100Hz)
• Open-source
• General purpose
  (in the same way that a physics engine can simulate any rigid body robot)
• Flexible
• Accurate
  • In terms of output
    (Realistic rendering calibrated on real sensor)
  • In terms of physics
    (Not important for perceptual tasks. Very difficult to model non-rigid contacts accurately)

Making Touch Sensing Ubiquitous

Touch Processing Requirements

• Standardized tools for processing touch readings
• Touch Processing as an easy-to-use service for all robots
  (Just connect a tactile sensor and obtain high-level features useful for control)
• Invariant to variation between sensors
• (Likely to be Machine Learning based)
• Open-source

Towards a Science of Touch Processing

• What are good features for touch?
• Do we need sensor standardization?
  • In Image Processing, images are represented as RGB [HxWx3] matrices
  • What representation do we want/need for touch?
  • What sensors do we even want/need for touch?
• What are the useful structures in computational models for touch?
• What are the useful metrics to characterize touch?
• How can we quantify the human psychophysics of touch?
• What are the different tasks that can benefit from touch?
• What are meaningful benchmarks for touch processing?

Making Touch Sensing Ubiquitous

Benchmarks & Datasets Requirements

• What are the tasks that we care about as a community?
• The first great enabler of benchmarks will be accurate touch simulators
• The second great enabler of benchmarks will be easily available hardware

• Very few touch sensing datasets available nowadays
• Data collection is currently limited by the reliability of hardware
• Simulators, and available hardware will enable new and larger datasets
  (But we also need as a community to encourage and nurture this)

Making Touch Sensing Ubiquitous

Human Collaborators

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Summary

• Touch is a key sensor modality for robotics
• With increased availability of hardware, the community will now need:
  • Simulators
  • Benchmarks and Datasets
  • Software to process touch readings
• A science of Touch Processing will be crucial to standardize, understand and process tactile readings...
• ...and ultimately make touch ubiquitous in robots

References

• Yuan, W.; Dong, S. & Adelson, E. H.
  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
  Sensors, 2017
• Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
  More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
  IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
• Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
  Integration of vision, force and tactile sensing for grasping
  Int. J. Intelligent Machines, 1999, 4, 129-149
• Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
  Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
  International Conference on Intelligent Robots and Systems (IROS), 2016
• Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
  Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
  BioRob, 2012
• Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
  Assessing grasp stability based on learning and haptic data
  Transactions on Robotics, 2011, 27
• Sommer, N. & Billard, A.
  Multi-contact haptic exploration and grasping with tactile sensors
  Robotics and autonomous systems, 2016, 85, 48-61

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

Model-based Reinforcement Learning

Trajectories