Towards a Science of
Touch Processing
Roberto Calandra
Facebook AI Research
Workshop on Visuo-tactile Sensors for Robust Manipulation - 13 July 2020
The Importance of Touch


What do we need to make Touch Sensing ubiquitous in Robotics?

Touch as a Commodity
Audio
Touch
Vision


(~1890)
(~1990)
(2020s ?)



Making Touch Sensing Ubiquitous


Hardware Requirements
- Readily available (e.g., mass-produced)
- Inexpensive
- Reliable
- Form factor suitable for the tasks of interests
- Rich in measurements w.r.t. the tasks of interests (e.g., high-resolution)
- Open-source
DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Comparison
BioTac
DIGIT

~10,000 $
Cost
~15 $
Resolution
29
contact points
307,200
contact points
Mounted on multi-finger hands


Open-sourced



https://digit.ml

Marble Manipulation Trajectories




Making Touch Sensing Ubiquitous

Simulators Requirement
- Fast, fast, fast... (>100Hz)
- Open-source
- General purpose
(in the same way that a physics engine can simulate any rigid body robot) - Flexible
- Accurate
- In terms of output
(Realistic rendering calibrated on real sensor) - In terms of physics
(Not important for perceptual tasks. Very difficult to model non-rigid contacts accurately)
- In terms of output


Making Touch Sensing Ubiquitous
Touch Processing Requirements
- Standardized tools for processing touch readings
- Touch Processing as an easy-to-use service for all robots
(Just connect a tactile sensor and obtain high-level features useful for control) - Invariant to variation between sensors
- (Likely to be Machine Learning based)
- Open-source
Towards a Science of Touch Processing
- What are good features for touch?
- Do we need sensor standardization?
- In Image Processing, images are represented as RGB [HxWx3] matrices
- What representation do we want/need for touch?
- What sensors do we even want/need for touch?
- What are the useful structures in computational models for touch?
- What are the useful metrics to characterize touch?
- How can we quantify the human psychophysics of touch?
- What are the different tasks that can benefit from touch?
- What are meaningful benchmarks for touch processing?
Making Touch Sensing Ubiquitous


Benchmarks & Datasets Requirements
Benchmarks:
Datasets:
- What are the tasks that we care about as a community?
- The first great enabler of benchmarks will be accurate touch simulators
- The second great enabler of benchmarks will be easily available hardware
- Very few touch sensing datasets available nowadays
- Data collection is currently limited by the reliability of hardware
- Simulators, and available hardware will enable new and larger datasets
(But we also need as a community to encourage and nurture this)
Making Touch Sensing Ubiquitous


Human Collaborators
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845











Summary
- Touch is a key sensor modality for robotics
- With increased availability of hardware, the community will now need:
- Simulators
- Benchmarks and Datasets
- Software to process touch readings
- A science of Touch Processing will be crucial to standardize, understand and process tactile readings...
- ...and ultimately make touch ubiquitous in robots
Towards Touch Processing as the field that studies Touch

References
- Yuan, W.; Dong, S. & Adelson, E. H.
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
Sensors, 2017 - Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307 - Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
Integration of vision, force and tactile sensing for grasping
Int. J. Intelligent Machines, 1999, 4, 129-149 - Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
International Conference on Intelligent Robots and Systems (IROS), 2016 - Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
BioRob, 2012 - Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
Assessing grasp stability based on learning and haptic data
Transactions on Robotics, 2011, 27 - Sommer, N. & Billard, A.
Multi-contact haptic exploration and grasping with tactile sensors
Robotics and autonomous systems, 2016, 85, 48-61
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
The Importance of Touch (in Humans)
Model-based Reinforcement Learning
Trajectories



Towards a Science of Touch Processing
By Roberto Calandra
Towards a Science of Touch Processing
[RSS Workshop on Visuo-tactile Sensors for Robust Manipulation: From Perception to Control]
- 942