Perceiving, Understanding, and Interacting through Touch

Roberto Calandra

UC Berkeley - CS280 - 29 Apr 2021

Facebook AI Research

https://slides.com/rcalandra/berkeley-cs280

The Importance of Touch

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Importance of Touch (in Robots)

Touch for Robot Manipulation

[Allen et al. 1999]

[Chebotar et al. 2016]

[Bekiroglu et al. 2011]

[Sommer and Billard 2016]

[Schill et al. 2012]

The Next Breakthrough will be in Touch

Audio

Touch

Vision

(~1890)

(~1990)

(2020s ?)

Touch and Vision are Complementary

What do we need to make Touch Sensing practical and useful?

Making Touch Sensing Ubiquitous

Tactile Sensors in Robotics

Important factors:

Availability
Cost
Form factor
Capabilities
(e.g., what is measured, resolution)
Reliability

Many many sensors in the literature:

Most are prototypes
A handful are commercially available
or can be easily manufactured

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

Resistive or Capacitive technology
Several limitations:
- Usually, measure force or orthogonal component of force
- Relatively low density
- Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
- Often need to be calibrated

Vision-based Tactile Sensors

[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Comparison

BioTac

DIGIT

~10,000 $

Cost

~15 $

Resolution

29
contact points

307,200
contact points

Mounted on multi-finger hands

Open-source

https://digit.ml

Being replicated in 15+ universities

1000x
Higher resolution

1000x
Cheaper

Elastomer Robustness

Design

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

Making Touch Sensing Ubiquitous

Touch Simulators

Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics

Most rigid-body-dynamics physics engines include some form of traditional tactile sensor (i.e., low-dimensional)
These simulators became slower with the increasing number of contact points
For the hundreds of thousands of contact points provided by vision-based tactile sensors, they became impractical
No dedicated simulator for vision-based tactile sensors so far...

Simulator Desiderata

Fast, fast, fast... (>100Hz)
General purpose
(in the same way that a physics engine can simulate any rigid body robot)
Flexible
Accurate
- In terms of output
  (Realistic rendering calibrated on real sensor)
- In terms of physics
  (Not important for perceptual tasks. Very difficult to model non-rigid contacts accurately)
Open-source

TACTO

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Software Architecture

Principle

Realistic Rendering

Making Touch Sensing Ubiquitous

Towards a Science of Touch Processing

Very limited literature about computational processing of touch sensing
- What information do we need from touch?
- What are good features for touch?
- What are the useful structures in computational models for touch?
- What are the useful metrics to characterize touch?
- What are meaningful benchmarks for touch processing?
The literature so far in Touch Processing has been using traditional CV models
Touch has a very pronounced temporal aspect

Making Touch Sensing Ubiquitous

Benchmarks & Datasets

Benchmarks:

Datasets:

What are the tasks that we care about as a community?
The first great enabler of benchmarks is accurate touch simulators
The second great enabler of benchmarks is easily available hardware

Very few touch sensing datasets available nowadays
Data collection is currently limited by the reliability of hardware
Simulators, and available hardware will enable new and larger datasets
(But we also need as a community to encourage and nurture this)

Learning Grasp Stability

Making Touch Sensing Ubiquitous

Some Touch Sensing Applications

Predicting Good Grasps

Learning how to Grasp

Active Tactile Exploration

3D Reconstruction from Vision and Touch

Identify Objects from Touch

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Results

Self-supervised Data Collection

Setting:
- 7-DOF Sawyer arm
- Weiss WSG-50 Parallel gripper
- one GelSight on each finger
- Two RGB-D cameras in front and on top

(Almost) fully autonomous data collection:
- Estimates the object position using depth, and perform a random grasp of the object.
- Labels automatically generated by looking at the presence of contacts after each attempted lift

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Gentle Grasping

Since our model considers forces, we can select grasps that are effective, but gentle
Reduces the amount of force used by 50%, with no significant loss in grasp success

Touch Sensing for Tele-operation

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 997-1002

Learning Fine In-finger Manipulation

Model-based Reinforcement Learning

Marble Manipulation

Towards Practical Touch Sensing

Touch is a key sensor modality for humans and robots
Touch complements vision by providing rich, but highly localized information
Today, we briefly discussed several important aspects of touch:
- Hardware
- Simulator
- Touch Processing
- Benchmarks and Datasets
- Some Applications
Vision-based Tactile sensing are leading to a renaissance of Touch

Feedback at: https://tinyurl.com/cs280-touch

References (of our work on touch sensing)

Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
A Machine Learning Framework for Touch Processing
Under Review, 2020
Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020
Lin, J.; Calandra, R. & Levine, S.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
Active Tactile Object Exploration with Gaussian Processes
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
Learning Inverse Dynamics Models with Contacts
IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695