Perceiving, Understanding, and Interacting through Touch

Roberto Calandra

UC Berkeley - CS280 - 29 Apr 2021

Facebook AI Research

The Importance of Touch

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Importance of Touch (in Robots)

Touch for Robot Manipulation

[Allen et al.  1999]

[Chebotar et al. 2016]

[Bekiroglu et al. 2011]

[Sommer and Billard 2016]

[Schill et al. 2012]

The Next Breakthrough will be in Touch

Audio

Touch

Vision

(~1890)

(~1990)

(2020s ?)

Touch and Vision are Complementary

What do we need to make Touch Sensing practical and useful?

Making Touch Sensing Ubiquitous

Tactile Sensors in Robotics

Important factors:

  • Availability
  • Cost
  • Form factor
  • Capabilities
    (e.g., what is measured, resolution)
  • Reliability

Many many sensors in the literature:

  • Most are prototypes
  • A handful are commercially available
    or can be easily manufactured

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

  • Resistive or Capacitive technology
  • Several limitations:
    • Usually, measure force or orthogonal component of force
    • Relatively low density
    • Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
    • Often need to be calibrated

Vision-based Tactile Sensors

[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~10,000 $

Cost

~15 $

Resolution

29
contact points

307,200
contact points

Mounted on multi-finger hands

Open-source

Being replicated in 15+ universities

1000x
Higher resolution

1000x
Cheaper

Elastomer Robustness

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Making Touch Sensing Ubiquitous

Touch Simulators

Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics

  • Most rigid-body-dynamics physics engines include some form of traditional tactile sensor (i.e., low-dimensional)
  • These simulators became slower with the increasing number of contact points
  • For the hundreds of thousands of contact points provided by vision-based tactile sensors, they became impractical
  • No dedicated simulator for vision-based tactile sensors so far...

Simulator Desiderata

  • Fast, fast, fast... (>100Hz)
  • General purpose
    (in the same way that a physics engine can simulate any rigid body robot)
  • Flexible
  • Accurate
    • In terms of output
      (Realistic rendering calibrated on real sensor)
    • In terms of physics
      (Not important for perceptual tasks. Very difficult to model non-rigid contacts accurately)
  • Open-source

TACTO

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Software Architecture

Principle

Realistic Rendering

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Making Touch Sensing Ubiquitous

Towards a Science of Touch Processing

  • Very limited literature about computational processing of touch sensing
    • What information do we need from touch?
    • What are good features for touch?
    • What are the useful structures in computational models for touch?
    • What are the useful metrics to characterize touch?
    • What are meaningful benchmarks for touch processing?
  • The literature so far in Touch Processing has been using traditional CV models
  • Touch has a very pronounced temporal aspect

Making Touch Sensing Ubiquitous

Benchmarks & Datasets

Benchmarks:

 

Datasets:

 
  • What are the tasks that we care about as a community?
  • The first great enabler of benchmarks is accurate touch simulators
  • The second great enabler of benchmarks is easily available hardware
  • Very few touch sensing datasets available nowadays
  • Data collection is currently limited by the reliability of hardware
  • Simulators, and available hardware will enable new and larger datasets
    (But we also need as a community to encourage and nurture this)

Learning Grasp Stability

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Making Touch Sensing Ubiquitous

Some Touch Sensing Applications

Predicting Good Grasps

Learning how to Grasp

Active Tactile Exploration

3D Reconstruction from Vision and Touch

Identify Objects from Touch

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Results

Self-supervised Data Collection

  • Setting:
    • 7-DOF Sawyer arm
    • Weiss WSG-50 Parallel gripper
    • one GelSight on each finger
    • Two RGB-D cameras in front and on top

 

  • (Almost) fully autonomous data collection:
    • Estimates the object position using depth, and perform a random grasp of the object.
    • Labels automatically generated by looking at the presence of contacts after each attempted lift

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Gentle Grasping

  • Since our model considers forces, we can select grasps that are effective, but gentle
  • Reduces the amount of force used by 50%, with no significant loss in grasp success

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Touch Sensing for Tele-operation

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 997-1002

Learning Fine In-finger Manipulation

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Model-based Reinforcement Learning

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Marble Manipulation

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Towards Practical Touch Sensing

  • Touch is a key sensor modality for humans and robots
  • Touch complements vision by providing rich, but highly localized information
  • Today, we briefly discussed several important aspects of touch:
    • Hardware
    • Simulator
    • Touch Processing
    • Benchmarks and Datasets
    • Some Applications
  • Vision-based Tactile sensing are leading to a renaissance of Touch

References (of our work on touch sensing)

  • Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
    TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
    Under Review, 2020, Online:
    https://arxiv.org/abs/2012.08456

  • Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
    A Machine Learning Framework for Touch Processing
    Under Review, 2020
  • Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
    3D Shape Reconstruction from Vision and Touch

    Advances in Neural Information Processing Systems (NeurIPS), 2020
  • Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
    DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
    IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
  • Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
    OmniTact: A Multi-Directional High-Resolution Touch Sensor
    IEEE International Conference on Robotics and Automation (ICRA), 2020
  • Lin, J.; Calandra, R. & Levine, S.
    Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
  • Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
    Manipulation by Feel: Touch-Based Control with Deep Predictive Models
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
  • Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
    More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
    IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
  • Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
    The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
    Conference on Robot Learning (CORL), 2017, 314-323
  • Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
    Active Tactile Object Exploration with Gaussian Processes
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
    Learning Inverse Dynamics Models with Contacts
    IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
    Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
    IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Learning to Manipulate a Marble

Marble Manipulation Results

Motivation

How to scale to more complex, unstructured domains?

Robotics

Finance

Biological Sciences

Logistics /
Decision Making

Why Robots?

Disaster Relief

Industrial Automation

Exploration

Medicine & Eldercare

State of the Art in Robotics

What are we missing?