Digitizing Touch

Roberto Calandra

Connect - 14 June 2023

Learning, Adaptive Systems, and Robotics (LASR) Lab

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Next Breakthrough will be in Touch

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
taxels

307,200
taxels

Mounted on multi-finger hands

Open-source

https://digit.ml

1000x
Higher resolution

1000x
Cheaper

* component cost for 1000 units, not including labor

PyTouch: A Machine Learning Library
for Touch Processing

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Key Applications

Touch for the Metaverse

• Touch is a deeply social and emotional sense
• How can we sense, calibrate and reproduce touch?
• How can we accurately and easily digitize objects and their physical properties into virtual worlds?

Telepresence via Haptic Feedback

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Touch in Robotics

Overview

• Touch is a key sensor modality for humans and robots
• Vertical ecosystem in touch sensing to simplify integration and enable to perceive, understand, and interact through touch
• Touch sensing is at the verge of a "digital revolution" that will make it ubiquitous and enable new applications

Backup Slides

Our work on touch sensing

• Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
  TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
  IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456

• Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
  Active 3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2021
• Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
  PyTouch: A Machine Learning Library for Touch Processing
  IEEE International Conference on Robotics and Automation (ICRA), 2021
• Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
  3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2020
• Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
  IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
• Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
  OmniTact: A Multi-Directional High-Resolution Touch Sensor
  IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
• Lin, J.; Calandra, R. & Levine, S.
  Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
• Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
  Manipulation by Feel: Touch-Based Control with Deep Predictive Models
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
• Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
  More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
  IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
• Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
  The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
  Conference on Robot Learning (CORL), 2017, 314-323
• Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
  Active Tactile Object Exploration with Gaussian Processes
  IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
  Learning Inverse Dynamics Models with Contacts
  IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
  Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
  IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

DIGIT Commercialization

• Replicated in 20+ universities

• Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience

• Partnership with GelSight Inc. to commercialize DIGIT

• Most widespread tactile sensor in robotics!

  • Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)

Some Touch Sensing Applications

Predicting Grasp Stability
[Calandra et al. 2017]

Learning how to (Re)Grasp
[Calandra et al. 2018]

Active Tactile Exploration
[Yi at al. 2016]

3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]

Identify Objects from Touch

[Lin et al. 2019]

Learning to Play Piano from Touch
[Xu at al. 2022]

Creating a New Science of Touch Processing

Some of the open questions:

• What are good features for touch?
• Do we need sensor standardization?
  • What representation do we want/need for touch?
  • What sensorial information do we even want/need for touch?
• What are the useful structures in computational models for touch?
• What are the useful metrics to characterize touch?
• How can we quantify the human psychophysics of touch?
• What are the different tasks that can benefit from touch?
• What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

Features of PyTouch

• First Machine Learning library dedicated to Touch Processing
  (Based on PyTorch)
• Hardware-agnostic abstractions for rapid experimentation
• Native integration with DIGIT (hardware) and Tacto (simulator)
  (Working on supporting non vision-based sensors)
• Platform for standardizing evaluation and comparison of different models
• Touch "as-a-service"
  • Allows non-ML-experts to use SOTA ML models in their applications
  • Pre-trained models (e.g., Touch detection and slip for DIGIT)
    • (Can also be used for fast fine-tuning)
• Open-source -- Anybody can contribute

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Code Example

Learning from Multiple Sensors Improve
Performance (Touch Detection Task)

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Learning Re-usable Visio-Tactile Representations

Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, Ken Goldberg
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Accepted to RSS 2023, Online: https://arxiv.org/pdf/2209.13042

Tactile Sensors in Robotics

Important factors:

• Availability
• Cost
• Form factor
• Capabilities
  (e.g., what is measured, resolution)
• Reliability

Many many sensors in the literature:

• Most are prototypes
• A handful are commercially available
  or can be easily manufactured

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Vision-based Tactile Sensors

[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]

[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]

[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

Motivation

Achieving Human-level Manipulation with Robots

Why Robots?

State of the Art in Robotics

From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

What are we missing?

Key Challenges

• Multi-modal Sensing

• Adaptive Hardware configuration

• Quick adaptation to new tasks

Hardware

Software

In-Hand Object Rotation via Rapid Motor Adaptation

Elastomer Robustness

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Self-supervised Data Collection

• Setting:
  • 7-DOF Sawyer arm
  • Weiss WSG-50 Parallel gripper
  • one GelSight on each finger
  • Two RGB-D cameras in front and on top

• (Almost) fully autonomous data collection:
  • Estimates the object position using depth, and perform a random grasp of the object.
  • Labels automatically generated by looking at the presence of contacts after each attempted lift

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Gentle Grasping

• Since our model considers forces, we can select grasps that are effective, but gentle
• Reduces the amount of force used by 50%, with no significant loss in grasp success

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

• Resistive or Capacitive technology
• Several limitations:
  • Usually, measure force or orthogonal component of force
  • Relatively low density
  • Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
  • Often need to be calibrated

Learning to Play Piano with Touch

Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, online: https://arxiv.org/abs/2106.02040