Building an Ecosystem for Research on Touch Sensing

Roberto Calandra

ICRA2021 ViTac Workshop - 04 June 2021

Facebook AI Research

https://slides.com/rcalandra/vitac2021

Touch Sensing is Hard

  • Interdisciplinary field which requires vertical integration. From hardware design to touch processing; From robot control to applications.
  • Many ad-hoc solutions and little re-use of existing components
    (i.e., We keep reinventing the wheel over and over)
  • High entrance bar for new researchers and practitioners
  • How can we lower the entrance bar?
  • How can we improve reproducibility?
  • How can we accelerate research by re-using existing components?

Standardization, and the creation of an ecosystem of tools

Hardware

Software

What do we need to make Touch Sensing practical and useful?

Making Touch Sensing Ubiquitous

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
contact points

307,200
contact points

Mounted on multi-finger hands

Open-source

https://digit.ml

Being replicated in 15+ universities

1000x
Higher resolution

1000x
Cheaper

* manufacturing cost for 1000 units

Making Touch Sensing Ubiquitous

Touch Simulation

Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics

  • Most rigid-body-dynamics physics engines include some form of traditional tactile sensor (i.e., low-dimensional)
  • These simulators became slower with the increasing number of contact points
  • For 1000+ contact points provided by vision-based tactile sensors, they became impractical

TACTO

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Features

  • Fast (>100 fps)
  • General-purpose
  • Physics accuracy:
    • Modeling of the contacts is delegated to the physics engine of choice (e.g., PyBullet)
    • Works with rigid body
  • Rendering accuracy:
    • Very accurate
    • Simulate "soft" deformation of the elastomer with growing forces
    • Can be calibrated from real sensors (e.g., DIGIT)
  • Open-source -- Anybody can contribute

  

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Software Architecture

Might be possible to use it for optimizing sensor design (?)

Rendering from Simulated DIGIT

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Making Touch Sensing Ubiquitous

The Next Breakthrough will be in Touch

Audio

Touch

Vision

(~1890)

(~1990)

(2020s ?)

Creating a Science of Touch Processing

Some of the open questions:

  • What are good features for touch?
  • Do we need sensor standardization?
    • What representation do we want/need for touch?
    • What sensorial information do we even want/need for touch?
  • What are the useful structures in computational models for touch?
  • What are the useful metrics to characterize touch?
  • How can we quantify the human psychophysics of touch?
  • What are the different tasks that can benefit from touch?
  • What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

PyTouch: A Machine Learning Library
for Touch Processing

Goal: Create the equivalent of OpenCV for Touch

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Features of PyTouch

  • First Machine Learning library dedicated to Touch Processing
    (Based on PyTorch)
  • Hardware-agnostic abstractions for rapid experimentation
  • Platform for standardizing evaluation and comparison of different models
  • Touch "as-a-service"
    • Allows non-ML-experts to use SOTA ML models in their applications
    • Pre-trained models (e.g., Touch detection and slip for DIGIT)
      • (Can also be used for fast fine-tuning)
  • Native integration with DIGIT (hardware) and Tacto (simulator)
  • (Working on supporting non vision-based sensors)
  • Open-source -- Anybody can contribute

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Code Example

Learning from Multiple Sensors Improve
Performance (Touch Detection Task)

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Making Touch Sensing Ubiquitous

Benchmarks & Datasets

Benchmarks:

 

Datasets:

 
  • What are the tasks that we care about as a community?
  • Now is the time to think about Benchmarks:
    • The first enabler  is accurate touch simulators
    • The second enabler is standardized easily available hardware (e.g., DIGIT)
  • Very few touch sensing datasets available nowadays
  • Data collection is currently limited by the reliability of hardware
  • Simulators, and available hardware will enable new and larger datasets
    (But we also need as a community to encourage and nurture this)

Learning Grasp Stability

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456

Making Touch Sensing Ubiquitous

3D Reconstruction from Vision and Touch

Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020, online: https://arxiv.org/abs/2007.03778

Learning to Play Piano with Touch

Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
Under Review, 2021, online: https://arxiv.org/abs/2106.02040

Learned Compositional Policy

Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
Under Review, 2021, online: https://arxiv.org/abs/2106.02040

Making Touch Sensing Ubiquitous

Community

Slack Workplace

www.touch-sensing.org

To Conclude

Human Collaborators

Overview

  • Touch is a key sensor modality for humans and robots
  • Our goal is create an ecosystem that simplify integration and enable to perceive, understand, and interact through touch
    • Hardware
    • Simulation
    • Touch Processing
    • Benchmarks and Datasets
    • Applications
  • We want to grow and support Community
  • Towards the long-term goal of making touch sensing practical and more accessible (both in research and in the real-world)

Questions?

Backup Slides

References (of our work on touch sensing)

  • Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
    TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
    Under Review, 2020, Online:     https://arxiv.org/abs/2012.08456

  • Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
    A Machine Learning Framework for Touch Processing
    Under Review, 2020
  • Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
    3D Shape Reconstruction from Vision and Touch
    Advances in Neural Information Processing Systems (NeurIPS), 2020
  • Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
    DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
    IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
  • Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
    OmniTact: A Multi-Directional High-Resolution Touch Sensor
    IEEE International Conference on Robotics and Automation (ICRA), 2020
  • Lin, J.; Calandra, R. & Levine, S.
    Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
  • Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
    Manipulation by Feel: Touch-Based Control with Deep Predictive Models
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
  • Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
    More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
    IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
  • Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
    The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
    Conference on Robot Learning (CORL), 2017, 314-323
  • Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
    Active Tactile Object Exploration with Gaussian Processes
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
    Learning Inverse Dynamics Models with Contacts
    IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
    Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
    IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Human Collaborators

The Importance of Touch

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

What do we want to achieve?

  • Spearhead advances in Tactile Sensing

 

  • Develop touch-based applications. Among others:
    • Robotics (e.g., Manipulation)
    • AR/VR (e.g., the social aspect of touch)

Achieving Human-level Manipulation with Robots

Learning to Manipulate a Marble

Marble Manipulation Results

Motivation

How to scale to more complex, unstructured domains?

Robotics

Finance

Biological Sciences

Logistics /
Decision Making

Why Robots?

Disaster Relief

Industrial Automation

Exploration

Medicine & Eldercare

State of the Art in Robotics

From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo

What are we missing?

Tactile Sensors in Robotics

Important factors:

  • Availability
  • Cost
  • Form factor
  • Capabilities
    (e.g., what is measured, resolution)
  • Reliability

Many many sensors in the literature:

  • Most are prototypes
  • A handful are commercially available
    or can be easily manufactured

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

  • Resistive or Capacitive technology
  • Several limitations:
    • Usually, measure force or orthogonal component of force
    • Relatively low density
    • Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
    • Often need to be calibrated

Vision-based Tactile Sensors

[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

Elastomer Robustness

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Replaceable Elastomer

Reflective

Reflective
+
Markers

Transparent
+
Markers

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Some Touch Sensing Applications

Predicting Good Grasps

Learning how to Grasp

Active Tactile Exploration

3D Reconstruction from Vision and Touch

Identify Objects from Touch

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Results

Self-supervised Data Collection

  • Setting:
    • 7-DOF Sawyer arm
    • Weiss WSG-50 Parallel gripper
    • one GelSight on each finger
    • Two RGB-D cameras in front and on top

 

  • (Almost) fully autonomous data collection:
    • Estimates the object position using depth, and perform a random grasp of the object.
    • Labels automatically generated by looking at the presence of contacts after each attempted lift

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Gentle Grasping

  • Since our model considers forces, we can select grasps that are effective, but gentle
  • Reduces the amount of force used by 50%, with no significant loss in grasp success

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Touch Sensing for Tele-operation

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 997-1002

Learning Fine In-finger Manipulation

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Model-based Reinforcement Learning

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Marble Manipulation

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845