Building an Ecosystem for Research on (Artificial) Touch Sensing

Roberto Calandra

Workshop @IEEE World Haptics Conference - 10 July 2023

Learning, Adaptive Systems, and Robotics (LASR) Lab

State of the Art in Robotics

From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo

What are we missing?

Touch Sensing is Hard

• Interdisciplinary field which requires vertical integration. From hardware design to touch processing; From robot control to applications.
• Many ad-hoc solutions and little re-use of existing components
  (i.e., We keep reinventing the wheel over and over)
• High entrance bar for new researchers and practitioners

• How can we lower the entrance bar?
• How can we improve reproducibility?
• How can we accelerate research by re-using existing components?

Grand Vision

+ Applications

+ Community

Hardware

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
taxels

307,200
taxels

Mounted on multi-finger hands

Open-source

https://digit.ml

1000x
Higher resolution

1000x
Cheaper

* component cost for 1000 units, not including labor

DIGIT Commercialization

• Replicated in 20+ universities

• Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience

• Partnership with GelSight Inc. to commercialize DIGIT

• Most widespread tactile sensor in robotics!

  • Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Touch Processing

Creating a New Science of Touch Processing

Some of the open questions:

• What are good features for touch?
• Do we need sensor standardization?
  • What representation do we want/need for touch?
  • What sensorial information do we even want/need for touch?
• What are the useful structures in computational models for touch?
• What are the useful metrics to characterize touch?
• How can we quantify the human psychophysics of touch?
• What are the different tasks that can benefit from touch?
• What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

PyTouch: A Machine Learning Library
for Touch Processing

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Learning from Multiple Sensors Improve
Performance (Touch Detection Task)

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Learning Re-usable Visio-Tactile Representations

Kerr, J.; Huang, H.; Wilcox, A.; Hoque, R.; Ichnowski, J.; Calandra, R. & Goldberg, K.
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Accepted to RSS 2023, Online: https://arxiv.org/pdf/2209.13042

Applications

Some Touch Sensing Applications

Predicting Grasp Stability
[Calandra et al. 2017]

Learning how to (Re)Grasp
[Calandra et al. 2018]

Active Tactile Exploration
[Yi at al. 2016]

3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]

Identify Objects from Touch

[Lin et al. 2019]

Learning to Play Piano from Touch
[Xu at al. 2022]

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Achieving Human-level Manipulation with Robots

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

To Conclude

Human Collaborators

Slack Channel

www.touch-sensing.org

1st Workshop on Touch Processing

www.touchprocessing.org/2023/

15/16 December 2023 @ NeurIPS
New Orleans, United States

We are Hiring

www.lasr.org/join-us/

• Post-docs
• PhD Students
• Robot Lab Manager

Overview

• Touch is a key sensor modality for humans and robots
• Presented our ecosystem for research in tactile sensing aimed at simplifying integration and enable to perceive, understand, and interact through touch:
  • Hardware
  • Simulation
  • Touch Processing
• Towards the long-term goal of making touch sensing practical and more accessible (both in research and in the real-world)
• Touch sensing is at the verge of a "digital revolution" that will make it ubiquitous and enable new applications

DIGIT

TACTO

PyTouch