Building an Ecosystem for Research on (Artificial) Touch Sensing

Roberto Calandra

Workshop @IEEE World Haptics Conference - 10 July 2023

Learning, Adaptive Systems, and Robotics (LASR) Lab

State of the Art in Robotics

What are we missing?

  • Optimization
  • Modeling dynamical systems
  • Reinforcement Learning
  • Control and planning
  • Dexterous manipulation
  • Locomotion
  • Hardware design
  • Touch Processing
  • Applications

Learning, Adaptive Systems, and Robotics (LASR) Lab

Machine Learning

Robotics

Touch
Sensing

Hardware

Software

In this talk

The Importance of Touch

Touch Sensing is Hard

  • Interdisciplinary field which requires vertical integration. From hardware design to touch processing; From robot control to applications.
  • Many ad-hoc solutions and little re-use of existing components
    (i.e., We keep reinventing the wheel over and over)
  • High entrance bar for new researchers and practitioners
  • How can we lower the entrance bar?
  • How can we improve reproducibility?
  • How can we accelerate research by re-using existing components?

Standardization, and the creation of an ecosystem of tools

Grand Vision

+ Applications

+ Community

Hardware

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
taxels

307,200
taxels

Mounted on multi-finger hands

Open-source

1000x
Higher resolution

1000x
Cheaper

* component cost for 1000 units, not including labor

DIGIT Commercialization

  • Replicated in 20+ universities

  • Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience

  • Partnership with GelSight Inc. to commercialize DIGIT

  • Most widespread tactile sensor in robotics!

    • Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Touch Processing

Creating a New Science of Touch Processing

Some of the open questions:

  • What are good features for touch?
  • Do we need sensor standardization?
    • What representation do we want/need for touch?
    • What sensorial information do we even want/need for touch?
  • What are the useful structures in computational models for touch?
  • What are the useful metrics to characterize touch?
  • How can we quantify the human psychophysics of touch?
  • What are the different tasks that can benefit from touch?
  • What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

PyTouch: A Machine Learning Library
for Touch Processing

Goal: Create the equivalent of OpenCV for Touch

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021,
Online: https://arxiv.org/abs/2105.12791

Learning from Multiple Sensors Improve
Performance (Touch Detection Task)

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021,
Online: https://arxiv.org/abs/2105.12791

Learning Re-usable Visio-Tactile Representations

Kerr, J.; Huang, H.; Wilcox, A.; Hoque, R.; Ichnowski, J.; Calandra, R. & Goldberg, K.
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Accepted to RSS 2023, Online: https://arxiv.org/pdf/2209.13042

Applications

Some Touch Sensing Applications

Predicting Grasp Stability
[Calandra et al. 2017]

Learning how to (Re)Grasp
[Calandra et al. 2018]

Active Tactile Exploration
[Yi at al. 2016]

3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]

Identify Objects from Touch

[Lin et al. 2019]

Learning to Play Piano from Touch
[Xu at al. 2022]

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Achieving Human-level Manipulation with Robots

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

To Conclude

Human Collaborators

Slack Channel

1st Workshop on Touch Processing

15/16 December 2023 @ NeurIPS
New Orleans, United States

We are Hiring

  • Post-docs
  • PhD Students
  • Robot Lab Manager

Overview

  • Touch is a key sensor modality for humans and robots
  • Presented our ecosystem for research in tactile sensing aimed at simplifying integration and enable to perceive, understand, and interact through touch:
    • Hardware
    • Simulation
    • Touch Processing
  • Towards the long-term goal of making touch sensing practical and more accessible (both in research and in the real-world)
  • Touch sensing is at the verge of a "digital revolution" that will make it ubiquitous and enable new applications

Thank you!

DIGIT

TACTO

PyTouch