Touch: The Next
Sensing Modality for Robotics

Roberto Calandra

BAIR Robotics Workshop - 04 April 2025

Learning, Adaptive Systems, and Robotics (LASR) Lab

The Importance of Touch

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

The Next Breakthrough will be in Touch

Audio

Touch

Vision

(~1890)

(~1990)

(2020s ?)

Grand Vision

+ Applications

+ Community

Democratizing Touch with DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Tactile SLAM

Suresh, S.; Qi, H.; Wu, T.; Fan, T.; Pineda, L.; Lambeta, M.; Malik, J.; Kalakrishnan, M.; Calandra, R.; Kaess, M.; Ortiz, J. & Mukadam, M.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Science Robotics 2024 https://arxiv.org/abs/2312.13469

Limitations

Sampling Rate
(Camera are relatively slow)
Data Bandwidth
(Camera generate hundreds of Mb/s)
Bulky
(Focal distance and electronics)
No Multimodality
(Human skin has very heterogeneous receptors)

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2312.01236

Neuromorphic tactile sensor
Event-based
Up to 1KHz
Modular and using off-the-shelf components
(Inspired by the DigiTac by Nathan)

Bandwidth

12%

1.7% over
entire
trajectory

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

Cancer Detection in Prostate Tissue

Cancer

No Cancer

Digit360: A Superhuman Multi-modal Fingertip

Lambeta M.; Wu T.; Sengül A.; Most V. R.; Black N.; Sawyer K.; Mercado R.; Qi H.; Sohn A.; Taylor B.; Tydingco N.; Kammerer G.; Stroud D.; Khatha J.; Jenkins K.; Most K.; Stein N.; Chavira R.; Craven-Bartle T.; Sanchez E.; Ding Y.; Malik J. & Calandra R.
Digitizing Touch with an Artificial Multimodal Fingertip
Arxiv Preprint. 2024 https://arxiv.org/abs/2411.02479

CFP:

1000+ sensors

200+ labs

(For DIGIT, 40 labs)

Resolution

~100x

Multi-modality

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Demo

Dynamic Illumination and Image Fusion

Artemii, R.; Dugonjic, Z.; Lambeta, M. & Calandra, R.
Enhance Vision-Based Tactile Sensors via Dynamic Illumination and Image Fusion
Arxiv Preprint, 2025, https://www.arxiv.org/abs/2504.00017

Dynamic Illumination and Image Fusion

Artemii, R.; Dugonjic, Z.; Lambeta, M. & Calandra, R.
Enhance Vision-Based Tactile Sensors via Dynamic Illumination and Image Fusion
Arxiv Preprint, 2025, https://www.arxiv.org/abs/2504.00017

Creating a New Science of Touch Processing

Some of the open questions:

What are good features for touch?
Do we need sensor standardization?
- What representation do we want/need for touch?
- What sensorial information do we even want/need for touch?
What are the useful structures in computational models for touch?
What are the useful metrics to characterize touch?
How can we quantify the human psychophysics of touch?
What are the different tasks that can benefit from touch?
What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

What Representations do we Need for Touch?

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021

Kerr, J.; Huang, H.; Wilcox, A.; Hoque, R.; Ichnowski, J.; Calandra, R. & Goldberg, K.
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Robotics: Science and Systems (RSS) 2023, Online: https://arxiv.org/pdf/2209.13042

Transfer Across Sensors

Transfer across Tasks

Touch and Language

Fu, L.; Datta, G.; Huang, H.; Panitch, W. C.-H.; Drake, J.; Ortiz, J.; Mukadam, M.; Lambeta, M.; Calandra, R. & Goldberg, K.
A Touch, Vision, and Language Dataset for Multimodal Alignment
ICML 2024 https://arxiv.org/abs/2402.13232

Learning Gentle Grasping Using Vision, Sound, and Touch

Nakahara, K. & Calandra, R.
Learning Gentle Grasping Using Vision, Sound, and Touch
Arxiv Preprint, 2025, https://arxiv.org/abs/2503.07926

Learning Gentle Grasping Using Vision, Sound, and Touch

Nakahara, K. & Calandra, R.
Learning Gentle Grasping Using Vision, Sound, and Touch
Arxiv Preprint, 2025, https://arxiv.org/abs/2503.07926

Overview

Touch is a key sensor modality for humans and robots
Building an ecosystem for research in tactile sensing, towards the long-term goal of making touch sensing practical and more accessible (both in research and in the real-world)
Touch sensing is at the verge of a "digital revolution" that will make it ubiquitous and enable new applications

Live Demo (DIGIT360 + OpenTouch)

https://github.com/lasr-lab/opentouch-interface

Our work on touch sensing

Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
Active 3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2021
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021
Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
Lin, J.; Calandra, R. & Levine, S.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
Active Tactile Object Exploration with Gaussian Processes
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
Learning Inverse Dynamics Models with Contacts
IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Additional Slides

Learning to Play Piano in the Real World
(Without Touch)

Zeulner, Y.; Sandeep S.; and Roberto Calandra & Calandra, R.
Learning to Play Piano in the Real World
Arxiv preprint, 2025, https://arxiv.org/abs/2503.15481

Learning to Play Piano with Touch

Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, online: https://arxiv.org/abs/2106.02040

LASR Lab @ TU Dresden

Supported by

Touch Sensing and Processing Summer School 2025

To Be Announced Soon!

www.lasr.org/tspss2025

Achieving Human-level Manipulation with Robots

Model

Learning General In-Hand Rotation

Applications

Key Applications

Robotics

Metaverse
(AR/VR)

E-commerce

Medical

Touch Processing
(i.e., AI for Touch)

PyTouch: A Machine Learning Library
for Touch Processing

Goal: Create the equivalent of OpenCV for Touch

DIGIT's History

GelSight

(2018)

(2020)

(2022)

Vision-based Tactile Sensors

[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]

[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]

[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Examples of DIGIT Measurements

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
taxels

307,200
taxels

Mounted on multi-finger hands

Open-source

https://digit.ml

1000x
Higher resolution

1000x
Cheaper

* component cost for 1000 units, not including labor

DIGIT Commercialization

Replicated in 20+ universities
Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience
Partnership with GelSight Inc. to commercialize DIGIT
Most widespread tactile sensor in robotics!
- Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)

DIGIT Pinky

Cancer Detection in Prostate Tissue

Cancer

No Cancer

Simulation

TACTO

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456

Software Architecture

Rendering from Simulated DIGIT

Sim2Real

Slack Channel

www.touch-sensing.org

Model-based Reinforcement Learning

In-Hand Object Rotation via Rapid Motor Adaptation

Some Touch Sensing Applications

Predicting Grasp Stability
[Calandra et al. 2017]

Learning how to (Re)Grasp
[Calandra et al. 2018]

Active Tactile Exploration
[Yi at al. 2016]

3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]

Identify Objects from Touch

[Lin et al. 2019]

Learning to Play Piano from Touch
[Xu at al. 2022]

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Touch for the Metaverse

Touch is a deeply social and emotional sense
How can we sense, calibrate and reproduce touch?
How can we accurately and easily digitize objects and their physical properties into virtual worlds?

Sense

Process

Reproduce

Telepresence via Haptic Feedback

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015

Robot Learning

Code Example

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Self-supervised Data Collection

Setting:
- 7-DOF Sawyer arm
- Weiss WSG-50 Parallel gripper
- one GelSight on each finger
- Two RGB-D cameras in front and on top

(Almost) fully autonomous data collection:
- Estimates the object position using depth, and perform a random grasp of the object.
- Labels automatically generated by looking at the presence of contacts after each attempted lift

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

Resistive or Capacitive technology
Several limitations:
- Usually, measure force or orthogonal component of force
- Relatively low density
- Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
- Often need to be calibrated

Features of PyTouch

First Machine Learning library dedicated to Touch Processing
(Based on PyTorch)
Hardware-agnostic abstractions for rapid experimentation
Native integration with DIGIT (hardware) and Tacto (simulator)
(Working on supporting non vision-based sensors)
Platform for standardizing evaluation and comparison of different models
Touch "as-a-service"
- Allows non-ML-experts to use SOTA ML models in their applications
- Pre-trained models (e.g., Touch detection and slip for DIGIT)
  - (Can also be used for fast fine-tuning)
Open-source -- Anybody can contribute

Learning to Play Piano with Touch

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

Failure Case

Understanding the Learned Model

More force = better grasp

Understanding the Learned Model

But not always ?

Understanding the Learned Model

Gentle Grasping

Since our model considers forces, we can select grasps that are effective, but gentle
Reduces the amount of force used by 50%, with no significant loss in grasp success

BAIR Robotics

By Roberto Calandra

BAIR Robotics

10 days ago
51

Roberto Calandra PRO

Full Professor at TU Dresden. Head of the LASR Lab. Working in AI, Robotics and Touch Sensing.

Touch: The Next Sensing Modality for Robotics

The Importance of Touch

The Importance of Touch (in Humans)

The Next Breakthrough will be in Touch

Audio

Touch

Vision

Grand Vision

Democratizing Touch with DIGIT

Tactile SLAM

Limitations

Evetac

Bandwidth

DIGIT Pinky

Cancer Detection in Prostate Tissue

Digit360: A Superhuman Multi-modal Fingertip

Resolution

Multi-modality

Learning General In-Hand Rotation

Demo

Dynamic Illumination and Image Fusion

Dynamic Illumination and Image Fusion

Creating a New Science of Touch Processing

What Representations do we Need for Touch?

Transfer Across Sensors

Transfer across Tasks

Touch and Language

Learning Gentle Grasping Using Vision, Sound, and Touch

Learning Gentle Grasping Using Vision, Sound, and Touch

Overview

Live Demo (DIGIT360 + OpenTouch)

Our work on touch sensing

Additional Slides

Learning to Play Piano in the Real World (Without Touch)

Learning to Play Piano with Touch

LASR Lab @ TU Dresden

Touch Sensing and Processing Summer School 2025

To Be Announced Soon!

www.lasr.org/tspss2025

Achieving Human-level Manipulation with Robots

Model

Learning General In-Hand Rotation

Applications

Key Applications

Touch Processing (i.e., AI for Touch)

PyTouch: A Machine Learning Library for Touch Processing

DIGIT's History

Vision-based Tactile Sensors

OmniTact

Examples of DIGIT Measurements

Comparison

DIGIT Commercialization

DIGIT Pinky

DIGIT Pinky

Cancer Detection in Prostate Tissue

Simulation

TACTO

Software Architecture

Rendering from Simulated DIGIT

Sim2Real

Slack Channel

Model-based Reinforcement Learning

In-Hand Object Rotation via Rapid Motor Adaptation

Some Touch Sensing Applications

Visuo-tactile Learned Model

Examples of Training Objects

Grasp Success on Unseen Objects

Touch for the Metaverse

Telepresence via Haptic Feedback

Robot Learning

Code Example

Learning Grasp Stability

Self-supervised Data Collection

Traditional Sensors

Features of PyTouch

Learning to Play Piano with Touch

Do we really need Touch?

Failure Case

Understanding the Learned Model

Understanding the Learned Model

Touch: The Next
Sensing Modality for Robotics

Learning to Play Piano in the Real World
(Without Touch)

Touch Processing
(i.e., AI for Touch)

PyTouch: A Machine Learning Library
for Touch Processing