Towards General In-hand Manipulation with Touch

Roberto Calandra

ICRA Vitac - 17 May 2024

Learning, Adaptive Systems, and Robotics (LASR) Lab

https://slides.com/rcalandra/vitac-2024

Grand Vision

+ Applications

+ Community

Hardware

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Limitations

• Sampling Rate
  (Camera are relatively slow)
• Data Bandwidth
  (Camera generate hundreds of Mb/s)
• Bulky
  (Focal distance and electronics)
• ...

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

• Neuromorphic tactile sensor
• Event-based
• Up to 1KHz
• Modular and using off-the-shelf components
  (Inspired by the DigiTac by Nathan)

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

Slipping Experiments

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

Bandwidth

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Design

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

Cancer Detection in Prostate Tissue

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

Cancer

No Cancer

Touch Processing
(i.e., AI for Touch)

Creating a New Science of Touch Processing

Some of the open questions:

• What are good features for touch?
• Do we need sensor standardization?
  • What representation do we want/need for touch?
  • What sensorial information do we even want/need for touch?
• What are the useful structures in computational models for touch?
• What are the useful metrics to characterize touch?
• How can we quantify the human psychophysics of touch?
• What are the different tasks that can benefit from touch?
• What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

Touch and Language

Fu, L.; Datta, G.; Huang, H.; Panitch, W. C.-H.; Drake, J.; Ortiz, J.; Mukadam, M.; Lambeta, M.; Calandra, R. & Goldberg, K.
A Touch, Vision, and Language Dataset for Multimodal Alignment
Accepted to ICML 2024 https://arxiv.org/abs/2402.13232

Tactile SLAM

Suresh, S.; Qi, H.; Wu, T.; Fan, T.; Pineda, L.; Lambeta, M.; Malik, J.; Kalakrishnan, M.; Calandra, R.; Kaess, M.; Ortiz, J. & Mukadam, M.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.13469

Applications

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Model

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

To Conclude

Human Collaborators

Overview

• Very happy to see Touch Sensing community Growing
• Optimistic that we will have meaningful progress in manipulation thanks to Touch in the coming years.

Our work on touch sensing

• Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
  TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
  IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456

• Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
  Active 3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2021
• Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
  PyTouch: A Machine Learning Library for Touch Processing
  IEEE International Conference on Robotics and Automation (ICRA), 2021
• Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
  3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2020
• Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
  IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
• Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
  OmniTact: A Multi-Directional High-Resolution Touch Sensor
  IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
• Lin, J.; Calandra, R. & Levine, S.
  Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
• Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
  Manipulation by Feel: Touch-Based Control with Deep Predictive Models
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
• Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
  More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
  IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
• Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
  The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
  Conference on Robot Learning (CORL), 2017, 314-323
• Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
  Active Tactile Object Exploration with Gaussian Processes
  IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
  Learning Inverse Dynamics Models with Contacts
  IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
  Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
  IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Additional Slides

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887