Towards General In-hand Manipulation with Touch

Roberto Calandra

ICRA Vitac - 17 May 2024

Learning, Adaptive Systems, and Robotics (LASR) Lab

Grand Vision

+ Applications

+ Community

Hardware

Vision-based Tactile Sensors

[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]

[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]

[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Limitations

  • Sampling Rate
    (Camera are relatively slow)
  • Data Bandwidth
    (Camera generate hundreds of Mb/s)
  • Bulky
    (Focal distance and electronics)
  • ...

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

  • Neuromorphic tactile sensor
  • Event-based
  • Up to 1KHz
  • Modular and using off-the-shelf components
    (Inspired by the DigiTac by Nathan)

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

Slipping Experiments

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

Bandwidth

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

12%

1.7% over
entire
trajectory

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Design

Two advantages:

1) More compact form factor

2) Replaceable sensing element

IDEA: Decouple Sensing elements and Electronics

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

Cancer Detection in Prostate Tissue

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Under Review, 2024 https://arxiv.org/abs/2403.05500

Cancer

No Cancer

Touch Processing
(i.e., AI for Touch)

Creating a New Science of Touch Processing

Some of the open questions:

  • What are good features for touch?
  • Do we need sensor standardization?
    • What representation do we want/need for touch?
    • What sensorial information do we even want/need for touch?
  • What are the useful structures in computational models for touch?
  • What are the useful metrics to characterize touch?
  • How can we quantify the human psychophysics of touch?
  • What are the different tasks that can benefit from touch?
  • What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

Touch and Language

Fu, L.; Datta, G.; Huang, H.; Panitch, W. C.-H.; Drake, J.; Ortiz, J.; Mukadam, M.; Lambeta, M.; Calandra, R. & Goldberg, K.
A Touch, Vision, and Language Dataset for Multimodal Alignment

Accepted to ICML 2024 https://arxiv.org/abs/2402.13232

Tactile SLAM

Suresh, S.; Qi, H.; Wu, T.; Fan, T.; Pineda, L.; Lambeta, M.; Malik, J.; Kalakrishnan, M.; Calandra, R.; Kaess, M.; Ortiz, J. & Mukadam, M.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.13469

Applications

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Model

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

To Conclude

Human Collaborators

Overview

  • Very happy to see Touch Sensing community Growing
  • Optimistic that we will have meaningful progress in manipulation thanks to Touch in the coming years.

Thank you!

Our work on touch sensing

  • Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
    TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
    IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456

  • Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
    Active 3D Shape Reconstruction from Vision and Touch
    Advances in Neural Information Processing Systems (NeurIPS), 2021
  • Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
    PyTouch: A Machine Learning Library for Touch Processing
    IEEE International Conference on Robotics and Automation (ICRA), 2021
  • Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
    3D Shape Reconstruction from Vision and Touch
    Advances in Neural Information Processing Systems (NeurIPS), 2020
  • Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
    DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
    IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
  • Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
    OmniTact: A Multi-Directional High-Resolution Touch Sensor
    IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
  • Lin, J.; Calandra, R. & Levine, S.
    Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
  • Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
    Manipulation by Feel: Touch-Based Control with Deep Predictive Models
    IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
  • Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
    More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
    IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
  • Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
    The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
    Conference on Robot Learning (CORL), 2017, 314-323
  • Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
    Active Tactile Object Exploration with Gaussian Processes
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
    Learning Inverse Dynamics Models with Contacts
    IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
  • Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
    Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
    IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Additional Slides

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

ICRA Vitac 2024

By Roberto Calandra

ICRA Vitac 2024

  • 164