Towards Embodied
Artificial Intelligence

Roberto Calandra

Oxford University - 26 August 2024

Learning, Adaptive Systems, and Robotics (LASR) Lab

https://slides.com/rcalandra/2024-oxford-university

Motivation

Why Robots?

Disaster Relief

Industrial Automation

State of the Art in Robotics

From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo

What are we missing?

Our Mission

1) Make Robots more useful in the real world
(Through the use of AI)
 

2) Advance AI
(Using embodiment as a case study)
 

Key Challenges

• Multi-modal Sensing

• Adaptive Hardware configuration

• Quick adaptation to new tasks

Hardware

Software

Key Challenges

• Multi-modal Sensing

• Adaptive Hardware configuration

• Quick adaptation to new tasks

Touch Sensing

Morphological adaptation

Model-based
Reinforcement Learning

Hardware

Software

From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden

The Importance of Touch (in Humans)

Touch Sensing is Hard

• Interdisciplinary field which requires vertical integration. From hardware design to touch processing; From robot control to applications.
• Many ad-hoc solutions and little re-use of existing components
  (i.e., We keep reinventing the wheel over and over)
• High entrance bar for new researchers and practitioners

• How can we lower the entrance bar?
• How can we improve reproducibility?
• How can we accelerate research by re-using existing components?

The Next Breakthrough will be in Touch

Grand Vision

+ Applications

+ Community

Hardware

Tactile Sensors in Robotics

Important factors:

• Availability
• Cost
• Form factor
• Capabilities
  (e.g., what is measured, resolution)
• Reliability

Many many sensors in the literature:

• Most are prototypes
• A handful are commercially available
  or can be easily manufactured

[Wilson et al., 2019]

[Piacenza et al., 2020]

[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]

Vision-based Tactile Sensors

[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]

[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]

[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]

[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]

[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]

DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Comparison

BioTac

DIGIT

~15,000 $

Cost

~15 $*

Resolution

29
taxels

307,200
taxels

Mounted on multi-finger hands

Open-source

https://digit.ml

1000x
Higher resolution

1000x
Cheaper

* component cost for 1000 units, not including labor

DIGIT Commercialization

• Replicated in 20+ universities

• Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience

• Partnership with GelSight Inc. to commercialize DIGIT

• Most widespread tactile sensor in robotics!

  • Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)

Limitations

• Sampling Rate
  (Camera are relatively slow)
• Data Bandwidth
  (Camera generate hundreds of Mb/s)
• Bulky
  (Focal distance and electronics)
• ...

OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.01236

• Neuromorphic tactile sensor
• Event-based
• Up to 1KHz
• Modular and using off-the-shelf components
  (Inspired by the DigiTac by Nathan)

Evetac

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Accepted to T-RO, 2023 https://arxiv.org/abs/2312.01236

Bandwidth

Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation 
Accepted to T-RO, 2023 https://arxiv.org/abs/2312.01236

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

DIGIT Pinky

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

Cancer Detection in Prostate Tissue

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

Cancer Detection in Prostate Tissue

Cancer

No Cancer

Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500

Touch Processing
(i.e., AI for Touch)

Creating a New Science of Touch Processing

Some of the open questions:

• What are good features for touch?
• Do we need sensor standardization?
  • What representation do we want/need for touch?
  • What sensorial information do we even want/need for touch?
• What are the useful structures in computational models for touch?
• What are the useful metrics to characterize touch?
• How can we quantify the human psychophysics of touch?
• What are the different tasks that can benefit from touch?
• What are meaningful benchmarks for touch processing?

Very limited literature about computational processing of touch sensing

PyTouch: A Machine Learning Library
for Touch Processing

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

What Representations do we Need for Touch?

Touch and Language

Fu, L.; Datta, G.; Huang, H.; Panitch, W. C.-H.; Drake, J.; Ortiz, J.; Mukadam, M.; Lambeta, M.; Calandra, R. & Goldberg, K.
A Touch, Vision, and Language Dataset for Multimodal Alignment
ICML 2024 https://arxiv.org/abs/2402.13232

Tactile SLAM

Suresh, S.; Qi, H.; Wu, T.; Fan, T.; Pineda, L.; Lambeta, M.; Malik, J.; Kalakrishnan, M.; Calandra, R.; Kaess, M.; Ortiz, J. & Mukadam, M.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation 
Under Review, 2023 https://arxiv.org/abs/2312.13469

Applications

Key Applications

Robotics

Metaverse
(AR/VR)

E-commerce

Medical

Some Touch Sensing Applications

Predicting Grasp Stability
[Calandra et al. 2017]

Learning how to (Re)Grasp
[Calandra et al. 2018]

Active Tactile Exploration
[Yi at al. 2016]

3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]

Identify Objects from Touch

[Lin et al. 2019]

Learning to Play Piano from Touch
[Xu at al. 2022]

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307

Examples of Training Objects

Collected 6450 grasps from over 60 training objects over ~2 weeks.

Grasp Success on Unseen Objects

83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Achieving Human-level Manipulation with Robots

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Learning General In-Hand Rotation

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Key Challenges

• Multi-modal Sensing

• Adaptive Hardware configuration

• Quick adaptation to new tasks

Touch Sensing

Morphological adaptation

Model-based
Reinforcement Learning

Hardware

Software

Learning Models of the World

Humans seem to make extensive use of models for planning and control *
(e.g., to predict the effects of our actions)

* [Kawato, M. Internal models for motor control and trajectory planning Current Opinion in Neurobiology , 1999, 9, 718 - 727],
   [Gläscher, J.; Daw, N.; Dayan, P. & O'Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement
    learning Neuron, Elsevier, 2010, 66, 585-595]

Learning to Walk in 80 Trials

Calandra, R.; Seyfarth, A.; Peters, J. & Deisenroth, M. P.
Bayesian Optimization for Learning Gaits under Uncertainty
Annals of Mathematics and Artificial Intelligence (AMAI), 2015, 76, 5-23

Learned model

Probabilistic Ensembles with Trajectory Sampling (PETS)

Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765

Experimental Results

Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765

Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765

Learning to Fly a Quadcopter

Lambert, N.O.; Drew, D.S.; Yaconelli, J; Calandra, R.; Levine, S.; & Pister, K.S.J.
Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning
IEEE Robotics and Automation Letters (RA-L), 2019, 4, 4224-4230

On-line Adaptation to Different Payloads

Belkhale, S.; Li, R.; Kahn, G.; McAllister, R.; Calandra, R. & Levine, S.
Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads

IEEE Robotics and Automation Letters (RA-L), 2021, 6, 1471-1478

Model-based Reinforcement Learning

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Understand and Overcome the Limitations of MBRL

• Can we avoid multiplicative error of recursive one-step predictions?

Lambert, N.; Wilcox, A.; Zhang, H.; Pister, K. S. J. & Calandra, R.
Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning
IEEE Conference on Decision and Control (CDC), 2021

(YES)

• Can we dynamically tune hyperparameters?

• Are accurate models condition necessary for good control performance?

• Are accurate models condition sufficient for good control performance?

Bansal, S.; Calandra, R.; Xiao, T.; Levine, S. & Tomlin, C. J.
Goal-Driven Dynamics Learning via Bayesian Optimization
IEEE Conference on Decision and Control (CDC), 2017, 5168-5173

(NO)

(NO)

Lambert, N.; Amos, B.; Yadan, O. & Calandra, R.
Objective Mismatch in Model-based Reinforcement Learning
Learning for Dynamics and Control (L4DC), 2020, 761-770

Objective Mismatch

[Lambert, N.; Amos, B.; Yadan, O. & Calandra, R. Objective Mismatch in Model-based Reinforcement Learning  Learning for Dynamics and Control (L4DC), 2020, 761-770]

[Wei R., Lambert N., McDonald A., Garcia A., and Calandra R.  2024. “A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning.” Transactions on Machine Learning Research (TMLR). https://arxiv.org/abs/2310.06253.]

Adversarially Generated Dynamics

Exploration as Key for Open-Endedness?

• Shape exploration towards novel and learnable behaviors is key problem in RL
• Foundation models have become quite popular in the realm of RL to provide guidance to the agent
• However, foundation models itself are not considered to be open-ended
• An open-ended foundation model is able to both:
  • vary (i.e. mutate) data
  • assess novelty to decide what to explore

Can we use Diffusion Models to guide exploration and achieve open-endedness in RL?

Hughes et al., 2024 - Open-Endedness is Essential for Artificial Superhuman Intelligence

Jiang et al., 2022 - General intelligence requires rethinking exploration

Contact: elia.ruehle@mailbox.tu-dresden.de

Exploration with Diffusion Models

• Diffusion Models (DM) can be used to diffuse trajectories of variable length
• Idea: guide the model to trajectories that have a high novelty score 

• Open Questions:
  • How to guide? (classifier/classifier-free)
  • How to model the novelty score?
  • Can we use gradients of the DM for guidance?
  • Will the system be open-ended?

Janner et al., 2022 - Planning with Diffusion for Flexible Behaviour Synthesis

Ajay et al., 2023 - Is Conditional Generative Modelling all you need for Decision-Making?

Contact: elia.ruehle@mailbox.tu-dresden.de

Key Challenges

• Multi-modal Sensing

• Adaptive Hardware configuration

• Quick adaptation to new tasks

Touch Sensing

Morphological adaptation

Model-based
Reinforcement Learning

Hardware

Software

Joint Morphology/Controller Optimization

• In Robotics, there is a tight relationship between morphologies and controllers
• Design of morphologies is a complex and time-consuming process
• Can we automate it?
• Same simulated hexapod as before:
  • Each manufacturing round takes about 1 month in real-world...
  • ...But we can fabricate multiple different morphology configurations at once (up to 5)

Liao, T.; Wang, G.; Yang, B.; Lee, R.; Pister, K.; Levine, S. & Calandra, R.
Data-efficient Learning of Morphology and Controller for a Microrobot
IEEE International Conference on Robotics and Automation (ICRA), 2019

Co-design of Hardware and Software

Results

Results - Learned Model

Luck, K.; Amor, H. B. & Calandra, R.
Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning
Conference on Robot Learning (CORL), 2019

To Conclude

Human Collaborators

LASR Lab @ TU Dresden

Supported by

2st Workshop on Touch Processing

www.touchprocessing.org/2024/

December 2024 @ NeurIPS
Vancouver

Overview

• LASR works at the intersection of AI and Robotics
• Three main axes:
  • Digitalization of touch
  • Develop novel AI algorithms for decision-making
  • Enable new real-world robot capabilities

• Towards the long-term goal of making Robots more useful in the real-world (and understand more about AI along the way)

Our work on touch sensing

• Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
  TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
  IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456

• Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
  Active 3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2021
• Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
  PyTouch: A Machine Learning Library for Touch Processing
  IEEE International Conference on Robotics and Automation (ICRA), 2021
• Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
  3D Shape Reconstruction from Vision and Touch
  Advances in Neural Information Processing Systems (NeurIPS), 2020
• Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
  IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
• Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
  OmniTact: A Multi-Directional High-Resolution Touch Sensor
  IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
• Lin, J.; Calandra, R. & Levine, S.
  Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650
• Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
  Manipulation by Feel: Touch-Based Control with Deep Predictive Models
  IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824
• Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
  More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
  IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
• Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
  The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
  Conference on Robot Learning (CORL), 2017, 314-323
• Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
  Active Tactile Object Exploration with Gaussian Processes
  IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
  Learning Inverse Dynamics Models with Contacts
  IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191
• Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
  Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
  IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695

Additional Slides

Slack Channel

www.touch-sensing.org

Model

Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979

Model-based Reinforcement Learning

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

In-Hand Object Rotation via Rapid Motor Adaptation

We are Hiring

https://lasr.org/join-us/

• Post-docs
• PhD Students
• Robot Lab Manager

Touch for the Metaverse

• Touch is a deeply social and emotional sense
• How can we sense, calibrate and reproduce touch?
• How can we accurately and easily digitize objects and their physical properties into virtual worlds?

Telepresence via Haptic Feedback

Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015

Robot Learning

Code Example

Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323

Self-supervised Data Collection

• Setting:
  • 7-DOF Sawyer arm
  • Weiss WSG-50 Parallel gripper
  • one GelSight on each finger
  • Two RGB-D cameras in front and on top

• (Almost) fully autonomous data collection:
  • Estimates the object position using depth, and perform a random grasp of the object.
  • Labels automatically generated by looking at the presence of contacts after each attempted lift

Traditional Sensors

Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438

• Resistive or Capacitive technology
• Several limitations:
  • Usually, measure force or orthogonal component of force
  • Relatively low density
  • Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
  • Often need to be calibrated

Features of PyTouch

• First Machine Learning library dedicated to Touch Processing
  (Based on PyTorch)
• Hardware-agnostic abstractions for rapid experimentation
• Native integration with DIGIT (hardware) and Tacto (simulator)
  (Working on supporting non vision-based sensors)
• Platform for standardizing evaluation and comparison of different models
• Touch "as-a-service"
  • Allows non-ML-experts to use SOTA ML models in their applications
  • Pre-trained models (e.g., Touch detection and slip for DIGIT)
    • (Can also be used for fast fine-tuning)
• Open-source -- Anybody can contribute

Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791

Learning to Play Piano with Touch

Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, online: https://arxiv.org/abs/2106.02040

Do we really need Touch?

Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887

Failure Case

Understanding the Learned Model

More force = better grasp

Understanding the Learned Model

But not always ?

Understanding the Learned Model

Gentle Grasping

• Since our model considers forces, we can select grasps that are effective, but gentle
• Reduces the amount of force used by 50%, with no significant loss in grasp success

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307