Roberto Calandra PRO
Full Professor at TU Dresden. Head of the LASR Lab. Working in AI, Robotics and Touch Sensing.
Towards Embodied
Artificial Intelligence
Roberto Calandra
Oxford University - 26 August 2024
Learning, Adaptive Systems, and Robotics (LASR) Lab
Why Robots?
Disaster Relief
Industrial Automation
Exploration
Medicine & Eldercare
State of the Art in Robotics
From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo
What are we missing?
Our Mission
1) Make Robots more useful in the real world
(Through the use of AI)
2) Advance AI
(Using embodiment as a case study)
Key Challenges
-
Multi-modal Sensing
-
Adaptive Hardware configuration
-
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
In this talk
Model-based
Reinforcement Learning
Hardware
Software
Key Challenges
-
Multi-modal Sensing
-
Adaptive Hardware configuration
-
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
Model-based
Reinforcement Learning
Hardware
Software
The Importance of Touch
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
The Importance of Touch (in Humans)
The Next Breakthrough will be in Touch
Audio
Touch
Vision
(~1890)
(~1990)
(2020s ?)
Grand Vision
+ Applications
+ Community
Hardware
Vision-based Tactile Sensors
[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]
[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]
[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]
[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]
[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]
Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]
DIGIT
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Examples of DIGIT Measurements
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Comparison
BioTac
DIGIT
~15,000 $
Cost
~15 $*
Resolution
29
taxels
307,200
taxels
Mounted on multi-finger hands
Open-source
1000x
Higher resolution
1000x
Cheaper
* component cost for 1000 units, not including labor
DIGIT Commercialization
-
Replicated in 20+ universities
-
Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience
-
Partnership with GelSight Inc. to commercialize DIGIT
-
Most widespread tactile sensor in robotics!
-
Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)
-
OmniTact
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020
Evetac
Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
Under Review, 2023 https://arxiv.org/abs/2312.01236
- Neuromorphic tactile sensor
- Event-based
- Up to 1KHz
- Modular and using off-the-shelf components
(Inspired by the DigiTac by Nathan)
Bandwidth
Funk, N.; Helmut, E.; Chalvatzaki, G.; Calandra, R. & Peters, J.
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
Accepted to T-RO, 2023 https://arxiv.org/abs/2312.01236
12%
1.7% over
entire
trajectory
DIGIT Pinky
Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500
DIGIT Pinky
Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500
Cancer Detection in Prostate Tissue
Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500
Cancer Detection in Prostate Tissue
Cancer
No Cancer
Di, J.; Dugonjic, Z.; Fu, W.; Wu, T.; Mercado, R.; Sawyer, K.; Most, V. R.; Kammerer, G.; Speidel, S.; Fan, R. E.; Sonn, G.; Cutkosky, M. R.; Lambeta, M. & Calandra, R.
Using Fiber Optic Bundles to Miniaturize Vision-Based Tactile Sensors
Accepted to IEEE Transactions on Robotics (T-RO), 2024 https://arxiv.org/abs/2403.05500
Touch Processing
(i.e., AI for Touch)
Creating a New Science of Touch Processing
Some of the open questions:
- What are good features for touch?
-
Do we need sensor standardization?
- What representation do we want/need for touch?
- What sensorial information do we even want/need for touch?
- What are the useful structures in computational models for touch?
- What are the useful metrics to characterize touch?
- How can we quantify the human psychophysics of touch?
- What are the different tasks that can benefit from touch?
- What are meaningful benchmarks for touch processing?
Very limited literature about computational processing of touch sensing
PyTouch: A Machine Learning Library
for Touch Processing
Goal: Create the equivalent of OpenCV for Touch
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
What Representations do we Need for Touch?
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021
Kerr, J.; Huang, H.; Wilcox, A.; Hoque, R.; Ichnowski, J.; Calandra, R. & Goldberg, K.
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Robotics: Science and Systems (RSS) 2023, Online: https://arxiv.org/pdf/2209.13042
Transfer Across Sensors
Transfer across Tasks
Touch and Language
Fu, L.; Datta, G.; Huang, H.; Panitch, W. C.-H.; Drake, J.; Ortiz, J.; Mukadam, M.; Lambeta, M.; Calandra, R. & Goldberg, K.
A Touch, Vision, and Language Dataset for Multimodal Alignment
ICML 2024 https://arxiv.org/abs/2402.13232
Tactile SLAM
Suresh, S.; Qi, H.; Wu, T.; Fan, T.; Pineda, L.; Lambeta, M.; Malik, J.; Kalakrishnan, M.; Calandra, R.; Kaess, M.; Ortiz, J. & Mukadam, M.
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Under Review, 2023 https://arxiv.org/abs/2312.13469
Applications
Key Applications
Robotics
Metaverse
(AR/VR)
E-commerce
Medical
Some Touch Sensing Applications
Predicting Grasp Stability
[Calandra et al. 2017]
Learning how to (Re)Grasp
[Calandra et al. 2018]
Active Tactile Exploration
[Yi at al. 2016]
3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]
Identify Objects from Touch
[Lin et al. 2019]
Learning to Play Piano from Touch
[Xu at al. 2022]
Visuo-tactile Learned Model
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Examples of Training Objects
Collected 6450 grasps from over 60 training objects over ~2 weeks.
Grasp Success on Unseen Objects
83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Achieving Human-level Manipulation with Robots
Learning General In-Hand Rotation
Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979
Learning General In-Hand Rotation
Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979
Key Challenges
-
Multi-modal Sensing
-
Adaptive Hardware configuration
-
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
Model-based
Reinforcement Learning
Hardware
Software
Learning Models of the World
Humans seem to make extensive use of models for planning and control *
(e.g., to predict the effects of our actions)
Can Robots learn and use models of the world?
Hyphothesis: better predictive capabilities will lead to more efficient adaptation
* [Kawato, M. Internal models for motor control and trajectory planning Current Opinion in Neurobiology , 1999, 9, 718 - 727],
[Gläscher, J.; Daw, N.; Dayan, P. & O'Doherty, J. P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement
learning Neuron, Elsevier, 2010, 66, 585-595]
+
predictive models allow us to peek into the beliefs of our models
and explain their decisions
Learning to Walk in 80 Trials
Calandra, R.; Seyfarth, A.; Peters, J. & Deisenroth, M. P.
Bayesian Optimization for Learning Gaits under Uncertainty
Annals of Mathematics and Artificial Intelligence (AMAI), 2015, 76, 5-23
Learned model
Not Symmetrical (about 5° difference). Why?
Because it is walking in a circle!
Probabilistic Ensembles with Trajectory Sampling (PETS)
Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765
Experimental Results
Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765
Chua, K.; Calandra, R.; McAllister, R. & Levine, S.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Advances in Neural Information Processing Systems (NIPS), 2018, 4754-4765
Learning to Fly a Quadcopter
Lambert, N.O.; Drew, D.S.; Yaconelli, J; Calandra, R.; Levine, S.; & Pister, K.S.J.
Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning
IEEE Robotics and Automation Letters (RA-L), 2019, 4, 4224-4230
On-line Adaptation to Different Payloads
Belkhale, S.; Li, R.; Kahn, G.; McAllister, R.; Calandra, R. & Levine, S.
Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
IEEE Robotics and Automation Letters (RA-L), 2021, 6, 1471-1478
Model-based Reinforcement Learning
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Understand and Overcome the Limitations of MBRL
- Can we avoid multiplicative error of recursive one-step predictions?
Lambert, N.; Wilcox, A.; Zhang, H.; Pister, K. S. J. & Calandra, R.
Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning
IEEE Conference on Decision and Control (CDC), 2021
(YES)
- Can we dynamically tune hyperparameters?
Zhang, B.; Rajan, R.; Pineda, L.; Lambert, N.; Biedenkapp, A.; Chua, K.; Hutter, F. & Calandra, R.
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
International Conference on Artificial Intelligence and Statistics (AISTATS), 2021
(YES)
- Are accurate models condition necessary for good control performance?
- Are accurate models condition sufficient for good control performance?
Bansal, S.; Calandra, R.; Xiao, T.; Levine, S. & Tomlin, C. J.
Goal-Driven Dynamics Learning via Bayesian Optimization
IEEE Conference on Decision and Control (CDC), 2017, 5168-5173
(NO)
(NO)
Lambert, N.; Amos, B.; Yadan, O. & Calandra, R.
Objective Mismatch in Model-based Reinforcement Learning
Learning for Dynamics and Control (L4DC), 2020, 761-770
Objective Mismatch
Negative Log-Likelihood
Task Reward
[Lambert, N.; Amos, B.; Yadan, O. & Calandra, R. Objective Mismatch in Model-based Reinforcement Learning Learning for Dynamics and Control (L4DC), 2020, 761-770]
[Wei R., Lambert N., McDonald A., Garcia A., and Calandra R. 2024. “A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning.” Transactions on Machine Learning Research (TMLR). https://arxiv.org/abs/2310.06253.]
Adversarially Generated Dynamics
Key Challenges
-
Multi-modal Sensing
-
Adaptive Hardware configuration
-
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
Model-based
Reinforcement Learning
Hardware
Software
Joint Morphology/Controller Optimization
- In Robotics, there is a tight relationship between morphologies and controllers
- Design of morphologies is a complex and time-consuming process
- Can we automate it?
- Same simulated hexapod as before:
- Each manufacturing round takes about 1 month in real-world...
- ...But we can fabricate multiple different morphology configurations at once (up to 5)
Liao, T.; Wang, G.; Yang, B.; Lee, R.; Pister, K.; Levine, S. & Calandra, R.
Data-efficient Learning of Morphology and Controller for a Microrobot
IEEE International Conference on Robotics and Automation (ICRA), 2019
Co-design of Hardware and Software
Thomas L., Wang G., Yang B., Lee R., Pister K., Levine S., and Calandra R.
Data-Efficient Learning of Morphology and Controller for a Microrobot
IEEE International Conference on Robotics and Automation (ICRA), 2488–2494
Results
From 21 months down to 4 months
5x faster!
Results - Learned Model
Luck, K.; Amor, H. B. & Calandra, R.
Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning
Conference on Robot Learning (CORL), 2019
To Conclude
Human Collaborators
LASR Lab @ TU Dresden
Supported by
2st Workshop on Touch Processing
December 2024 @ NeurIPS
Vancouver
Overview
- LASR works at the intersection of AI and Robotics
- Three main axes:
- Digitalization of touch
- Develop novel AI algorithms for decision-making
- Enable new real-world robot capabilities
- Towards the long-term goal of making Robots more useful in the real-world (and understand more about AI along the way)
Thank you!
Our work on touch sensing
-
Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456 - Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
Active 3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2021 - Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021 - Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020 - Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845 - Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624 - Lin, J.; Calandra, R. & Levine, S.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650 - Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824 - Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307 - Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323 - Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
Active Tactile Object Exploration with Gaussian Processes
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
Learning Inverse Dynamics Models with Contacts
IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695
Additional Slides
Slack Channel
Model
Qi, Haozhi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.
General In-Hand Object Rotation with Vision and Touch.
Conference on Robot Learning (CORL). 2023 https://arxiv.org/abs/2309.09979
Model-based Reinforcement Learning
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
In-Hand Object Rotation via Rapid Motor Adaptation
We are Hiring
- Post-docs
- PhD Students
- Robot Lab Manager
Code Example
Learning Grasp Stability
Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Self-supervised Data Collection
- Setting:
- 7-DOF Sawyer arm
- Weiss WSG-50 Parallel gripper
- one GelSight on each finger
- Two RGB-D cameras in front and on top
- (Almost) fully autonomous data collection:
- Estimates the object position using depth, and perform a random grasp of the object.
- Labels automatically generated by looking at the presence of contacts after each attempted lift
Traditional Sensors
Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438
- Resistive or Capacitive technology
- Several limitations:
- Usually, measure force or orthogonal component of force
- Relatively low density
- Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
- Often need to be calibrated
Features of PyTouch
- First Machine Learning library dedicated to Touch Processing
(Based on PyTorch) - Hardware-agnostic abstractions for rapid experimentation
- Native integration with DIGIT (hardware) and Tacto (simulator)
(Working on supporting non vision-based sensors) - Platform for standardizing evaluation and comparison of different models
- Touch "as-a-service"
- Allows non-ML-experts to use SOTA ML models in their applications
- Pre-trained models (e.g., Touch detection and slip for DIGIT)
- (Can also be used for fast fine-tuning)
- Open-source -- Anybody can contribute
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Learning to Play Piano with Touch
Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, online: https://arxiv.org/abs/2106.02040
Do we really need Touch?
Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887
Failure Case
Understanding the Learned Model
More force = better grasp
Understanding the Learned Model
But not always ?
Understanding the Learned Model
Gentle Grasping
- Since our model considers forces, we can select grasps that are effective, but gentle
- Reduces the amount of force used by 50%, with no significant loss in grasp success
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Towards Embodied Artificial Intelligence Roberto Calandra Oxford University - 26 August 2024 Learning, Adaptive Systems, and Robotics (LASR) Lab https://slides.com/rcalandra/2024-oxford-university
Oxford University
By Roberto Calandra
Oxford University
- 295
