Roberto Calandra
University of Oxford - 26 May 2023
Learning, Adaptive Systems, and Robotics (LASR) Lab
How to scale to more complex, unstructured domains?
Robotics
Finance
Biological Sciences
Logistics /
Decision Making
Disaster Relief
Industrial Automation
Exploration
Medicine & Eldercare
From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo
Multi-modal Sensing
Adaptive Hardware configuration
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
In this talk
Model-based
Reinforcement Learning
Hardware
Software
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
Standardization, and the creation of an ecosystem of tools
Hardware
Software
(~1890)
(~1990)
(2020s ?)
+ Applications
+ Community
Important factors:
Many many sensors in the literature:
[Wilson et al., 2019]
[Piacenza et al., 2020]
[ Fischel et al., 2012]
[Zhang et al., 2018]
[Church et al., 2019]
[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]
[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]
[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]
[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]
[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]
Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
BioTac
DIGIT
~15,000 $
Cost
~15 $*
Resolution
29
taxels
307,200
taxels
Mounted on multi-finger hands
Open-source
1000x
Higher resolution
1000x
Cheaper
* component cost for 1000 units, not including labor
Replicated in 20+ universities
Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience
Partnership with GelSight Inc. to commercialize DIGIT
Most widespread tactile sensor in robotics!
Part of Mitsubishi Electric RAISE (Robotics as an Intelligent Services Ecosystem)
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C.; Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA) , 2020
Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456
Some of the open questions:
Very limited literature about computational processing of touch sensing
Goal: Create the equivalent of OpenCV for Touch
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, Ken Goldberg
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Accepted to RSS 2023, Online: https://arxiv.org/pdf/2209.13042
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, Online: https://arxiv.org/abs/2012.08456
Robotics
Metaverse
(AR/VR)
E-commerce
Prosthesis
Sense
Process
Reproduce
Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015
Predicting Grasp Stability
[Calandra et al. 2017]
Learning how to (Re)Grasp
[Calandra et al. 2018]
Active Tactile Exploration
[Yi at al. 2016]
3D Reconstruction from Vision and Touch
[Smith et al. 2020; Smith et al. 2021]
Identify Objects from Touch
[Lin et al. 2019]
Learning to Play Piano from Touch
[Xu at al. 2022]
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Qi, H.; Kumar, A.; Calandra, R.; Ma, Y. & Malik J.
In-Hand Object Rotation via Rapid Motor Adaptation
Conference on Robot Learning (CORL), 2022, https://arxiv.org/abs/2210.04887
Thank you!
DIGIT
TACTO
PyTouch
Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Reflective
Reflective
+
Markers
Transparent
+
Markers
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Collected 6450 grasps from over 60 training objects over ~2 weeks.
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438
Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022, online: https://arxiv.org/abs/2106.02040