Roberto Calandra PRO
Full Professor at TU Dresden. Head of the LASR Lab. Working in AI, Robotics and Touch Sensing.
Tactile Sensing for Manipulation: Trends, Challenges, and Opportunities
Roberto Calandra
9.357 Special Topics in Perception - 22/02/2022
Facebook AI Research
Motivation
How to scale to more complex, unstructured domains?
Robotics
Finance
Biological Sciences
Logistics /
Decision Making
Why Robots?
Disaster Relief
Industrial Automation
Exploration
Medicine & Eldercare
State of the Art in Robotics
From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo
What are we missing?
Key Challenges
-
Multi-modal Sensing
-
Optimized Hardware Design
-
Quick adaptation to new tasks
Touch Sensing
Morphological adaptation
In this talk
Model-based
Reinforcement Learning
Hardware
Software
The Importance of Touch
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
The Importance of Touch (in Humans)
The Importance of Touch (in Robots)
1. Touch is Useful
Learning Grasp Stability
Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Self-supervised Data Collection
- Setting:
- 7-DOF Sawyer arm
- Weiss WSG-50 Parallel gripper
- one GelSight on each finger
- Two RGB-D cameras in front and on top
- (Almost) fully autonomous data collection:
- Estimates the object position using depth, and perform a random grasp of the object.
- Labels automatically generated by looking at the presence of contacts after each attempted lift
Examples of Training Objects
Collected 9269 grasps from over 106 training objects over ~2 weeks.
Classification Accuracy
Grasping Success Rate
Touch improves grasping performance compared to vision only
Visuo-tactile Learned Model
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Examples of Training Objects
Collected 6450 grasps from over 60 training objects over ~2 weeks.
Grasp Success on Unseen Objects
83.8% grasp success on 22 unseen objects using vision and touch
(using only vision yields 56.6% success rate)
Full Grasping Results
Failure Case
Understanding the Learned Model
More force = better grasp
Understanding the Learned Model
But not always ?
Understanding the Learned Model
Gentle Grasping
- Since our model considers forces, we can select grasps that are effective, but gentle
- Reduces the amount of force used by 50%, with no significant loss in grasp success
Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Manipulation as Force Balancing Tasks
- Manipulation tasks are often defined in terms of positions and velocities
- Many manipulation tasks should instead be defined in terms of forces
Filipe Veiga , Benoni Edin, and Jan Peters. "Grip stabilization through independent finger tactile feedback control." Sensors 20.6 (2020): 1748.
2. Hardware is Getting There
Tactile Sensors in Robotics
Important factors:
- Availability
- Cost
- Form factor
- Capabilities
(e.g., what is measured, resolution) - Reliability
Many many sensors in the literature:
- Most are prototypes
- A handful are commercially available
or can be easily manufactured
[Wilson et al., 2019]
[Piacenza et al., 2020]
[ Fischel et al., 2012]
[Zhang et al., 2018]
[Church et al., 2019]
Vision-based Tactile Sensors
[Hillis, W. D. A High-Resolution Imaging Touch Sensor The International Journal of Robotics Research, 1982, 1, 33-44 ]
[Tanie, K.; Komoriya, K.; Kaneko, M.; Tachi, S. & Fujikawa, A. A high resollution tactile sensor Proc. of 4th Int. Conf. on Robot Vision and Sensory Controls, 1984, 251, 260]
[Begej, S. Planar and finger-shaped optical tactile sensors for robotic applications IEEE Journal on Robotics and Automation, 1988, 4, 472-484]
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]
[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]
[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]
Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]
Examples of GelSight Measurements
OmniTact
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
OmniTact
Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624
3. To Lower the Entry-bar
Touch Sensing is Hard
- Interdisciplinary field which requires vertical integration. From hardware design to touch processing; From robot control to applications.
- Many ad-hoc solutions and little re-use of existing components
(i.e., We keep reinventing the wheel over and over) - High entrance bar for new researchers and practitioners
- How can we lower the entrance bar?
- How can we improve reproducibility?
- How can we accelerate research by re-using existing components?
Standardization, and the creation of an ecosystem of tools
Hardware
Software
What do we need to make Touch Sensing practical and useful?
Making Touch Sensing Ubiquitous
DIGIT
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Examples of DIGIT Measurements
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Elastomer Robustness
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Replaceable Elastomer
Reflective
Reflective
+
Markers
Transparent
+
Markers
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Comparison
BioTac
DIGIT
~15,000 $
Cost
~15 $*
Resolution
29
contact points
307,200
contact points
Mounted on multi-finger hands
Open-source
1000x
Higher resolution
1000x
Cheaper
* component cost for 1000 units, not including labor
DIGIT Availability
-
Replicated in 15+ universities
-
Yet, it can still be challenging to manufacture a sensor without mechanical/electrical experience
-
Partnership with GelSight Inc. to commercialize DIGIT
Making Touch Sensing Ubiquitous
Touch Simulation
Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics
- Most rigid-body-dynamics physics engines include some form of traditional tactile sensor (i.e., low-dimensional)
- These simulators became slower with the increasing number of contact points
- For 1000+ contact points provided by vision-based tactile sensors, they became impractical
TACTO
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Software Architecture
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Rendering from Simulated DIGIT
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Perceptual Sim2Real
Making Touch Sensing Ubiquitous
The Next Breakthrough will be in Touch
Audio
Touch
Vision
(~1890)
(~1990)
(2020s ?)
Creating a Science of Touch Processing
Some of the open questions:
- What are good features for touch?
-
Do we need sensor standardization?
- What representation do we want/need for touch?
- What sensorial information do we even want/need for touch?
- What are the useful structures in computational models for touch?
- What are the useful metrics to characterize touch?
- How can we quantify the human psychophysics of touch?
- What are the different tasks that can benefit from touch?
- What are meaningful benchmarks for touch processing?
Very limited literature about computational processing of touch sensing
PyTouch: A Machine Learning Library
for Touch Processing
Goal: Create the equivalent of OpenCV for Touch
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Features of PyTouch
- First Machine Learning library dedicated to Touch Processing
(Based on PyTorch) - Hardware-agnostic abstractions for rapid experimentation
- Native integration with DIGIT (hardware) and Tacto (simulator)
(Working on supporting non vision-based sensors) - Platform for standardizing evaluation and comparison of different models
- Touch "as-a-service"
- Allows non-ML-experts to use SOTA ML models in their applications
- Pre-trained models (e.g., Touch detection and slip for DIGIT)
- (Can also be used for fast fine-tuning)
- Open-source -- Anybody can contribute
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Code Example
Learning from Multiple Sensors Improve
Performance (Touch Detection Task)
Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021, Online: https://arxiv.org/abs/2105.12791
Making Touch Sensing Ubiquitous
Benchmarks & Datasets
Benchmarks:
Datasets:
- What are the tasks that we care about as a community?
- Now is the time to think about Benchmarks:
- The first enabler is accurate touch simulators (e.g., TACTO)
- The second enabler is standardized easily available hardware (e.g., DIGIT)
- Very few touch sensing datasets available nowadays
- Data collection has been so far limited by the reliability of hardware
- Simulators, and ready-available hardware will enable new and larger datasets
(But we also need as a community to encourage and nurture this)
Learning Grasp Stability
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456
Democratizing Touch Sensing
- Open-sourced an Ecosystem for touch sensing :
- Hardware (DIGIT)
- Simulator (TACTO)
- Touch processing library (PyTouch)
- Partnership with GelSight Inc. to commercialize DIGIT
- Organized a workshop on "The Future of Touch Sensing: Applications and Challenges"
- Participants received 200 free DIGIT sensors
- Pre-recorded talks by the participants are available online
- More standardization necessary
4. Applications
Making Touch Sensing Ubiquitous
Some Touch Sensing Applications
Predicting Grasp Stability
[Calandra et al. 2017]
Learning how to (Re)Grasp
[Calandra et al. 2018]
Active Tactile Exploration
[Yi at al. 2016]
3D Reconstruction from Vision and Touch
[Smith et al. 2020]
Identify Objects from Touch
[Lin et al. 2019]
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Model-based Reinforcement Learning
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Marble Manipulation
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Active 3D Reconstruction from Vision and Touch
Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
Active 3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2021, online: https://arxiv.org/abs/2107.09584
Learning to Play Piano with Touch
Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
Under Review, 2021, online: https://arxiv.org/abs/2106.02040
Learned Compositional Policy
Xu, H.; Luo, Y.; Wang, S.; Darrell, T. & Calandra, R.
Towards Learning to Play Piano with Dexterous Hands and Touch
Under Review, 2021, online: https://arxiv.org/abs/2106.02040
Future Outlook
Touch vs Soft Robots
- Argument: Do we even need human-like hands?
- Ni (si/no)
- Maybe tentacles-like would be better?
- Sensing in soft robots is difficult
Future Challenges
- Hardware development
- Several key challenges before reaching human-level capabilities
- Multi-modal sensing
(Surface geometry is not enough) - Use of custom solutions
(Current sensors are built from off-the-shelf components)
-
Touch Processing
- In 5 years from now, most of the research will be on making sense of Touch
(Like computer vision. Little interest in designing your own camera)
- In 5 years from now, most of the research will be on making sense of Touch
-
Integrating Touch into Control
- How do we achieve human-level dexterity?
- Touch is an intrinsically dynamic and interactive sense
- More Applications ...
Opportunities
- Massive potential for growth in robotics:
- Industrial manipulation
- Agricultural robotics
- Marine robotics
- Telemedicine
- Metaverse (AR/VR)
- E-commerce
Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 997-1002
To Conclude
Human Collaborators
Slack Channel
Internship Summer 2022
Looking for one intern at the conjunction between Robotics and Touch Sensing
If interested write me NOW!
rcalandra@fb.com
Overview
- Touch is a key sensor modality for humans and robots
- Given overview of some of trends, challenges and opportunities
- Presented our ecosystem that simplify integration and enable to perceive, understand, and interact through touch
- Hardware
- Simulation
- Touch Processing
- Benchmarks and Datasets
- Towards the long-term goal of making touch sensing practical and more accessible (both in research and in the real-world)
- Exciting, fast-growing field!
DIGIT
TACTO
PyTouch
Our work on touch sensing
-
Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
IEEE Robotics and Automation Letters (RA-L), 2022, 7, 3930-3937, Online: https://arxiv.org/abs/2012.08456 - Smith, E. J.; Meger, D.; Pineda, L.; Calandra, R.; Malik, J.; Romero, A. & Drozdzal, M.
Active 3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2021 - Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
PyTouch: A Machine Learning Library for Touch Processing
IEEE International Conference on Robotics and Automation (ICRA), 2021 - Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020 - Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845 - Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020, 618-624 - Lin, J.; Calandra, R. & Levine, S.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650 - Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824 - Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307 - Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323 - Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
Active Tactile Object Exploration with Gaussian Processes
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
Learning Inverse Dynamics Models with Contacts
IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695
Tactile Sensing for Manipulation: Trends, Challenges, and Opportunities Roberto Calandra 9.357 Special Topics in Perception - 22/02/2022 Facebook AI Research https://slides.com/rcalandra/mit-9357
MIT
By Roberto Calandra
MIT
- 939
