Perceiving, Understanding, and Interacting through Touch
Roberto Calandra
UC Berkeley - CS280 - 29 Apr 2021
Facebook AI Research
The Importance of Touch
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
The Importance of Touch (in Humans)
The Importance of Touch (in Robots)






Touch for Robot Manipulation
[Allen et al. 1999]
[Chebotar et al. 2016]
[Bekiroglu et al. 2011]
[Sommer and Billard 2016]
[Schill et al. 2012]





The Next Breakthrough will be in Touch
Audio
Touch
Vision

(~1890)
(~1990)
(2020s ?)






Touch and Vision are Complementary



What do we need to make Touch Sensing practical and useful?

Making Touch Sensing Ubiquitous


Tactile Sensors in Robotics
Important factors:
- Availability
- Cost
- Form factor
- Capabilities
(e.g., what is measured, resolution) - Reliability

Many many sensors in the literature:
- Most are prototypes
- A handful are commercially available
or can be easily manufactured


[Wilson et al., 2019]
[Piacenza et al., 2020]
[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]
Traditional Sensors


Cannata, G.; Maggiali, M.; Metta, G. & Sandini, G.
An embedded artificial skin for humanoid robots
IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2008, 434-438
- Resistive or Capacitive technology
- Several limitations:
- Usually, measure force or orthogonal component of force
- Relatively low density
- Usually, low-dimensional (i.e., <100) due to cost, mechanical and communication reasons
- Often need to be calibrated
Vision-based Tactile Sensors
[Kamiyama, K.; Kajimoto, H.; Kawakami, N. & Tachi, S. Evaluation of a vision-based tactile sensor IEEE International Conference on Robotics and Automation (ICRA), 2004, 2, 1542-1547 ]
[Johnson, M. K. & Adelson, E. H. Retrographic sensing for the measurement of surface texture and shape Computer Vision and Pattern Recognition (CVPR), 2009, 1070-1077]
[Abad, A. C. & Ranasinghe, A. Visuotactile Sensors With Emphasis on GelSight Sensor: A Review IEEE Sensors Journal, 2020, 20, 7628-7638]

Credit:
[Yuan, W.; Dong, S. & Adelson, E. H. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors, Multidisciplinary Digital Publishing Institute, 2017]
DIGIT

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Comparison
BioTac
DIGIT

~10,000 $
Cost
~15 $
Resolution
29
contact points
307,200
contact points
Mounted on multi-finger hands


Open-source



Being replicated in 15+ universities
1000x
Higher resolution
1000x
Cheaper
Elastomer Robustness

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Replaceable Elastomer




Reflective
Reflective
+
Markers
Transparent
+
Markers
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845

Making Touch Sensing Ubiquitous

Touch Simulators
Simulators play a crucial role in prototyping, debugging and benchmarking new advances in robotics
- Most rigid-body-dynamics physics engines include some form of traditional tactile sensor (i.e., low-dimensional)
- These simulators became slower with the increasing number of contact points
- For the hundreds of thousands of contact points provided by vision-based tactile sensors, they became impractical
- No dedicated simulator for vision-based tactile sensors so far...

Simulator Desiderata
- Fast, fast, fast... (>100Hz)
- General purpose
(in the same way that a physics engine can simulate any rigid body robot) - Flexible
- Accurate
- In terms of output
(Realistic rendering calibrated on real sensor) - In terms of physics
(Not important for perceptual tasks. Very difficult to model non-rigid contacts accurately)
- In terms of output
- Open-source
TACTO
Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456
Software Architecture

Principle
Realistic Rendering

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456


Making Touch Sensing Ubiquitous
Towards a Science of Touch Processing
-
Very limited literature about computational processing of touch sensing
- What information do we need from touch?
- What are good features for touch?
- What are the useful structures in computational models for touch?
- What are the useful metrics to characterize touch?
- What are meaningful benchmarks for touch processing?
- The literature so far in Touch Processing has been using traditional CV models
- Touch has a very pronounced temporal aspect
Making Touch Sensing Ubiquitous


Benchmarks & Datasets
Benchmarks:
Datasets:
- What are the tasks that we care about as a community?
- The first great enabler of benchmarks is accurate touch simulators
- The second great enabler of benchmarks is easily available hardware
- Very few touch sensing datasets available nowadays
- Data collection is currently limited by the reliability of hardware
- Simulators, and available hardware will enable new and larger datasets
(But we also need as a community to encourage and nurture this)
Learning Grasp Stability

Wang, S.; Lambeta, M.; Chou, P.-W. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456
Making Touch Sensing Ubiquitous


Some Touch Sensing Applications
Predicting Good Grasps
Learning how to Grasp
Active Tactile Exploration
3D Reconstruction from Vision and Touch
Identify Objects from Touch




Learning Grasp Stability

Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323
Results

Self-supervised Data Collection
- Setting:
- 7-DOF Sawyer arm
- Weiss WSG-50 Parallel gripper
- one GelSight on each finger
- Two RGB-D cameras in front and on top
- (Almost) fully autonomous data collection:
- Estimates the object position using depth, and perform a random grasp of the object.
- Labels automatically generated by looking at the presence of contacts after each attempted lift
Examples of Training Objects
Collected 6450 grasps from over 60 training objects over ~2 weeks.

Visuo-tactile Learned Model

Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Grasp Success on Unseen Objects
83.8% grasp success on 22 unseen objects
(using only vision yields 56.6% success rate)
Gentle Grasping
- Since our model considers forces, we can select grasps that are effective, but gentle
- Reduces the amount of force used by 50%, with no significant loss in grasp success


Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307
Touch Sensing for Tele-operation
Fritsche, L.; Unverzagt, F.; Peters, J. & Calandra, R.
First-Person Tele-Operation of a Humanoid Robot
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 997-1002

Learning Fine In-finger Manipulation

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Model-based Reinforcement Learning
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Marble Manipulation



Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Towards Practical Touch Sensing
- Touch is a key sensor modality for humans and robots
- Touch complements vision by providing rich, but highly localized information
- Today, we briefly discussed several important aspects of touch:
- Hardware
- Simulator
- Touch Processing
- Benchmarks and Datasets
- Some Applications
- Vision-based Tactile sensing are leading to a renaissance of Touch

Feedback at: https://tinyurl.com/cs280-touch
References (of our work on touch sensing)
-
Wang, S.; Lambeta, M.; Chou, L. & Calandra, R.
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
Under Review, 2020, Online: https://arxiv.org/abs/2012.08456 - Lambeta, M.; Xu, H.; Xu, J.; Chou, P.-W.; Wang, S.; Darrell, T. & Calandra, R.
A Machine Learning Framework for Touch Processing
Under Review, 2020 -
Smith, E. J.; Calandra, R.; Romero, A.; Gkioxari, G.; Meger, D.; Malik, J. & Drozdzal, M.
3D Shape Reconstruction from Vision and Touch
Advances in Neural Information Processing Systems (NeurIPS), 2020 - Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845 - Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020 - Lin, J.; Calandra, R. & Levine, S.
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
IEEE International Conference on Robotics and Automation (ICRA), 2019, 3644-3650 - Tian, S.; Ebert, F.; Jayaraman, D.; Mudigonda, M.; Finn, C.; Calandra, R. & Levine, S.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
IEEE International Conference on Robotics and Automation (ICRA), 2019, 818-824 - Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307 - Calandra, R.; Owens, A.; Upadhyaya, M.; Yuan, W.; Lin, J.; Adelson, E. H. & Levine, S.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Conference on Robot Learning (CORL), 2017, 314-323 - Yi, Z.; Calandra, R.; Veiga, F. F.; van Hoof, H.; Hermans, T.; Zhang, Y. & Peters, J.
Active Tactile Object Exploration with Gaussian Processes
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, 4925-4930 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P.; Rueckert, E. & Peters, J.
Learning Inverse Dynamics Models with Contacts
IEEE International Conference on Robotics and Automation (ICRA), 2015, 3186-3191 - Calandra, R.; Ivaldi, S.; Deisenroth, M. P. & Peters, J.
Learning Torque Control in Presence of Contacts using Tactile Sensing from Robot Skin
IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), 2015, 690-695
Learning to Manipulate a Marble
Marble Manipulation Results


Motivation
How to scale to more complex, unstructured domains?








Robotics
Finance
Biological Sciences
Logistics /
Decision Making
Why Robots?
Disaster Relief

Industrial Automation


Exploration


Medicine & Eldercare


State of the Art in Robotics
From YouTube: https://www.youtube.com/watch?v=g0TaYhjpOfo
What are we missing?

Perceiving, Understanding, and Interacting through Touch [UC Berkeley]
By Roberto Calandra
Perceiving, Understanding, and Interacting through Touch [UC Berkeley]
[UC Berkeley - 29 Apr 2021]
- 489