Towards In-hand Manipulation from Vision and Touch
Roberto Calandra
Facebook AI Research
ICRA 2020 ViTac workshop - 31 May 2020
The Importance of Touch (in Humans)
From the lab of Dr. Ronald Johansson, Dept. of Physiology, University of Umea, Sweden
The Importance of Touch (in Humans)
The Importance of Touch (in Robots)






Challenges
Difficult to integrate/process touch in traditional analytical control scheme
Lack of good available tactile sensors
Tactile Sensors in Robotics
Important factors:
- Availability
- Cost
- Form factor
- Capabilities (e.g., resolution)
- Reliability

Many many sensors in the literature:
- Most are prototypes
- Just a few are commercially available
or can be manufactured easily


[Wilson et al., 2019]
[Piacenza et al., 2020]
[ Fischel et al., 2012]

[Zhang et al., 2018]

[Church et al., 2019]
Examples of DIGIT Measurements

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Comparison
BioTac
DIGIT

~10,000 $
Cost
~15 $
Resolution
29
contact points
307,200
contact points
Mounted on multi-finger hands


Open-sourced



https://digit.ml
Elastomer Robustness

Design

Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845
Replaceable Elastomer




Reflective
Reflective
+
Markers
Transparent
+
Markers
DIGIT for manipulation
- Problem: Integrating tactile sensors in the control scheme is challenging, especially for rich tactile measurements.
DIGITs mounted on a Wonik's Allegro hand
- Task: Demonstrate fine manipulation skills from touch.
- Approach: Integrating rich tactile sensors through .
machine learning

Learning to Manipulate a Marble
Model-based Reinforcement Learning
Marble Manipulation Results


Marble Manipulation Trajectories



Human Collaborators
Lambeta, M.; Chou, P.-W.; Tian, S.; Yang, B.; Maloon, B.; Most, V. R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; Jayaraman, D. & Calandra, R.
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
IEEE Robotics and Automation Letters (RA-L), 2020, 5, 3838-3845











OmniTact

Padmanabha, A.; Ebert, F.; Tian, S.; Calandra, R.; Finn, C. & Levine, S.
OmniTact: A Multi-Directional High-Resolution Touch Sensor
IEEE International Conference on Robotics and Automation (ICRA), 2020

Summary
Grand Goal:
Towards Human-level Manipulation using Vision and Touch
-
Touch is a key sense for in-hand manipulation
- Introduced , the first mass-producible, low-cost, high-resolution, compact tactile sensor.
- Demonstrated robot marble manipulation using DIGIT
- Open-source at
DIGIT
https://digit.ml
DIGIT: @ICRA on Wednesday - Slack: #wec01_6
OmniTact: @ICRA on Monday - Slack: #moa15_3
References
- Yuan, W.; Dong, S. & Adelson, E. H.
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force
Sensors, 2017 - Calandra, R.; Owens, A.; Jayaraman, D.; Yuan, W.; Lin, J.; Malik, J.; Adelson, E. H. & Levine, S.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
IEEE Robotics and Automation Letters (RA-L), 2018, 3, 3300-3307 - Allen, P. K.; Miller, A. T.; Oh, P. Y. & Leibowitz, B. S.
Integration of vision, force and tactile sensing for grasping
Int. J. Intelligent Machines, 1999, 4, 129-149 - Chebotar, Y.; Hausman, K.; Su, Z.; Sukhatme, G. S. & Schaal, S.
Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning
International Conference on Intelligent Robots and Systems (IROS), 2016 - Schill, J.; Laaksonen, J.; Przybylski, M.; Kyrki, V.; Asfour, T. & Dillmann, R.
Learning continuous grasp stability for a humanoid robot hand based on tactile sensing
BioRob, 2012 - Bekiroglu, Y.; Laaksonen, J.; Jorgensen, J. A.; Kyrki, V. & Kragic, D.
Assessing grasp stability based on learning and haptic data
Transactions on Robotics, 2011, 27 - Sommer, N. & Billard, A.
Multi-contact haptic exploration and grasping with tactile sensors
Robotics and autonomous systems, 2016, 85, 48-61
Towards In-hand Manipulation From Touch
By Roberto Calandra
Towards In-hand Manipulation From Touch
[ICRA WS ViTaC - 31 May 2020]
- 1,206