Principles of Sim-and-Real Cotraining For Robot Manipulation
Apr 10, 2025
Adam Wei
Joint work with Abhinav Agarwal, Boyuan Chen, Rohan Bosworth, Nicholas Pfaff, Russ Tedrake
Robot Data Diet
Big data
Big transfer gap
Small data
No transfer gap
Ego-Exo
robot teleop
Open-X
simulation
How can we obtain data for imitation learning?
(ex. sim & real)
Sim-and-Real Cotraining
\red{\mathcal{D}_{R}\sim p_{R}(O,A)}
\blue{\mathcal{D}_S\sim p_{S}(O,A)}
Cotraining: Use both datasets to train a model that maximizes some real-world performance objective
Text
Experimental Setup
Cotraining: Use both datasets to train a model that maximizes some real-world performance objective
Objective:
Success rate on planar pushing from pixels
- Focusing on single task enables controlled experiments and thorough analysis
- more on this later...
Experimental Setup
Cotraining: Use both datasets to train a model that maximizes some real-world performance objective
Datasets:
Model:
Diffusion Policy [2]
\mathcal L_{\mathcal D^\alpha} = \alpha \textcolor{red}{\mathcal L_{\mathcal D_R}} + (1-\alpha) \textcolor{blue}{\mathcal L_{\mathcal D_S}}
[1] Graedsal et. al, "Towards Tight Convex Relaxations For Contact-Rich Manipulation"
[2] Chi et. al "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion"
Real Data
Sim Data
Objective:
Success rate on planar pushing from pixels
Does Cotraining Improve Performance?
50 real demos
50 real demos
2000 sim demos
Success rate: 10/20
Success rate: 18/20
1.8x improvement!
Does Cotraining Improve Performance?
10 real demos
10 real demos
2000 sim demos
Success rate: 2/20
Success rate: 14/20
7x improvement!
Real World Cotraining Results
- Cotraining improves policy performance by up to 2-7x
- Scaling sim data improves performance and reduces sensitivity to \(\alpha\)
- What happens when we continue scaling sim data?
\mathcal L_{\mathcal D^\alpha} = \alpha \textcolor{red}{\mathcal L_{\mathcal D_R}} + (1-\alpha) \textcolor{blue}{\mathcal L_{\mathcal D_S}}
Research Questions
1. Does sim-and-real cotraining improve performance?
2. How does performance scale with data? How does \(\alpha\) affect performance?
3. What qualities matter for synthetic data?
4. What are some underlying mechanisms in cotraining?
Ex: Visual gap
Sim Demo
Target Sim Demo
Scaling Sim Data
- Performance gains from scaling sim data plateau; additional real data raises the performance ceiling
- Sim is valuable! But cannot fully replace real data
Distribution Shift Experiments
Should I be investing in my physics engine or my renderer?
Increasing color shift
Target Color
How do different sim2real gaps affect cotraining?
Example: Analyzing color shift
Experiment: Vary color shift and analyze the downstream policies
Distribution Shift Experiments
We investigate 6 sim2real gaps:
- Visual shifts: color shift, color randomization, camera shift
- Physical shifts: center of mass shift
- Task shifts: goal shifts, object shifts
Key Findings
- All shifts reduce performance; physics and task shift are most impactful
- Paradoxically, some visual shift is required for good performance!
(for planar pushing...)
Research Questions
1. Does sim-and-real cotraining improve performance?
2. How does performance scale with data? How does \(\alpha\) affect performance?
3. What qualities matter for synthetic data?
4. What are some underlying mechanisms in cotraining?
SDE Interpretation
Sim-and-Real Discernability
Real-World Demo
Cotrained Policy (50 real, 2000 sim)
Simulated Demo
- Fix orientation, then translation
- Sticking & sliding contacts
- Similar to real-world demo
- Fix orientation and translation simultaneously
- Sticking contacts only
2x
2x
2x
SDE Interpretation
Sim-and-Real Discernability
High-performing policies must learn to identify sim vs real
since the physics of each environment requires different actions
SDE Interpretation
Positive Transfer: Power Law
\(\implies\)
Sim demo worth ~0.49-0.83 real demos
Scaling sim reduces test loss & MSE in real!
SDE Interpretation
Positive Transfer: Power Laws
SDE Interpretation
Are results limited to single-task?
Zhenyu Jiang
- Concurrent & independent work shows similar results!
- Strong signal that our analysis extends to the multi-task setting
Thank You!
Lujie Yang
Nicholas Pfaff
Scalable Real2sim
Physics-Driven Data Generation
IROS 2025 Presentation Template
By weiadam
IROS 2025 Presentation Template
Slides for my talk at the Amazon CoRo Symposium 2025. For more details, please see the paper: https://arxiv.org/abs/2503.22634
