russtedrake PRO
Roboticist at MIT and TRI
Part 1
MIT 6.4210/2
Robotic Manipulation
Fall 2022, Lecture 11
Follow live at https://slides.com/d/4hDt7rQ/live
(or later at https://slides.com/russtedrake/fall22-lec11)
A sample annotated image from the COCO dataset
source: https://d2l.ai/chapter_computer-vision/fine-tuning.html
source: https://towardsdatascience.com/understanding-regions-with-cnn-features-r-cnn-ec69c15f8ea7
source: https://www.analyticsvidhya.com/blog/2018/07/building-mask-r-cnn-model-detecting-damage-cars-python/
* - partly due to low render quality?
https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html
https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html
"Contrastive visual representation learning"
from TRI Medium Blog Post
Goal: Testing in simulation matches testing in reality. Continual learning / improvement.
Challenge: Distribution shift / non-iid data
Why not aggregate all data in the cloud and train a centralized model?
Site A
Site B
Site C
Site D
Average number of segments per induct
Sites
Distributed training on the primary objective (e.g. classification / segmentation) is subject to over-fitting and shows limited robustness to distribution shift.
Distributed training on a surrogate self-supervised objective (e.g. SimCLR, SimSiam) reduces overfitting and shows superior generalization across distributions.
Data: We created distribution shift datasets grouped by clustering labels, images, or features.
Algorithm 1: Supervised Learning (SL): Trains classification or segmentation objective directly.
Algorithm 2: Self-Supervised Learning (SSL): Train common visual representation, then only "fine-tune" a small "head" on the supervised data.
\(N\) robots.
\(p_k\) is weight for robot \(k\)
\(\ell(x)\) is the loss function
\(n_k\) training samples at \(k\)
distributed client update: (\(E\) steps with random samples \(\xi\))
server update: (after responses from \(K\) clients)
E (number of decentralized steps)
5
10
25
50
100
125
250
500
Classification success rate
quick experiments using CLIP "out of the box" by Kevin Zakka
By russtedrake
MIT Robotic Manipulation Fall 2022 http://manipulation.csail.mit.edu