Kaggle 101: Why to participate, how to participate and how to win Deep Learning Kaggle competitions
Data Scientist at Lyft
PhD in Theoretical Physics
Kaggle GrandMaster (31st out of 76000+)
Salaries in Industry vs Academia
- L3 (Junior): Base $105-145k + Stocks $45-110k + bonus
- L4 (Middle): Base: $145-170k + Stocks $80-162k + bonus
- L5 (Senior): Base $170-200k + Stocks $115-250k + bonus
- Grad student: $25k
- PostDoc $50k
You need to get results.
Deep Learning is alchemy.
- Most of the brilliant ideas do not work.
- Some stupid ideas work really well.
Q: What can Kaggle teach?
A: How to get good results fast. (Research Engineer)
Q: But I know how to iterate fast?
A: Yeah, sure.
Fast Iterations in Deep Learning
- PyTorch / Keras
- Jupyter notebook
- numpy, scipy, etc
Human Grid Search (1000+ teams)
- Share what works
- Share what does not work
- Share relevant literature (papers / blog posts / videos)
- Share code
Many DL tricks published in papers were used long ago at competitions.
- 2-4+ GTX 1080 Ti
- CPU with 4-6+ cores
- Fast SSD
- 32-64GB RAM
- 4Tb+ HDD
Top competitors have 2 machines:
- Prototyping (1-2 GPUs)
- Heavy lifting (4+ GPUs)
My home DevBox
Crypto mining gives $1000-2000 per month :)
First step: appear on the LeaderBoard
Kindergarten stage (2-4 hours)
- Downloaded the data.
- Did exploratory analysis.
- Wrote some code.
Adult stage (1-10 weeks)
- Wrote pipeline.
- If you can write pipeline in 2-4 hours => do it
- If not => use someone's code (even if you do not understand)
split into folds
- Code for Data preparation and model training => 2-4 hours
- While training write code for local validation and prediction.
- Local validation and Public LB should correlate.
Example: Carvana Challenge
Semantic Segmantation => FCNs (UNet, SegNet, FCN8-32, PSPNet, etc)
Adapted old UNet pipeline(~40 min)
Trained for 20 epochs => Intersection Over Union 0.992
=> Yeah, sure.
IoU = 0.992 (435 out of 735)
- More augmentations => YES
- Another optimizer (Adam to SGD or RMSProp) => NO
- More epochs => NO
- Even more augmentations => NO
- Different loss => NO
- Bigger batch size / smaller crops => NO
- Smaller batch size / full HD images => YES
IoU = 0.996 =>
=> yeah, sure (294 out of 735)
- Weighted on boundary loss => NO
- Adding fixed masks (was shared at the forum) => NO
- Even more epochs => NO
- PSPNet => NO
- Replacing UNet encoder with VGG encoder (TernausNet) => YES
IoU = 0.9970 =>
=> yeah, sure (21 out of 735)
TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
- Bagging => YES
- Cyclic LR schedule => YES
- More augmentations => YES
IoU = 0.9972 =>
=> Finally (5 out of 735)
Merged into a team
IoU = 0.9973 => 1st out of 735
In the academia
Competitions for conferences can often be won with Kaggle baselines:
Approaches developed in competitions can be published
- V Iglovikov, A Rakhlin, A Kalinin, A Shvets, Pediatric Bone Age Assessment Using Deep Convolutional Neural Networks, arXiv:1712.05053
- V Iglovikov, A Shvets, TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation, arXiv:1801.05746
- A Rakhlin, V Iglovikov, A Shvets, A Kalinin, Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis , arXiv: 1802: 00572
- Real-time Leaderboard.
- Large, willing to share their knowledge, community of DL partitioners.
What you get within few months at Kaggle:
- Pipelines for many different problems.
- Research Engineer skillset.
- Deep Learning practitioner skillet.
- Ability to iterate extremely fast.
- Strong line in your resume.
- Prize money (I got $17,000 last year)
By Vladimir Iglovikov