Kaggle 101: Why to participate, how to participate and how to win Deep Learning Kaggle competitions

Vladimir Iglovikov

Data Scientist at Lyft

PhD in Theoretical Physics

Kaggle GrandMaster (31st out of 76000+)

Salaries in Industry vs Academia

https://www.teamblind.com/article/Stop-asking-for-tech-company-offer-numbers-Read-this-and-thank-me-later-sN8uYpop

• L3 (Junior): Base $105-145k +  Stocks $45-110k + bonus
• L4 (Middle):  Base: $145-170k + Stocks $80-162k + bonus
• L5 (Senior): Base $170-200k + Stocks $115-250k + bonus

You need to get results.

Deep Learning is alchemy.

=

Fast iterations.

• Most of the brilliant ideas do not work.
• Some stupid ideas work really well.

Q: What can Kaggle teach?

A: How to get good results fast. (Research Engineer)

Q: But I know how to iterate fast?

A: Yeah, sure.

Fast Iterations in Deep Learning

Tools:

1. Python
2. PyTorch / Keras
3. OpenCV
4. Jupyter notebook
5. Pycharm
6. numpy, scipy, etc

Human  Grid Search (1000+ teams)

1. Share what works
2. Share what does not work
3. Share relevant literature (papers / blog posts / videos)
4. Share code

Many DL tricks published in papers were used long ago at competitions.

Hardware

• 2-4+ GTX 1080 Ti
• CPU with 4-6+ cores
• Fast SSD
• 32-64GB RAM
• 4Tb+ HDD

Top competitors have 2 machines:

1. Prototyping (1-2 GPUs)
2. Heavy lifting (4+ GPUs)

My home DevBox

Crypto mining gives $1000-2000 per month :)

First step: appear on the LeaderBoard

Kindergarten stage (2-4 hours)

• Downloaded the data.
• Did exploratory analysis.
• Wrote some code.

Adult stage (1-10 weeks)

• Wrote pipeline.

• Made submission.

• Iterations.

• If you can write pipeline in 2-4 hours => do it
• If not => use someone's code (even if you do not understand)

Pipeline

DATA:

clean 

pre-process

cache

split into folds

Train

model

Public

Leaderboard

1. Code for Data preparation and model training => 2-4 hours
2. While training write code for local validation and prediction.
3. Local validation and Public LB should correlate.

Example: Carvana Challenge

Semantic Segmantation => FCNs (UNet, SegNet, FCN8-32, PSPNet, etc)

Input

Output

Adapted old UNet pipeline(~40 min)

Trained for 20 epochs => Intersection Over Union 0.992

 IoU = 0.992 (435 out of 735)

Attempts

1. More augmentations => YES
2. Another optimizer (Adam to SGD or RMSProp) => NO
3. More epochs => NO
4. Even more augmentations => NO
5. Different loss => NO
6. Bigger batch size / smaller crops => NO
7. Smaller batch size / full HD images => YES

IoU = 0.996         => 

=>  yeah, sure  ​(294 out of 735)

Attempts

1. Weighted on boundary loss => NO
2. Adding fixed masks (was shared at the forum) => NO
3. Even more epochs => NO
4. PSPNet => NO
5. Replacing UNet encoder with VGG encoder (TernausNet) => YES

IoU = 0.9970         => 

=>  yeah, sure  ​(21 out of 735)

TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation

Attempts

1. Bagging => YES
2. Cyclic LR schedule => YES
3. More augmentations => YES

IoU = 0.9972         => 

=>  Finally  ​(5 out of 735)

IoU = 0.9973         => 1st out of 735

Full solution

In the academia

Competitions for conferences can often be won with Kaggle baselines:

1. Vladimir Iglovikov and Alexey Shvets (8 evenings):  MICCAI 2017: Gastrointestinal Image ANAlysis  => 1st place => paper

2. Vladimir Iglovikov and Alexey Shvets (4 evenings): ​MICCAI 2017: Robotic Instrument Segmentation  => 1st place => paper

Approaches developed in competitions can be published

1. V Iglovikov, A Rakhlin, A Kalinin, A Shvets, Pediatric Bone Age Assessment Using Deep Convolutional Neural Networks, arXiv:1712.05053
2. V Iglovikov, A Shvets, TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation, arXiv:1801.05746
3. A Rakhlin, V Iglovikov, A Shvets, A Kalinin, Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis , arXiv: 1802: 00572

Summary

Kaggle provides:

1. Real-time Leaderboard.
2. Large, willing to share their knowledge, community of DL partitioners.

What you get within few months at Kaggle:

1. Pipelines for many different problems.
2. Research Engineer skillset.
3. Deep Learning practitioner skillet.
4. Ability to iterate extremely fast.
5. Networking.
6. Strong line in your resume.
7. Prize money (I got $17,000 last year)

Thank you!

Kaggle: https://www.kaggle.com/iglovikov

LinkedIn: https://www.linkedin.com/in/iglovikov/

Twitter: https://twitter.com/viglovikov