Everything I learned about Spark, I learned while eating candy.
Can you grab the bag...
...open the bag...
...and eat all the brown ones...
...in only 30 seconds?
All the candies in 30 seconds?
Where do we start?
Scenario #1
A
B
C
D
E
F
G
H
I
J
K
L
1 2 3 4 5 6 7 8 9...
Let's estimate how much data we have...
And assess who is free to pitch in and help...
Student A gets bag #1... Student B gets bag #2...
Student C gets bag #3...
Student D...
A
B
C
D
E
F
G
H
I
J
K
L
A
B
C
D
E
F
G
H
I
J
K
L
A
B
C
D
E
F
G
H
I
J
K
L
Eliminate the brown pieces of candy...
Pile the rest in the corner
A
B
C
D
E
F
G
H
I
J
K
L
A
B
C
D
E
F
G
H
I
J
K
L
B
C
D
F
G
I
K
L
Student J gets bag #13... Student E gets bag #14...
Student H gets bag #15...
Student J...
A
E
H
J
A
B
C
D
E
F
G
H
I
J
K
L
A
E
H
J
C
B
D
F
G
I
K
L
A
E
H
J
B
C
D
F
G
I
K
L
A
E
H
J
B
C
D
F
G
I
K
L
Â
   All done!
Â
A
B
C
D
E
F
G
H
I
J
K
L
Scenario #2
Let's try another!
Â
This time, count `em all!
A
B
C
D
E
F
G
H
I
J
K
L
How do we do this?
A
B
C
D
E
F
G
H
I
J
K
L
Stage #1
Local Count
Student A gets bag #1... Student D gets bag #2...
Student I gets bag #3...
Student J gets bag #4...
Stage #1: Local Count
A
B
C
D
E
F
G
H
I
J
K
L
A
D
I
J
Stage #1: Local Count
B
C
E
F
G
H
K
L
A
D
I
J
Stage #1: Local Count
B
C
E
F
G
H
K
L
A
D
I
J
Stage #1: Local Count
B
C
E
F
G
H
K
L
J
5
4
5
Stage #1: Local Count
B
C
E
F
G
H
K
L
A
D
I
5
4
5
Student J is hung...
I cannot continue !!!
What do I do now?
Stage #1: Local Count
J
B
C
E
F
G
H
K
L
A
D
I
5
4
5
Student K, count bag #4!
Stage #1: Local Count
J
B
C
E
F
G
H
K
L
A
D
I
5
4
5
I'll settle for the result of student J or student K...
Whomever finishes first.
?
Stage #1: Local Count
J
B
C
E
F
G
H
K
L
A
D
I
5
4
5
6
Stage #1: Local Count
Cancel student J...
Â
...and take K's result
!
J
B
C
E
F
G
H
K
L
A
D
I
5
4
5
6
Stage #1: Local Count
Prepare for the shuffle!
Â
(A, D & I did earlier)
J
B
C
E
F
G
H
K
L
5
4
5
6
A
D
I
Stage #1: Local Count
OK, Stage #1 is all done!
J
B
C
E
F
G
H
K
L
5
4
5
6
A
D
I
Stage #2
Global Count
F
Stage #2: Global Count
Student F, fetch the counts from A, D, I and K
5
4
5
6
A
B
C
D
E
G
H
I
J
K
L
F
5
4
5
6
Stage #2: Global Count
5
4
5
6
A
B
C
D
E
G
H
I
J
K
L
F
20
Stage #2: Global Count
5
4
5
6
A
B
C
D
E
G
H
I
J
K
L
F
Stage #2: Global Count
5
4
5
6
20
20
A
B
C
D
E
G
H
I
J
K
L
F
20
Stage #2: Global Count
5
4
5
6
A
B
C
D
E
G
H
I
J
K
L
Thanks!
Scenario #3
Sort by color name:
Blue, Brown, Green, Orange, Red, Yellow
Stage #1: Local Sort
F
A
B
C
D
E
G
H
I
J
K
L
...fast forward to the end of Stage #1...
Stage #1: Local Sort
F
A
B
C
D
E
G
H
I
J
K
L
Prepare for the shuffle!
Stage #1: Local Sort
F
A
B
C
D
E
G
H
I
J
K
L
OK, Stage #1 is all done!
Stage #2
Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Student E, get the reds
Stage #2: Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Student K, get the blues
Stage #2: Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Student C... orange
Student J... green
Student H... brown
Student B... yellow
Stage #2: Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Collect all the results back to the driver...
Stage #2: Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Stage #2: Global Sort
F
A
B
C
D
E
G
H
I
J
K
L
Stage #2: Global Sort
Thanks!
5
4
5
6
5
4
5
6
Read data from there, do A, B, C & D to it, when you are all done, write it out over there
5
4
5
6
20
20
Spark Architecture
By Jacob D Parr
