It Could Be Worse: Lessons Learned In Distributed Systems Disasters
About Me
John Wittrock
Senior Software Engineer, Real Time Platform
@johnwittrock
jwittrock@appnexus.com
A Brief History of Time
Where are we, architecturally?
How did we get here?
...Do we like it here?
the work of many, many people
web page / SSP
Impbus
CM
Data
Providers
RTPP
Ext.
Bidders
AN
Bidder
Batches
Packrat
1
2
3
4
5
6
7
~120ms
Scale
It's all in C.
Core concepts
Data distribution reliability
Traffic availability
Performance across networks
C strugglez
Workload
2009: Multiprocess apps, all using the same external port.
- no crosstalk between machines of the same application
- ...had some memory issues
Batches
Worker events
Batches
Worker events
Po
r
t
80
Workload
2013: Multithreaded, single writer
Batches / Control
Worker events
Po
r
t
80
Worker events
Process
Workload
2014: Multithreaded, single writer
Workload
2014: Multithreaded, single writer
Thread 12 (Thread 0x7ffff5ed0a50 (LWP 17893)):
#0 0x00000033126306f7 in kill () from /lib64/libc.so.6
#1 0x00007ffff6d5fa62 in mmap () from /usr/local/lib64/libgimli.so.0
#2 0x00007ffff6d5fa89 in mmap () from /usr/local/lib64/libgimli.so.0
#3 <signal handler called>
#4 0x0000000000474120 in find_index (tree=0x7ffcc43ffa60, key=0x7fffffffe648) at ../common/btree.c:77
#5 0x0000000000474907 in btree_delete (tree=0x7ffcc43ffa60, key=0x7fffffffe648) at ../common/btree.c:152
#6 0x00000000004783ea in remove_int_from_set (set=<value optimized out>, val=69) at ../common/int_set.c:131
#7 0x00000000005332ed in segment_bidder_access_remove (s=0x7fff961637b0, bidder_id=<value optimized out>) at ../common/cache/segment.c:244
#8 0x00007fffe3f9b43a in receive (src=<value optimized out>, buf=<value optimized out>, len=<value optimized out>, cookie=<value optimized out>) at member_data_access_segment.c:22
#9 _member_data_access_segment_parse (src=<value optimized out>, buf=<value optimized out>, len=<value optimized out>, cookie=<value optimized out>) at member_data_access_segment.c:10
#10 0x00000000004583fe in batch_process_object (src=0x7ffff56d1480, buffer=<value optimized out>, len=7206) at ../common/batches.c:1595
#11 batch_process (src=0x7ffff56d1480, buffer=<value optimized out>, len=7206) at ../common/batches.c:1795
#12 0x00000000004585d2 in batch_receive_chunk (req=0x7ffd950723e0, arg=0x7ffff56d1480) at ../common/batches.c:2048
#13 0x0000000000470386 in dp_read (fd=<value optimized out>, what=<value optimized out>, arg=0x7ffdcddf1800) at ../common/dnspool.c:1470
#14 0x00007ffff6b4a757 in event_process_active_single_queue (base=0x7ffff5448604, activeq=0x7ffff558d1b4, max_to_process=<value optimized out>, endtime=<value optimized out>) at event.c:1526
#15 0x00007ffff6b4af87 in event_process_active (base=0x7ffff5448604, flags=<value optimized out>) at event.c:1610
#16 event_base_loop (base=0x7ffff5448604, flags=<value optimized out>) at event.c:1833
#17 0x00000000005c2cd5 in main (argc=<value optimized out>, argv=<value optimized out>) at impbus_main.c:198Workload
2014: Multithreaded, single writer
Workload
Validation Engines
On the batches boxes, run a copy of the prod app, verify that it's okay before you distribute wider.
VE
:)
Prod Apps
:)
:)
Workload
2016: Everything is great! Until you screw up.
Benefits and drawbacks to single-writer:
- data structures
- simple threading models
Really a decision made by necessity, but turned out well.
Workload
2016: Great! Until you screw up.
Two types of crashes:
- data distribution
- traffic
Workload
Data Distribution
- This low-frequency write workload allows us to be single writer
- But that single writer can bring down the world
- When that happens, how do we turn back time?
- max_ts - prevent distribution past a timestamp
Traffic
Traffic
Bad traffic or bad code can machine-gun our boxes to death. Quickly.
Load balancers make the crashes fair, I guess....
Traffic
We can't turn back time like with max_ts
But how can we limit traffic crashes? Servers grouped by collection.
Can we turn off traffic quickly?
Feature switches and AN_HOOK
Traffic
core_me_maybe
Replace crashing with leaks!
But what happens when your system isn't actively destroying itself?
A distributed system is only as good as its performance in failure modes, because a distributed system is always failing.
With 6000 nodes, something is literally always broken.
We build our code to expect failure as normal operation.
Load Conditions
We actually care more about consistency than peak performance.
Some Things Only Happen Under Load
By Jan Kameníček - Own work, CC BY-SA 3.0 https://commons.wikimedia.org/w/index.php?curid=32543484
Overload Conditions
What happens when you've just slightly run out of capacity?
Overload Conditions
Network flow plays a huge role here - it's hard to detect transient perf degradation on a single node.
Network Flow
Router
Shard 1
p = 99.9%
Losing a single shard dependency can bring down the whole shard.
p = 60%
Backing
provider
This gets worse and worse as you add more layers, especially as you add queues.
Network Flow
Router
Required
Source 1
10ms
5ms
Backing
provider
Required
Source 2
65ms
55ms
Backing
provider
If you require sources to respond before you act, you'll start to use your entire timeout period - errors amplify themselves.
total: 75ms
:(
Network Flow
In this case, a single node's bad performance can bring down the performance of the whole cluster, and cause cascading issues.
Answer? Might be more sharding.
Network Flow
Sources of Jitter
Reads vs. writes
Expensive members
Network Flow
Okay, how the hell do you understand behavior like this?
Samples per quantile: 3047
Quantile 0: 1.000000 to 1.000000
Quantile 1: 1.000000 to 1.000000
Quantile 2: 1.000000 to 1.000000
Quantile 3: 1.000000 to 1.000000
Quantile 4: 1.000000 to 1.000000
Quantile 5: 1.000000 to 2.000000
Quantile 6: 2.000000 to 7.000000
Quantile 7: 7.000000 to 18.000000
Quantile 8: 18.000000 to 54.000000
Quantile 9: 54.000000 to 155.000000
Quantile 10: 155.000000 to 372.000000
Quantile 11: 372.000000 to 745.000000
Quantile 12: 745.000000 to 1282.000000
Quantile 13: 1282.000000 to 2024.000000
Quantile 14: 2025.000000 to 2871.000000
Quantile 15: 2871.000000 to 3781.000000
Quantile 16: 3781.000000 to 4674.000000
Quantile 17: 4675.000000 to 5702.000000
Quantile 18: 5703.000000 to 6945.000000
Quantile 19: 6945.000000 to 14510.000000C Strugglez
Paul:"buffer[i] = (void*)((uintptr_t)buffer[i] | (1UL << 34));" is a code crime.
Maxime Henrion: >_<
Andrew Wong: (void*) makes it work anywhere
Maxime Henrion: it's magic!
Maxime Henrion: doesn't compile? add more void * casts.
Andrew Wong: that or switch the */& logic everywhere
Andrew Wong: All the combinations!
C Strugglez
The usual:
- Leaks
- Corruption
- Lack of a standard library
C Strugglez
We can fix the leaks! (by understanding our workload)
Corruption is harder, but good tooling is the most important.
- the 34th bit
C Strugglez
Paul: The tale of the 34th bit is closed.
We CAS "snapshot & ~(1ULL << current->id)" over deallocated campaigns.
Mike Wright: ???
C Strugglez
Event 1:
- Frees memory, but not timer
- Timer fires,
clears waiting bit of current thread
- Happens to be 34th bit
Event 2:
- Uses the same memory region for pointer
- Dereferences pointer...
C Strugglez
Paul: budget wakeup for long-deallocated campaigns.
Paul: an_array buffer takes its place.
we basically spinlock on in_pacing_sleep by clearing out the thread's bit.
mcen: you guys should be popping bottles in the office
The Lessons
0. Understand your workload
1. Partition your workload and specialize
2. Keep network flow graphs flat
3. Use every possible feature of your environment to your advantage
4. Find a way to replicate prod, even if it means testing in prod
Questions?
itcouldbeworse
By jwittrock