Distributed Systems Expectation vs Reality
Alan Braithwaite
@caust1c
https://abraithwaite.net
Segment Inc
Who Am I?
@caust1c
What are Distributed Systems?
@caust1c
![](https://media.slid.es/uploads/813781/images/7718119/TALL_-_White_on_Transparent.png)
![](https://media.slid.es/uploads/813781/images/7718137/spark-white.png)
![](https://media.slid.es/uploads/813781/images/7718153/1200px-Apache_ZooKeeper_Logo.svg.png)
![](https://media.slid.es/uploads/813781/images/7718180/1200px-BitTorrent_logo.svg.png)
![](https://media.slid.es/uploads/813781/images/7718191/pngfind.com-pluralsight-logo-png-4870857.png)
![](https://media.slid.es/uploads/813781/images/7718185/etcd-horizontal-white.png)
![](https://media.slid.es/uploads/813781/images/7718179/cropped-foldingathome-logo.png)
![](https://media.slid.es/uploads/813781/images/7718182/0_23823574869487363.png)
![](https://media.slid.es/uploads/813781/images/7718192/kisspng-domain-name-system-microsoft-azure-name-server-ama-name-5aec23a444cbc7.1601951315254250602818.png)
![](https://media.slid.es/uploads/813781/images/7718484/clickhouse.png)
![](https://media.slid.es/uploads/813781/images/7718241/iphone-410324_1920.jpg)
Image by Jan Vašek
![](https://media.slid.es/uploads/813781/images/7718282/PinClipart.com_chalk-clipart_25712.png)
?
?
![](https://media.slid.es/uploads/813781/images/7718302/toaster2.jpg)
![](https://media.slid.es/uploads/813781/images/7718311/bsgtoast-removebg-preview.png)
![](https://media.slid.es/uploads/813781/images/7718439/cylon.jpg)
![](https://media.slid.es/uploads/813781/images/7718119/TALL_-_White_on_Transparent.png)
![](https://media.slid.es/uploads/813781/images/7718137/spark-white.png)
![](https://media.slid.es/uploads/813781/images/7718153/1200px-Apache_ZooKeeper_Logo.svg.png)
![](https://media.slid.es/uploads/813781/images/7718191/pngfind.com-pluralsight-logo-png-4870857.png)
![](https://media.slid.es/uploads/813781/images/7718185/etcd-horizontal-white.png)
![](https://media.slid.es/uploads/813781/images/7718182/0_23823574869487363.png)
![](https://media.slid.es/uploads/813781/images/7718192/kisspng-domain-name-system-microsoft-azure-name-server-ama-name-5aec23a444cbc7.1601951315254250602818.png)
Distributed Systems We Use
![](https://media.slid.es/uploads/813781/images/7718484/clickhouse.png)
Distributed Systems
Crash Course
@caust1c
Concepts
@caust1c
High Availability
Consistency
Scalability
High Availability
@caust1c
![](https://media.slid.es/uploads/813781/images/7718922/DB-1.png)
High Availability
@caust1c
![](https://media.slid.es/uploads/813781/images/7718923/DB-2.png)
High Availability
@caust1c
![](https://media.slid.es/uploads/813781/images/7718921/DB-2_Fail.png)
High Availability
@caust1c
![](https://media.slid.es/uploads/813781/images/7718920/DB-3.png)
Consistency
@caust1c
- How do we handle hosts coming back online?
- Where do we write data to?
- Multiple Connecting Clients?
Consistency
@caust1c
- How do we handle hosts coming back online?
- Where do we write data to?
- Multiple Connecting Clients?
Single Writer Principle
Consistency
@caust1c
- How do we handle hosts coming back online?
- Where do we write data to?
- Multiple Connecting Clients?
Single Writer Principle
Autonomous Leader Elections
Sharding
Sharding
![](https://media.slid.es/uploads/813781/images/7719130/DB-Distrib.png)
Sharding
- Scale Out instead of Up
- Data Management & Shuffling
- Single Writer Principle within each shard
- Autonomous Leader Selection
- zookeeper & etcd most common
Linearizability
@caust1c
https://jepsen.io/consistency/models/linearizable
Every operation on an object appears to take place atomically, in some order, consistent with the real-time ordering of those operations.
A Note About CAP Theorem
@caust1c
A Note About CAP Theorem
@caust1c
- Consistency
- Availability
- Partition tolerance
A Note About CAP Theorem
@caust1c
- Consistency
- Availability
Partition tolerance
Expectation vs Reality
@caust1c
Expectation
![](https://media.slid.es/uploads/813781/images/6055713/7365392.jpg)
Reality
![](https://media.slid.es/uploads/813781/images/7719096/05onfire1_xp-videoSixteenByNineJumbo1600-v2.jpg)
Reality
@caust1c
- HA is standard now
Reality
@caust1c
- HA is standard now
- Operations Overhead
Reality
@caust1c
- HA is standard now
- Operations Overhead
- Pushes Complexity Clientside
Reality
@caust1c
- HA is standard now
- Operations Overhead
- Pushes Complexity Clientside
- Hot Shard Problem
Reality
@caust1c
- HA is standard now
- Operations Overhead
- Pushes Complexity Clientside
- Hot Shard Problem
- Late Arriving Data Problem
Reality
@caust1c
- HA is standard now
- Operations Overhead
- Pushes Complexity Clientside
- Hot Shard Problem
- Late Arriving Data Problem
- Disk Failures
Reality
@caust1c
![](https://media.slid.es/uploads/813781/images/7719107/night.jpg)
Reality
@caust1c
![](https://media.slid.es/uploads/813781/images/7719116/Screenshot_from_2020-09-11_13-53-24.png)
Advice
@caust1c
General Advice
@caust1c
- Learn Networking
- TCP, DNS, HTTP and basic routing at a minimum
- Dr. Kredo's advanced networking is great
- Beej's Guide to Network Programming (another alumni!)
- Reboot CSLUG! Chico State Linux User's Group
- Install Linux on Everything
- Homelab!
- reddit.com/r/homelab
General Advice
@caust1c
![](https://media.slid.es/uploads/813781/images/7719061/homelab.jpg)
https://www.reddit.com/r/homelab/comments/8vm9vm/raspberry_pi_datacenter/
Doesn't take a lot of money or a fancy lab
(although those things do help)
Job Advice
@caust1c
- Everything on the previous slide
- Attend (virtual) meetups, talk to people
- Read Hacker News (but not the comments)
- HN Who's Hiring Threads (1st Monday Monthly)
- Read tech blogs & papers
- https://jepsen.io/
- Papers We Love
- Open Source not necessary but definitely does help!
![](https://imgs.xkcd.com/comics/wisdom_of_the_ancients.png)
https://xkcd.com/979/
Distributed Systems Expectation vs Reality
Alan Braithwaite
@caust1c
https://abraithwaite.net
Segment Inc
Distributed Systems Expectation Vs Reality
By Alan Braithwaite
Distributed Systems Expectation Vs Reality
- 899