Distributed Systems

Neven Miculinic

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable

 

- Leslie Lamport

 

A distributed system is a collection of autonomous computing elements that appears to its users as a single coherent system.

Some attributes

  • It's consistent of 2 or more nodes
  • nodes can act independently from each other
  • Communication using message passing
  • Frequently nodes are geographically distributed
  • There's no global clock
  • Nodes can fail at any given time
  • The network can partition -> Split brain

Some attributes

A brief introduction to distributed systems:

https://link.springer.com/article/10.1007/s00607-016-0508-7

Meetup format

  • Whatever you want
  • Recommend multiple 25-maxish 45 min lectures
    • Split into multiple days/meetups is fine
  • If you wanna organize workshop that's fine

What it's about

  • Theory -- e.g. Raft/Paxos consensus,  consistency levels, two-phase commit protocol, ...
  • Case studies -- e.g. 
    • https://research.fb.com/publications/scuba-diving-into-data-at-facebook/
    • Cassandra architecture
    • Kubernetes arhitectures
  • Tools -- e.g. Prometheus, grafana, Jaeger, Envoy, ... 
  • Tips & trick -- e.g. I used vagrant & netem to simulate a poor network connection
  • War stories -- e.g. Horrible race condition bug I've encountered last month and what I've learnt from it

Related meetups

  • Zagreb DevOps Meetup (infobip), 4x this year
  • Kraken Kubernetes and Cloud Native Computing Croatia (KrakenSystems), no meetups so far

Example

netcat

nc -lu localhost 2345
nc -u localhost 2345

Example

single process concurrency

Each package is a number

Send

// Error handling omitted
conn, _ := net.ListenPacket("udp", ":0")
defer conn.Close()
dst, _ := net.ResolveUDPAddr("udp", addr)
for i := 0; ; i++ {
    conn.WriteTo([]byte(strconv.Itoa(i)+"\n"), dst)
    time.Sleep(*writeInterval)
}

Recv

conn, _ := net.ListenUDP("udp", addr)
defer conn.Close()
numbers := make(chan int64)
go readNumberdFromUDP(conn, numbers)

t := time.NewTicker(*summaryInterval)

Recv

func readNumberdFromUDP(
    conn *net.UDPConn, 
    numbers chan<- int64,
) {
	buf := make([]byte, 1024)
	for {
		n, _, _ := conn.ReadFromUDP(buf)
                x := ... // parse it
		numbers <- x
	}
}

Recv

for {
    select {
    case <-t.C:
        // Write summary of msgs recieved
    case x := <-numbers:
        // Add number x to the summaries
    }
}

Example

Procfile

Procfile

recv: go run *.go recv
send: go run *.go send

goreman start

Example

Vagrant

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "debian/jessie64"
  config.vm.network "private_network", type: "dhcp"

  config.vm.define "host1"
  config.vm.define "host2"
end

Vagrantfile

vagrant up 
vagrant ssh host1
vagrant ssh host2

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "debian/jessie64"
  config.vm.network "private_network", type: "dhcp"

  # Provisioning
  config.vm.provision :ansible do |ansible|
    ansible.playbook = "playbook.yml"
    ansible.compatibility_mode = "2.0"
  end

  config.vm.define "host1"
  config.vm.define "host2"
end

Ansible

playbook.yml

---
- hosts: all
  become: yes
  tasks:
    - name: Install list of packages
      apt:
        name: "{{ packages }}"
        state: present
        update_cache: yes
      vars:
        packages:
            - curl
            - iproute2
            - python3
            - netcat
            - python3-pip

Ansible

playbook.yml

    - pip:
        name: pip
        state: latest
        executable: pip3
    - pip:
        name: tcconfig
        state: latest
        executable: pip3
    - copy: 
        src: "dist-example"
        dest: "/usr/bin/dist-example"
        mode: 0755

Example

tcconfig

tcconfig

A tc command wrapper. Easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).

 

https://tcconfig.readthedocs.io

tcconfig

NetEm is an enhancement of the Linux traffic control facilities that allow to add delay, packet loss, duplication and more other characteristics to packets outgoing from a selected network interface. NetEm is built using the existing Quality Of Service (QOS) and Differentiated Services (diffserv) facilities in the Linux kernel.

man 8 tc-netem

tcconfig

tcset eth0 --rate 100Kbps \
    --delay 500ms \
    --loss 10 \
    --reordering 20
tcshow eth0
tcdel eth0

tcconfig

ansible -i 
(...)    .vagrant/provisioners/ansible/
(...)inventory/vagrant_ansible_inventory 
(...) all -b -a "ip a"

ansible trick

Q&A

https://slides.com/nmiculinic/distributed-systems

https://github.com/nmiculinic/distributed-systems-01

Made with Slides.com