Unison: a language for distributed, elastic computation

Paul Chiusano (@pchiusano)

Cofounder, Unison Computing

joint work with:

Rúnar Bjarnason (@runarorama)

Arya Irani (@aryairani)

DISCLAIMER!!!1!1! : UNDER CONSTRUCTION

Unison

  • Statically-typed, functional

  • Open source, MIT licensed

  • In R&D since 2013... now working toward spring 2019 release

WHY?

Programming = FUN!! 🦄 🌈 🎉

Goal: make every aspect of software creation delightful... or at least REASONABLE

λ> "ARI"
"ARI"
λ> replicate 6 "ARI"
["ARI","ARI","ARI","ARI","ARI","ARI"]
λ> replicate 92 "ARI"
["ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI","ARI","ARI","ARI","ARI",
"ARI","ARI","ARI","ARI"]
λ> replicate 92 "MAMA"
["MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA","MAMA",
"MAMA","MAMA","MAMA","MAMA"]

1 OS process

EVERYTHING ELSE

Docker

Kubernetes

Terraform

Kafka

DynamoDB

S3

EC2

ElasticSearch

Kibana

Prometheus

Grafana

PagerDuty

etcd

ELB

Route 53

Consul

systemd

Flannel

Weave

Lambda

App Engine

rkt

CoreOS

Zookeeper

Redis

memcached

Protobufs

Thrift

Envoy

Mesos

Nomad

ASGs

←JSON→

Haskell

Java

"Just set up an ASG connected to ELB"

Chef

Puppet

Distributed, elastic compute: the default

Cheaper/faster

Why?

Data deps of compute

Netflix, nytimes.com...

Also: people sleep, world popl not evenly distributed...

Before the internet:

1 OS process

✅ 👍

Unison

increment : Nat -> Nat
increment n = n + 1

> x = 1 + 40
> y = increment x

Unison basics

👀  Now evaluating any watch expressions (lines starting with `>`) ...

    4 | > x = 1 + 40
          ⧩
          41

    5 | > increment x
          ⧩
          42

⬇️

Unison distributed programming


doDistributedStuff lotsOfData =
  alice : Node
  alice = spawn
  job1 = at alice '(hugeComputation1 lotsOfData)
  job2 = at spawn '(hugeComputation2 lotsOfData)
  force job1 + force job2
spawn : {Remote} Node
at    : Node -> '{Remote} a ->{Remote} (Future a)
force : Future a ->{Remote} a
doDistributedStuff : Stuff -> Nat
doDistributedStuff : Stuff ->{Remote} Nat

👇

runLocal            : '{Remote} a -> a
runLocalChaosMonkey : '{Remote} a -> a
Cloud.run : Params -> '{Remote} a -> a

> runLocal '(doDistributedStuff lotsOfData)
> runLocalChaosMonkey '(doDistributedStuff lotsOfData)
> Cloud.run usEast '(doDistributedStuff lotsOfData)

👈

merge : (a -> a -> Boolean) -> [a] -> [a] -> [a]
merge lte a b =
  go out a b = case (uncons a, uncons b) of
    (None,_) -> out ++ b
    (_,None) -> out ++ a
    (Some (hA, tA), Some (hB, tB)) ->
      if lte hA hB then go (snoc hA out) tA b
      else go (snoc hB out) a tB
  go [] a b

> merge (<) [1,3,4,99,504,799] [0,19,22,23]

Ex: distributed mergesort in 17 LOC

   93 | > merge (<) [1,3,4,99,504,799] [0,19,22,23]
          ⧩
          [0, 1, 3, 4, 19, 22, 23, 99, 504, 799]

⬇️

sort : (a -> a -> Boolean) -> [a] -> [a]
sort lte s =
  if size s < 2 then s
  else case halve s of (left, right) ->
    r1 = sort lte left
    r2 = sort lte right
    merge lte r1 r2

> sort (<) [1,2,3,234,6,2,4,66,2,33,4,2,57]

Ex: distributed mergesort in 17 LOC

  112 | > sort (<) [1,2,3,234,6,2,4,66,2,33,4,2,57]
          ⧩
          [1, 2, 2, 2, 2, 3, 4, 4, 6, 33, 57, 66, 234]

⬇️

dsort : (a -> a -> Boolean) -> [a] ->{Remote} [a]
dsort lte s =
  if size s < 2 then s
  else case halve s of (left, right) ->
    r1 = at spawn '(dsort lte left)
    r2 = at spawn '(dsort lte right)
    merge lte (force r1) (force r2)

> runLocal '(dsort (<) [1,2,3,234,6,2,4,66,2,33,4,2,57])

Ex: distributed mergesort in 17 LOC

  114 | > Remote.runLocal '(dsort (<) [1,2,3,234,6,2,4,66,2,33,4,2,57])
          ⧩
          [1, 2, 2, 2, 2, 3, 4, 4, 6, 33, 57, 66, 234]

⬇️

dsort : (a -> a -> Boolean) -> [a] ->{Remote} [a]
dsort lte s =
  if size s < 2 then s
  else case halve s of (left, right) ->
    r1 = at spawn '(dsort lte left)
    r2 = at spawn '(dsort lte right)
    merge lte (force r1) (force r2)

Ex: distributed mergesort in 17 LOC

sort : (a -> a -> Boolean) -> [a] -> [a]
sort lte s =
  if size s < 2 then s
  else case halve s of (left, right) ->
    r1 = sort lte left
    r2 = sort lte right
    merge lte r1 r2

DEMO

Provisioning, deployment, and scaling are part of the programming model

Deployment

Scaling, failover...

Provisioning

doDistributedStuff lotsOfData =
  alice : Node
  alice = spawn
  job1 = at alice '(hugeComputation1 lotsOfData)
  job2 = at spawn '(hugeComputation2 lotsOfData)
  force job1 + force job2

Composability for distributed programs

dsort : (a -> a -> Boolean) -> [a] ->{Remote} [a]
dsort lte s =
  if size s < 2 then s
  else case halve s of (left, right) ->
    r1 = at spawn '(dsort lte left)
    r2 = at spawn '(dsort lte right)
    merge lte (force r1) (force r2)
dreduce : Monoid a -> [a] ->{Remote} a
dreduce m s =
  if size s < 2 then Monoid.orElse m (Sequence.head s)
  else case halve s of (left, right) ->
    r1 = at spawn '(dreduce m left)
    r2 = at spawn '(dreduce m right)
    Monoid.op m (force r1) (force r2)

dsort : (a -> a -> Boolean) -> [a] ->{Remote} [a]
dsort lte s = 
  dreduce (Monoid (merge lte) []) 
          (Sequence.map single s)
dreduce : Monoid a -> [a] ->{Remote} a







dsort : (a -> a -> Boolean) -> [a] ->{Remote} [a]
dsort lte s = 
  dreduce (Monoid (merge lte) []) 
          (Sequence.map single s)

Roadmap

meetup.com/boston-unison

Target: public release Spring 2019

A pre-alpha sooner than that

Also

🙏 Arya & Rúnar, for being such amazing coworkers 🦄 🌈

🙏 Chris Gibbs, new contributor, for work on pretty-printer + design reviews ❤️ 🎉

@unisonweb, unisonweb.org

Lots of cool work to do, impact can be HUGE

Appendix

bob = spawn
at bob '(factorial n)

... but how?

.

.

factorial n =
  foldLeft (*) 1 (range 1 (n + 1))

blah z =
  foldLeft (*) 1 (range 1 (z + 1))

Key idea: identify code via content hash

#Q82jfkasdf823jbc192
at bob '(factorial n)
at bob '(#Q82jfkasdf823jbc192 n)

.

.

.

.

.

.

.

.

factorial n =
  foldLeft (*) 1 (range 1 (n + 1))

Implication: codebase is a purely functional data structure

#Q82jfkasdf823jbc192
at bob '(factorial n)
at bob '(#Q82jfkasdf823jbc192 n)

.

.

factorial n = 42
#Zhsdf238jsfjasdWjd3

Advantages: perfect incremental compilation, better dependency management, easy renames that don't break user code, better refactoring...

Unison SBTB 2018

By Paul Chiusano

Unison SBTB 2018

  • 863
Loading comments...

More from Paul Chiusano