Hi, I'm Adam

The problem

We build systems like this:

We think about systems like this:

What if we reified our mental model of systems?

Why do we think about systems horizontally anyway?

  • It separates infrastructural concerns from application concerns
  • It allows us to actually reason about the correctness of our application code
  • It feels like the right level of abstraction

Introducing ulvm

universe level virtual machine

abrgr/ulvm

What're our goals?

  • Write reusable, self-contained modules
  • Define flows by stringing together invocations of those modules
  • Define nested scopes in which those modules exist; scopes are the "places" of our system and may contain state

What constraints can be removed?

  • We are looking for the minimal set of additional requirements that we need to impose in order to achieve our goals
    • Any language, any paradigm
    • Any infrastructure, any middleware
    • Any build and deployment processes

ulvm...

  • Is an open, full-system compiler with pluggable code generation
  • Abstracts over generalized "calling conventions"
  • Has deep macro support
  • Supports the type of logic that actually helps engineers prove and maintain correctness

What abstractions do we need?

Module

  • Self-contained code in any language
  • Defined by scope-specific locator (e.g. npm module, local file path, gihub repo + file path)
  • Module may be invoked and may provide one of a set of possible results
  • If a module installs a piece of state, that state must be initialized
  • Modules may also define transformers that are run at defined invocation steps, dependent upon the calling and receiving scope

Flow

  • A graph of module or flow invocations
  • Flows may contain modules from many scopes but have a home scope

Scope

  • A long-lived entity (e.g. a process, container, machine, cluster, etc.)
  • May be nested (processes live within containers, which live within clusters, etc.)
  • Responsible for transforming ASTs into source code

Module combinators

  • This is how we abstract over generalized "calling conventions"
  • Given a module invocation and an AST that depends upon one of the possible results of the invocation, returns a new AST
  • 2 cool implications:
    • Directly generating an AST allows us to embed constructs other than direct invocations
    • Requiring that module combinators can deal with multiple sets of results implies support for a generalized control flow construct

Scope builders

  • Responsible for building a set of source and configuration files into a compiled artifact

Logic

  • We can add arbitrary logical assertions/requirements to inputs and outputs
  • We can add logical implications on modules
    • We can generate tests for these assertions or require formal proofs that the underlying code satisfies the implications
  • For example: can statically restrict some sensitive data to a particular network segment

What does a ulvm system look like?


(ulvm.core/defscope :todo-svc
  "Todo todo-svc"
  {:ulvm.core/runnable-env-ref
   {:ulvm.core/builtin-runnable-env-loader-name :ulvm.re-loaders/project-file
    :ulvm.core/runnable-env-descriptor {:path "scopes/nodejs.ulvm"}}
   :ulvm.core/parent-scope :todo-svc-container
   :ulvm.core/modules {:validate-todo {:ulvm.core/tags #{:js-sync}
                                       :ulvm.core/mod-descriptor 
                                        {:local-filename "todo-svc/validators/auth"}
                                       :ulvm.core/config {
                                        :ulvm.arg-mappings/positional [[:todo]]}}
                      ... 

A scope

Users can implement their own scopes

Scopes have modules


(ulvm.core/defflow :create-todo [session todo-list-id todo-text due-date]
  {:ulvm.core/home-scope        :todo-svc
   :ulvm.core/output-descriptor {:err      [(:err valid-todo) (:err stored-session)]
                                 :success  [todo-response]}
  (session-user             {:session session}           :as session-user)
  (todo-validator           {:user         session-user
                             :todo-list-id todo-list-id} :as valid-todo)
  ((:store-session todo-db) {:user         session-user
                             :todo-list-id todo-list-id
                             :todo-text    todo-text
                             :due-date     due-date }    :as stored-todo)
  (:make-todo-response      {:todo stored-todo}          :as session-response
                                                         :after [stored-session]))

A flow

Flow arguments

Possible results

Remote invocation

Implementation

Module Combinator


(ulvm.core/defmodcombinator :js-sync
  "Synchronous javascript function"
  {:ulvm.core/runnable-env-ref
   {:ulvm.core/runnable-env-loader-name :ulvm.re-loaders/project-file
    :ulvm.core/runnable-env-descriptor {:path "mod-combinators/js-sync.ulvm"}})

Users can implement their own module combinators

Open compiler

  • Runnable environments define a set of runnable scopes, a set of exported flows, and a runner to invoke those flows
  • Plugins depend on ideal flows, essentially flow interfaces


{:org.ulvm.mod-combinators.js-sync/js-sync
 {:ulvm.core/artifact-loader
  {:ulvm.core/builtin-artifact-loader-name :ulvm.artifact-loaders/docker-hub
   :ulvm.core/artifact-descriptor {:image "ulvm-js-sync:latest"}}
  :ulvm.core/runner
  {:ulvm.core/builtin-runner-name :ulvm.runners/docker-container
   :ulvm.core/runner-descriptor
   {:image (ulvm.core/from-env :image)
    :host-cfg {:network-mode "bridge"}}}}}

Runnable scope

Artifacts are loaded from somewhere

Artifacts need to be run


{:org.ulvm.mod-combinators.js-sync/block-with-results
 {:ulvm.core/runner
  {:ulvm.core/builtin-runner-name :ulvm.runners/http
   :ulvm.core/runner-descriptor {:method :post
                                 :url (ulvm.core/eval (str "http://" 
                                                           (ulvm.core/from-env
                                                             :container-ip)
                                                           ":8080/block"))
                                 :body (ulvm.core/eval (pr-str *params*))
                                 :headers {"content-type" "application/edn"}
                                 :acceptable-statuses #{200}}}
  :ulvm.core/ideal-flows #{:org.ulvm.mod-combinator/block-with-results}}

Runnable scope (cont'd)

Flows must have some way of being invoked

Flows can declare that they implement some ideal flows

What's with the builtins?

  • Artifact loaders, runnable environment loaders, and runners all have a few builtin implementations but other implementations may be provided as their own runnable environments

Thank you