Hi, I'm Adam
The problem
We build systems like this:
We think about systems like this:
What if we reified our mental model of systems?
Why do we think about systems horizontally anyway?
- It separates infrastructural concerns from application concerns
- It allows us to actually reason about the correctness of our application code
- It feels like the right level of abstraction
Introducing ulvm
universe level virtual machine
What're our goals?
- Write reusable, self-contained modules
- Define flows by stringing together invocations of those modules
- Define nested scopes in which those modules exist; scopes are the "places" of our system and may contain state
What constraints can be removed?
- We are looking for the minimal set of additional requirements that we need to impose in order to achieve our goals
- Any language, any paradigm
- Any infrastructure, any middleware
- Any build and deployment processes
ulvm...
- Is an open, full-system compiler with pluggable code generation
- Abstracts over generalized "calling conventions"
- Has deep macro support
- Supports the type of logic that actually helps engineers prove and maintain correctness
What abstractions do we need?
Module
- Self-contained code in any language
- Defined by scope-specific locator (e.g. npm module, local file path, gihub repo + file path)
- Module may be invoked and may provide one of a set of possible results
- If a module installs a piece of state, that state must be initialized
- Modules may also define transformers that are run at defined invocation steps, dependent upon the calling and receiving scope
Flow
- A graph of module or flow invocations
- Flows may contain modules from many scopes but have a home scope
Scope
- A long-lived entity (e.g. a process, container, machine, cluster, etc.)
- May be nested (processes live within containers, which live within clusters, etc.)
- Responsible for transforming ASTs into source code
Module combinators
- This is how we abstract over generalized "calling conventions"
- Given a module invocation and an AST that depends upon one of the possible results of the invocation, returns a new AST
- 2 cool implications:
- Directly generating an AST allows us to embed constructs other than direct invocations
- Requiring that module combinators can deal with multiple sets of results implies support for a generalized control flow construct
Scope builders
- Responsible for building a set of source and configuration files into a compiled artifact
Logic
- We can add arbitrary logical assertions/requirements to inputs and outputs
- We can add logical implications on modules
- We can generate tests for these assertions or require formal proofs that the underlying code satisfies the implications
- For example: can statically restrict some sensitive data to a particular network segment
What does a ulvm system look like?
(ulvm.core/defscope :todo-svc
"Todo todo-svc"
{:ulvm.core/runnable-env-ref
{:ulvm.core/builtin-runnable-env-loader-name :ulvm.re-loaders/project-file
:ulvm.core/runnable-env-descriptor {:path "scopes/nodejs.ulvm"}}
:ulvm.core/parent-scope :todo-svc-container
:ulvm.core/modules {:validate-todo {:ulvm.core/tags #{:js-sync}
:ulvm.core/mod-descriptor
{:local-filename "todo-svc/validators/auth"}
:ulvm.core/config {
:ulvm.arg-mappings/positional [[:todo]]}}
...
A scope
Users can implement their own scopes
Scopes have modules
(ulvm.core/defflow :create-todo [session todo-list-id todo-text due-date]
{:ulvm.core/home-scope :todo-svc
:ulvm.core/output-descriptor {:err [(:err valid-todo) (:err stored-session)]
:success [todo-response]}
(session-user {:session session} :as session-user)
(todo-validator {:user session-user
:todo-list-id todo-list-id} :as valid-todo)
((:store-session todo-db) {:user session-user
:todo-list-id todo-list-id
:todo-text todo-text
:due-date due-date } :as stored-todo)
(:make-todo-response {:todo stored-todo} :as session-response
:after [stored-session]))
A flow
Flow arguments
Possible results
Remote invocation
Implementation
Module Combinator
(ulvm.core/defmodcombinator :js-sync
"Synchronous javascript function"
{:ulvm.core/runnable-env-ref
{:ulvm.core/runnable-env-loader-name :ulvm.re-loaders/project-file
:ulvm.core/runnable-env-descriptor {:path "mod-combinators/js-sync.ulvm"}})
Users can implement their own module combinators
Open compiler
- Runnable environments define a set of runnable scopes, a set of exported flows, and a runner to invoke those flows
- Plugins depend on ideal flows, essentially flow interfaces
{:org.ulvm.mod-combinators.js-sync/js-sync
{:ulvm.core/artifact-loader
{:ulvm.core/builtin-artifact-loader-name :ulvm.artifact-loaders/docker-hub
:ulvm.core/artifact-descriptor {:image "ulvm-js-sync:latest"}}
:ulvm.core/runner
{:ulvm.core/builtin-runner-name :ulvm.runners/docker-container
:ulvm.core/runner-descriptor
{:image (ulvm.core/from-env :image)
:host-cfg {:network-mode "bridge"}}}}}
Runnable scope
Artifacts are loaded from somewhere
Artifacts need to be run
{:org.ulvm.mod-combinators.js-sync/block-with-results
{:ulvm.core/runner
{:ulvm.core/builtin-runner-name :ulvm.runners/http
:ulvm.core/runner-descriptor {:method :post
:url (ulvm.core/eval (str "http://"
(ulvm.core/from-env
:container-ip)
":8080/block"))
:body (ulvm.core/eval (pr-str *params*))
:headers {"content-type" "application/edn"}
:acceptable-statuses #{200}}}
:ulvm.core/ideal-flows #{:org.ulvm.mod-combinator/block-with-results}}
Runnable scope (cont'd)
Flows must have some way of being invoked
Flows can declare that they implement some ideal flows
What's with the builtins?
- Artifact loaders, runnable environment loaders, and runners all have a few builtin implementations but other implementations may be provided as their own runnable environments
Thank you
ulvm
By abrgr
ulvm
- 502