Zapps
Warpforge

composable un-distros

let's do it

Overall:

  • Working towards more freely composable systems —
    towards “un-distros”
    • With a side of reliable, reproducible, secured supply chains.

2 parts:

  • First, an exciting part: linux packaging that anyone can use.

     
  • Second, a nerdier part: content-addressed build APIs.

Tarball, regular executable, ELF, dynamic, and yet totally portable

there are no post install hooks.  nothing.
read-only mount?  jumpdrive?  fine.

untar
run
done

DEMO

Recent discovery:

Zapps even work on FreeBSD.

(thanks largely to FreeBSD's capabilities to emulate linux...
but it's very cool that it "just worked" out-of-box!)

With many thanks to @FUZxxl

GOALS

  1. Want one answer.
    No "it depends".
     
  2. Must work on any host -- no host lib deps; no conflicts
     
  3. Work on any install path.  Even if unpredictable.
     
  4. No "post-install" steps.
    Work even on RO mounts.

High level:

... And in
   practice:

  1. Dynamic linking.
    Need it for Reasons.
     
  2. Want: able to dedup
    libraries on disk
    (if something else has exact same version.)

of Zapps

GOALS

  1. Want one answer.
    No "it depends".
     
  2. Must work on any host -- no host lib deps; no conflicts
     
  3. Work on any install path.  Even if unpredictable.
     
  4. No "post-install" steps.
    Work even on RO mounts.

High level:

... And in
   practice:

  1. Dynamic linking.
    Need it for Reasons.
     
  2. Want: able to dedup
    libraries on disk
    (if something else has exact same version.)

of Zapps

DONE.

The simplest thing that works

and nothing more

Please check out the Zapps website for more info:
 

https://zapps.app/

So why did I care about that?

So why did I care about that?

I want to build a whole decentralized,

low-coordination,

high-sharing,

ecosystem.

So let's talk about that
build tool now...

Warpforge

it builds things

Warpforge

it builds things

Declaratively.

Repeatably.

Shareably.

a build tool

collaborative, decentralized --
information designed to be shared.

Work together, but not in lockstep.

and supply chains

... that you put other build tools in, to universalize things.

for everyone.

... secured by hash trees.

warpforge

Warpforge

Demos again...

Let's see how we built the Zapps you saw in the previous demo.

Warpforge Replay instructions say exactly how.

warpforge

Demos again...

That was the "Catalog Site".

It's generated from a "Catalog" which is a merkle tree in JSON (or CBOR).

That's how we communicate about packages in Warpforge.

warpforge

In other words, that was the really high-level stuff.

...But low-level stuff
is Cool.

warpforge
warpforge

Formula

Results

L1

Compute

L2

Plot

Catalog

Release

Replay

Graphs

L3+

Starlark (?)

Other Scripts

Whatever

Generation

Warpforge API Layers

... because there's a lot of things going on here.

 <--- Simple ------------------ Fun --->

WareID
Warehouses

L0

Transport

warpforge

Formula

Results

L1

Compute

L2

Plot

Catalog

Release

Replay

Graphs

L3+

Starlark (?)

Other Scripts

Whatever

Generation

Warpforge API Layers

... because there's a lot of things going on here.

 <--- Simple ------------------ Fun --->

WareID
Warehouses

L0

Transport

We started here...

...now let's zoom in here.

Most Important Thing

Granular inputs.

 

Compose them freely.

warpforge
{"formula.v1": {
  "inputs": {
    "/zapp/bash": "ware:tar:57j2Ee9HEtDxRLE6uHA1xvmNB2LgkFyKqQuHFyGPN"
    "/zapp/coreutils": "ware:tar:mNB2LgqL3HeT5pCXr57j2Ee9HEtDxRLE6FyGPN"
    "/task/src": "ware:git:3616962f4dd2bda67b317583475f04486e763"
  },
  "action": {
    "script": {
      "interpreter": "/zapp/bash/bin/bash",
      "contents": [
        "mkdir /out && echo 'heyy' > /out/log"
        "echo done!"
      ]
    }
  },
  "outputs": {
    "yourlabel": {
      "packtype": "tar",
      "from": "/out"
    }
  }
}}

2nd Important Thing

Hashes.

Everything is

ID by hash.

warpforge
{"formula.v1": {
  "inputs": {
    "/zapp/bash": "ware:tar:57j2Ee9HEtDxRLE6uHA1xvmNB2LgkFyKqQuHFyGPN"
    "/zapp/coreutils": "ware:tar:mNB2LgqL3HeT5pCXr57j2Ee9HEtDxRLE6FyGPN"
    "/task/src": "ware:git:3616962f4dd2bda67b317583475f04486e763"
  },
  "action": {
    "script": {
      "interpreter": "/zapp/bash/bin/bash",
      "contents": [
        "mkdir /out && echo 'heyy' > /out/log"
        "echo done!"
      ]
    }
  },
  "outputs": {
    "yourlabel": {
      "packtype": "tar",
      "from": "/out"
    }
  }
}}

3rd Important Thing

This is an API.

 

It's just JSON.

warpforge
{"formula.v1": {
  "inputs": {
    "/zapp/bash": "ware:tar:57j2Ee9HEtDxRLE6uHA1xvmNB2LgkFyKqQuHFyGPN"
    "/zapp/coreutils": "ware:tar:mNB2LgqL3HeT5pCXr57j2Ee9HEtDxRLE6FyGPN"
    "/task/src": "ware:git:3616962f4dd2bda67b317583475f04486e763"
  },
  "action": {
    "script": {
      "interpreter": "/zapp/bash/bin/bash",
      "contents": [
        "mkdir /out && echo 'heyy' > /out/log"
        "echo done!"
      ]
    }
  },
  "outputs": {
    "yourlabel": {
      "packtype": "tar",
      "from": "/out"
    }
  }
}}

0'th Important Thing

It has to actually run.

 

So it does.

warpforge
warpforge

Formula

Results

L1

Compute

L2

Plot

Catalog

Release

Replay

Graphs

L3+

Starlark (?)

Other Scripts

Whatever

Generation

Warpforge API Layers

... because there's a lot of things going on here.

WareID
Warehouses

L0

Transport

Stuff on the right...
... creates the stuff on the left.
... Creates software.

So why did I care about that?

So why did I care about that?

I want to build a whole decentralized,

low-coordination,

high-sharing,

ecosystem.

So why did I care about that?

I want to build a whole decentralized,

low-coordination,

high-sharing,

ecosystem.

hashes help
with this

zapps & path-agno

help with this

catalog merkle-tree designed to sync and share

So why did I care about that?

And all that stuff I just showed

is reproducible

and ready to be shared.

Where are we going with this?

  • Build environment manager
    • For everything
  • Packaging and sharing
    • Datastructure driven
    • (bring your own frontend?)
  • CI and productivity
    • ...Obviously.
  • Not quite a distro.
    • But it should be very
      easy to build one.

warpforge.io

zapps.app

Stuff you can compose.

(Questions?)

github.com/warptools/

(extra slides)

Warpforge vs ... 

Bazel/Blaze/Nix/Guix/{Etc}:
Build descriptions depend on other build descriptions.

– have to keep building farther and farther back
– presumes compiler determinism (🚩)
– eventually have bootstrapping problem
– seems to defacto require monorepo


Warpforge:
Build descriptions depend on content.

– pause wherever you like
– checkpoints for determinism
– no trust needed for "substitutes"
– update freely (or don’t)
– can describe cycles (!)
      ^ describe DDC / thompson-atk-resistance!

 

There are more tools

  • `rio` -- Reproducible I/O
     
  • `linkwarp` -- $PATH management (a la Stow)
     
  • `gittreehash` -- it's the primary key you want for shared lib dedup, funny enough
     
  • `ldshim` -- part of the Zapp production system
# Langpkgs?

About language pkgman

LPMs do approx 4 things:

  1. package suggestion
  2. version selection
  3. lockfiles (hopefully!)
  4. transport & file unpack
     

First two are legit & interesting.

 

Second two are boring.
Can we do them once, please?

BHAG  (Big Hairy Audacious Goals)

  • Warpforge becomes the obvious choice for packaging, shipping, and CI.
     
  • Every repo on git{platform} has a .warpforge/ dir, and it's always a familiar landmark.
     
  • Language Pkgmgrs still exist... and focus on version selection and code integration.  Can freely reuse Warpforge infrastructure (Catalogs, ware ID’ing, warehouses and transports) rather than custom solutions.

I want something a little more granular than containers.

Building a clean room, then getting a bucket of mud from outside the door, and upending the whole thing on the floor...

...defeats the purpose.

zapps&warpforge@hip2022

By warpfork

zapps&warpforge@hip2022

  • 209

More from warpfork