Merkle Trees

and JS Bundles

Or how I learned to stop worrying and love JS

@liamzebedee | liamz.co

Context

  • Distributing apps on the web is like sending the bible for every single church reading

    • Bundles > 2MB

  • Recent innovations:

    • Webpack

    • Yarn / NPM united packages

    • React

    • Styled-components

    • PWA’s

  • The future

    • Edge computing

    • ?Blockchains?

Current things

  • Bundle identifiers (2abd79e7d9fg.js) and c-hash-ing

  • SplitChunks

    • Static analysis of dependencies into a graph

    • Split out into many <script src=> and load async

  • Minimization (FooBar -> a.b)

  • Prefetch/preload

  • HTTP2 Push

Problems/Opportunities

Problems:

  • Small changes => more download => less battery life (mobile)

  • Download a whole lot more than I need to

  • We don’t really cache code on webpages, so reloads are always slow

Oppportunity:

  • XKCD (web is instant apps) - AR etc.

  • WebAssembly structured stack machine

  • P2P CDN’s?

 

Task

Deliver code for execution

  • Minimize bandwidth
  • Minimize communications overhead
  • Same/negligible difference in processing power

Ideation

  • What if we downloaded only what changed?

    • How?

  • Naive approach: delta

    • Knowledge of \( C_t \) for \( 0..t \)

    • Problem: How can we synchronise state without high bandwidth usage?

  • We already compare the hashes of assets, but this is for the entire file. Can we do it on a more granular scale

Merkle Trees and AST’s

  • Merkle tree (popularised in blockchain world)

    • A tree data structure

    • Where every non-leaf node is hash of its children

Merkle Trees and AST’s

​Compilers turn code into lower-level code. Example:

 

// Compile JSX (React syntax) to JS (vanilla JavaScript)
//

source = fs.readFile('something.jsx')

// export default const blah = () => {
//    return <div>Hello world</div>;
// }


compiled = runLoaders(src)

// module.exports = function blah() {
//    return React.createElement('div', { children: "Hello world" })
// }

Merkle Trees and AST’s

Compilation:

  • lexed tokens (BRACE_START BRACE_END)
  • abstract syntax tree
  • bytecode

An AST

Merkle AST's

​

  1. Take the AST of code
  2. Generate a Merkle tree as a compact representation of our AST

 

What does that look like?

 

Merkle AST's - Example

​

function shout() {
    console.log("BLOCKCHAIN!");
}


// AST
Function [name=shout]
    Call [name=console.log]
        Literal ["BLOCKCHAIN!"]


// Merkle AST
42
    123
        543

Merkle AST's - Example

​

function shout() {
    console.log("BLOCKCHAIN!");
}


// AST
Function [name=shout]
    Call [name=console.log]
        Literal ["BLOCKCHAIN!"]


// Merkle AST
42
    123
        543

function shout() {
    console.log("BLOCKCHAIN!", "IOT!");
}


// AST
Function [name=shout]
    Call [name=console.log]
        Literal ["BLOCKCHAIN!"]
        Literal ["IOT!"]


// Merkle AST
86
    99
        543
        264

*changed

Merkle AST's - Example

​

Send client tree to server, compare the Merkle AST's


86*
    99*
        543
        264*

Client

42
    123
        543

Server

86
    99
        543
        264

What's changed?

Merkle AST's - Example

​

Send client to server, compare the Merkle AST's


86*
    99*
        543
        264*

Client

42
    123
        543

Server

86
    99
        543
        264

What's changed?

Merkle AST's - Example

​

We send only the diffs to the client

Merkle AST's - Example

​

We send only the diffs to the client

They patch their source

And run the bundle

Merkle AST's - Optimisations

​

  • Set root of tree as a cookie (automatically sent with every HTTP request)
  • Maintain mapping of (root => builtBundle) so diff'ing tree is memoized

JS and Delivery

​

Loading the bundle (client)

// GET /
tree = localStorage.get('tree') || {};

fetch('/bundle', { tree })
.then(res => {
    tree.merge(res.changes)
    webpackBootstrap(tree.modules())
})



class tree {
    modules: { [id]: string } = {};
    tree = {};
    
    merge(changes) {
        this.modules = changes.map((id, diffs) => {
            let module = this.modules[id];
            diffs.map({ from, to, content } => {
                module = module.updateRange(from, to, content);
            })
            return module;
        })
        
        this.recomputeTree()
    }

    recomputeTree() {
        for({ id, module } in Object.entries(this.modules)) {
            this.trees[id] = buildMerkleAst(module);
        })
    }

    get modules() {
        return Object.values(this.modules);
    }
}

JS and Delivery

​

Serving the bundle (server)

handle('/bundle', (req, res) => {
    let trees = req.trees;
    let changeset = trees.map((id, tree) => {
        let module = modules[id];
        
        let diff = treediff(module.tree, tree);
        let changes = diff.nodes.map(node => {
            let { from, to } = module.tree.getRange(node)
            let diff = module.getCode(from, to);
            return { from, to, diff }
        })
        
        return { id, changes }
    })
    
    res.send(changeset);
})

Current Status (WIP)

​

liamzebedee/go-json-merkle-tree - builds merkle tree of JSON blob (AST) and outputs differing nodes

 

webpack-merkle-ast (WIP)

hooks into Webpack build pipeline

 

merkle-ast-server (tbc)

serves these diff's to clients

Questions?

​

Liam Zebedee

liamz.co