Nea

the webserver that never allocates

Folkert de Vries, SYCL 2024

Nea is...

written in Rust

Proof of Concept

Nea is...

roc platform

web server

systems programming

Our Problem

webserver woes

A webserver

accept a new request
run a request handler
send response
GOTO 1

🔁

A webserver

in production

you pay for resources

locally

resource are infinite

🌄

💸

A webserver

running out of memory is bad for your users

"have you tried turning it off and on again" is not great ux

A webserver

you are at the mercy of your runtime

haskell/java/js/ruby/... runtimes are mediocre at everything.

Partial Solutions

heuristics

not guarantees

hard to find optimal settings

⚖️

Goal 1

get a firm upper bound on the amount of memory used by our webserver

🧠

Goal 2

a memory-hungry request should not impact any other request

Catching the Culprit

tracking down memory usage

catching the culprit

pub fn reserve(
    &mut self, additional: usize
);

pub fn try_reserve(
    &mut self, additional: usize,
) -> Result<(), TryReserveError>;

catching the culprit

in its lane, flourishing

catching the culprit

the malloc that broke the cammel's back

✖

catching the culprit

impossible to track down problematic requests

from https://stevenharman.net/so-we-have-a-memory-leak

Solutions

Nea's design

our starting point

refcounting, so no GC pauses

roc is (relatively) fast

🚀

♻️

solution 1: one big allocation

using mmap

(or similar)

don't want to recompile

get a firm upper bound on the amount of memory used by our webserver

constraint: worst case == best case

our memory consumption is always maximal

⛰️

solution 2: memory separation

a memory-hungry request should not impact other requests

idea: each request gets its own bump arena

the arena is cleared when a request is done

constraint: # of concurrent requests

we now run into a hard limit: there needs to be a bucket available in order to serve a request.

🪣

but we can do MATH!

constraint: similar work

bucket size is fixed: memory is used most effectively if most requests do a roughly equal amount of allocating

⚖️

error recovery

↩️

setjmp/longjmp

back up the stack

send error 500

log request

error recovery

#include < setjmp.h >

void main() {
  jmp_buf env;
  int i;

  i = setjmp(env);
  printf("i = %d\n", i);

  if (i != 0) { exit(0) };

  longjmp(env, 2);
  printf("Never gets here\n");
}

error recovery

setjmp:
    mov [rdi],    rbx     ; // Store caller saved registers
    mov [rdi+8],  rbp     ; // ^
    mov [rdi+16], r12     ; // ^
    mov [rdi+24], r13     ; // ^
    mov [rdi+32], r14     ; // ^
    mov [rdi+40], r15     ; // ^
    lea rdx,      [rsp+8] ; // go one value up (as if setjmp wasn't called)
    mov [rdi+48], rdx     ; // Store the new rsp pointer in env[7]
    mov rdx,      [rsp]   ; // go one value up (as if setjmp wasn't called)
    mov [rdi+56], rdx     ; // Store the address we will resume at in env[8]
    xor eax,      eax     ; // Always return 0
    ret

error recovery

longjmp:
    xor eax, eax         ; // set eax to 0
    cmp esi, 1           ; // CF = val ? 0 : 1
    adc eax, esi         ; // eax = val + !val
    mov rbx, [rdi]       ; // Load in caller saved registers
    mov rbp, [rdi+8]     ; // ^
    mov r12, [rdi+16]    ; // ^
    mov r13, [rdi+24]    ; // ^
    mov r14, [rdi+32]    ; // ^
    mov r15, [rdi+40]    ; // ^
    mov rsp, [rdi+48]    ; // Value of rsp before setjmp call
    jmp [rdi+56]         ; // goto saved address without altering rsp

error recovery

loop {
    work = queue.pop().await?;
    match setjmp(&mut jmp_buf) { 
        0 => { 
            // snapshot has been made 
            call_roc(work)
        }
        _ => {
            // allocation failed
            todo!()
        }
    }
}

fn roc_alloc() -> *mut c_void { 
    let ptr = ...;

    if ptr.is_null() {
        longjmp(&jmp_buf, 1)
    } else {
        ptr
    }
}

error recovery

setjmp/longjmp are so unsafe, even an unsafe block cannot contain them

#include < setjmp.h >

void main() {
  jmp_buf env;
  int i;

  i = setjmp(env);
  printf("i = %d\n", i);

  if (i != 0) { exit(0) };

  longjmp(env, 2);
  printf("Never gets here\n");
}

async IO

async IO makes sense when for a webserver

🧵

async IO

async fn foo() {
    // ...
    tcp_stream.flush().await;
}

rust has no idea how to actually run this

async IO

https://tweedegolf.nl/en/blog/114/building-an-async-runtime-with-mio

Future Work

make it more robust
run actual roc applications
benchmarks

Summary

roc platforms combine the convenient and performant

custom allocators are a superpower

hard bounds mean we can do MATH!

NEver Allocate!

Thanks

https://github.com/tweedegolf/nea

github: folkertdev

@folkertdev@hachyderm.io

constraints ⬌ guarantees

no free lunch

constraints ⬌ guarantees

> Array(8).join("wat" - 1) + " Batman!";

constraints ⬌ guarantees

> Array(8).join("wat" - 1) + " Batman!";
< 'NaNNaNNaNNaNNaNNaNNaN Batman!'

constraints ⬌ guarantees

Inductive Vector (A : Type) : nat -> Type :=
| VNil : Vector A 0
| VCons : forall (n : nat), A -> Vector A n -> Vector A (S n).