Compression Carcinized
drop-in compatible compression in rust
Folkert de Vries, GOSIM 2024
about
Trust Spanning Protocol
Rust + JS and Python APIs
Trifecta
We develop and maintain digital commons, open-source software and open standards for vital systems.
Trifecta Infrastructure Projects
privilege
boundary
time
synchronization
data
compression
sudo-rs
ntpd-rs
zlib-rs
Compression Carcinized
Folkert de Vries, GOSIM 2024
drop-in compatible compression in rust
a form of convergent evolution in which non-crab crustaceans evolve a crab-like body plan
carcinization
➡️
Compression Carcinized
Folkert de Vries, GOSIM 2024
zlib-rs, bzip2 and beyond
drop-in compatible compression in rust
zlib: a library you've used today
zlib: a library you've used today
zlib: a library you've used today
> objdump -T /usr/lib/x86_64-linux-gnu/libz.so | grep "compress"
0000000000010370 g DF .text 0000000000000022 ZLIB_1.2.0 compressBound
0000000000010360 g DF .text 000000000000000f Base compress
0000000000010220 g DF .text 000000000000013d Base compress2
0000000000010560 g DF .text 000000000000001c Base uncompress
00000000000103a0 g DF .text 00000000000001c0 ZLIB_1.2.9 uncompress2pub unsafe extern "C" fn compress2(
dest: *mut Bytef,
destLen: *mut c_ulong,
source: *const Bytef,
sourceLen: c_ulong,
level: c_int
) -> c_intproject goals
drop-in replacement for the zlib dynamic library
high-performance implementation for rust
Topics
crash course
compression
current state
the future
crash course compression
what does zlib do?
crash course compression
when few do trick?
Why use many byte
Why Compress?
cost
speed
Lossless Compression
assert_eq!(decompress(compress(data)), data)Recognizing patterns
foobarfoo
⬇️
foobar<offset = 6, len = 3>
Finding patterns
3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214808651328230664709384460955058223176
goal: find the (longest) <offset,len> insertions
Finding patterns
s
e
r
i
e
u
s
p
r
o
⬆️
Finding patterns
s
e
r
i
e
u
s
p
r
o
⬆️
Finding patterns
The window size determines how far back the offset can go
s
e
r
i
e
u
s
p
r
o
⬆️
Finding patterns
The compression level determines how hard we try to find the longest match
f
o
o
...
f
o
o
o
f
o
⬆️
o
o
Finding patterns
Finding patterns
f
o
o
b
a
r
f
o
o
...
⬆️
Finding patterns
f
o
o
b
a
r
f
o
o
...
⬆️
"foo" -> { 0 }
Finding patterns
f
o
o
b
a
r
f
o
o
...
⬆️
"foo" -> { 0 }
"oob" -> { 1 }Finding patterns
f
o
o
b
a
r
f
o
o
...
⬆️
"foo" -> { 0 }
"oob" -> { 1 }
"oba" -> { 2 }
"bar" -> { 3 }
"arf" -> { 4 }
"rfo" -> { 5 }
Finding patterns
very effective for web data, even at low compression levels
Streaming
zlib can stream compression and decompression
current state
how are we doing vs. the competition
World Domination
compatability
just better
zlib-adler: the OG
goal: stability
still supports 16-bit systems
does not use modern hardware well
zlib-ng: the next generation
goal: performance
removes legacy,
but API-compatible
uses SIMD to speed up the algorithm
miniz-oxide: better safe than sorry
goal: safety
a safe (but slow) rust implementation
does not cover the full zlib API
zlib-rs: a safer zlib
goal: safety & performance
faster through the use of SIMD
implements the full zlib API
compression speed
decompression speed
decompression on X86_64
~5% faster
decompression on ARM
on a Kunpeng 920 server: ~2% faster
on WASM
wasm work funded by devolutions: ~20% faster
how we beat C
staring at
assembly
detailed
benchmarks
improving the compiler
the future
evolving crab-like body plans
chickens and eggs
funding
adoption
towards adoption
mozilla is working towards testing zlib-rs on nightly
beyond zlib-rs: bzip2
will test-drive C2Rust
C2Rust
funded by nlnet / NGI
beyond zlib-rs: zstd
lack of diversity
pending funding
beyond zlib-rs: xz
lack of diversity
pending funding
Summary
Summary
why use many bytes when few do trick
Summary
why use many bytes when few do trick
unreasonably effective on web content
Summary
why use many bytes when few do trick
unreasonably effective on web content
try zlib-rs
Summary
why use many bytes when few do trick
unreasonably effective on web content
use more (unglamorous) rust in production
try zlib-rs
Summary
why use many bytes when few do trick
unreasonably effective on web content
use more (unglamorous) rust in production
evolve crab-like body plans
try zlib-rs
Thanks
Benchmark 1 (42 runs): target/release/examples/compress 1 rs silesia-small.tar
measurement mean ± σ min … max outliers delta
wall_time 119ms ± 1.97ms 117ms … 128ms 1 ( 2%) 0%
peak_rss 26.7MB ± 85.7KB 26.6MB … 26.9MB 0 ( 0%) 0%
cpu_cycles 406M ± 4.67M 399M … 424M 1 ( 2%) 0%
instructions 660M ± 469 660M … 660M 0 ( 0%) 0%
cache_references 8.06M ± 1.31M 5.65M … 11.3M 0 ( 0%) 0%
cache_misses 461K ± 36.5K 433K … 555K 5 (12%) 0%
branch_misses 3.59M ± 6.42K 3.58M … 3.61M 1 ( 2%) 0%
Benchmark 2 (43 runs): removed-bounds/release/examples/compress 1 rs silesia-small.tar
measurement mean ± σ min … max outliers delta
wall_time 118ms ± 2.53ms 115ms … 127ms 3 ( 7%) - 1.3% ± 0.8%
peak_rss 26.8MB ± 77.9KB 26.7MB … 27.0MB 0 ( 0%) + 0.2% ± 0.1%
cpu_cycles 400M ± 8.00M 391M … 437M 2 ( 5%) - 1.4% ± 0.7%
instructions 623M ± 522 623M … 623M 0 ( 0%) ⚡ - 5.6% ± 0.0%
cache_references 7.91M ± 1.45M 5.89M … 11.9M 1 ( 2%) - 1.9% ± 7.4%
cache_misses 458K ± 29.1K 433K … 550K 1 ( 2%) - 0.5% ± 3.1%
branch_misses 3.34M ± 7.35K 3.33M … 3.36M 0 ( 0%) ⚡ - 6.8% ± 0.1%is it bounds checks?
Lossy Compression
622kb
Atlantic Ghost Crab © Hans Hillewaert
Lossy Compression
622kb ➡ 87kb