Caching for Cash πŸ€‘

Kent C. Dodds

More than you knew you need to know about caching

Let's wake up

Your brain needs this 🧠

What this talk is

  • Deep dive on caching fundamentals
  • Code examples

What this talk is not

  • Comprehensive

Let's
Get
STARTED!

Two ways to make your code faster:

  1. Delete it
  2. Reduce the amount of stuff the code is doing

Delete it

Reduce the...

stuff

Can't delete it. Can't reduce it. Can't "make it fast."

πŸ€‘ Cash it! πŸ€‘

πŸ₯΄ Cache it! πŸ₯΄

What is caching?

In computing, a cache is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere. A cache hit occurs when the requested data can be found in a cache, while a cache miss occurs when it cannot. Cache hits are served by reading data from the cache, which is faster than recomputing a result or reading from a slower data store; thus, the more requests that can be served from the cache, the faster the system performs.

Caching by example

function computePi() {
	let pi = 0
	let sign = 1
	for (let i = 0; i < 1000000; i++) {
		const term = sign / (2 * i + 1)
		pi += term
		sign *= -1
	}
	return Math.round(pi * 4e10) / 1e10
}

Caching by example

let pi
function computePiCached() {
	if (typeof pi === 'undefined') {
		pi = computePi()
	}
	return pi
}

Store the result of a computation somewhere and return that stored value instead of recomputing it again.

πŸ’‘ Caching:

More complexity

function computePi(precision: number) {
	let pi = 0
	let sign = 1
	for (let i = 0; i < 1000000; i++) {
		pi += sign / (2 * i + 1)
		sign *= -1
	}
	const factor = 10 ** precision
	return Math.round(pi * 4 * factor) / factor
}

More complexity

const piCache = new Map<number, number>()
function computePiCached(precision: number) {
	if (!piCache.has(precision)) {
		piCache.set(precision, computePi(precision))
	}
	return piCache.get(precision)
}

πŸ”‘ Cache Keys!!

Another example

function sum(a: number, b: number) {
	 return a + b
}

const sumCache = new Map<string, number>()
function sumCached(a: number, b: number) {
	const key = `${a},${b}`
	if (!sumCache.has(key)) {
		sumCache.set(key, sum(a, b))
	}
	return sumCache.get(key)
}

sumCached(1, 2) // cache miss: 3
sumCached(1, 2) // cache hit: 3

The problem with keys

function addDays(count: number) {
  const millisecondsInDay = 1000 * 60 * 60 * 24
  return new Date(Date.now() + count * millisecondsInDay)
}

const cache = new Map<string, Date>()
function addDaysCached(count: number) {
  const key = `add-days:${count}`
  if (!cache.has(key)) {
    cache.set(key, addDays(count))
  }
  return cache.get(key)
}

find the bug πŸ›

πŸ›

addDaysCached(3) // cache miss: 3 days from today
addDaysCached(3) // cache hit: 3 days from today
// ... wait 24 hours...
addDaysCached(3) // cache hit: 3 days from yesterday 😱
// That's 2 days from today!

The cache key must* account for all inputs required to determine the result

πŸ’‘ Cache Keys:

*But...

  1. It’s easy to miss an input
  2. Too many inputs
  3. Computing correct cache keys is costly

1. It’s easy to miss an input

useMemo(() => {
	// ...
}, [/* ... ugh... */])

2. Too many inputs

3. Computing correct cache keys is costly

So we cheat:

Cache revalidation

Cache Revalidation

  1. Proactively Updating the Cache
    On post update, update the cache
  2. Timed Invalidation
    Cache-Control headers
  3. Stale While Revalidate
    Update cache in the background
  4. Forcing fresh value
    Manual cache updates
  5. Soft Purge 🀩
    Manual Stale While Revalidate

Another caching problem

import fs from 'fs'

function getVideoBuffer(filepath: string) {
	return fs.promises.readFile(filepath)
}

const videoBufferCache = new Map<string, Buffer>()
async function getVideoBufferCached(filepath: string) {
	if (!videoBufferCache.has(filepath)) {
		videoBufferCache.set(filepath, await getVideoBuffer(filepath))
	}
	return videoBufferCache.get(filepath)
}

find the bug πŸ›

πŸ›

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6E7A84E05 node::Abort+21
 2: 00007FF6E7A84F49 node::OnFatalError+297
 3: 00007FF6E7A84D2F node::OnFatalError+31
 4: 00007FF6E7A84D17 node::OnFatalError+23
 5: 00007FF6E812C7F8 v8::Utils::ReportOOMFailure+184
...

Cache Size Solutions

  1. Least Recently Used
    Ditch old stuff (npm.im/lru-cache)
  2. File System
    require('os').tmpdir() or ./node_modules/.cache
  3. SQLite
    Even distributed with LiteFS
  4. Redis
    Super common, very fast, "more than a cache"

Note: cache size can still get out of control, so keep an eye out!

Cache Warming

Problems

  1. You can get rate limited by APIs
  2. It requires a lot of resources
  3. Making users wait for the fresh values

Solution: Soft Purge

Cache Entry Value Validation

Cache Request Deduplication

Kinda like DataLoader

You're looking for cachified

Thank you!

Caching for Cash πŸ€‘

By Kent C. Dodds

Caching for Cash πŸ€‘

It's often said that the two hardest problems in programming are caching, naming things, and off by one errors. Some degree of caching is required in almost every application to drastically improve performance. Unfortunately, not only is it easy to get wrong, there are also lots of different layers and methods to implement caching with different trade-offs. In this talk, Kent will explain the key principles of caching. We'll go through several real world examples of issues where caching is a great solution. We'll also explore different approaches and their associated trade-offs. We'll cover a ton of ground here. This one's gonna be fun!

  • 1,524