Node.js Scalability Tips

Luciano Mammino (@loige)

2020-07-09

πŸ‘‹ Hello, I am Luciano!

Principal Software Engineer at FabFitFun

Β Blog: loige.co

Β Twitter: @loige

Β GitHub: @lmamminoΒ 

Get the slides!

Node.js + Scalability? πŸ€”

🀣

"ScalabilityΒ is the property of a system to handle a growing amount of work by adding resources to the system"

β€” Wikipedia

"A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added"

β€” Werner Vogels

πŸ›«
Tip 1.
Establish a baseline

/?data=Hello%20Shift
const { createServer } = require('http')
const { URL } = require('url')
const QRCode = require('qrcode')

createServer(function handler (req, res) {
  const url = new URL(req.url, 'http://localhost:8080')
  const data = url.searchParams.get('data')
  if (!data) {
    res.writeHead(400) // bad request
    return res.end()
  }
  res.writeHead(200, { 'Content-Type': 'image/png' })
  QRCode.toFileStream(res, data, { width: 300 })
})
  .listen(8080)
autocannon -c 200 --on-port / -- node server.js
node server.js&
wrk -t8 -c200 -d10s http://localhost:8080/
autocannon -c 200 --on-port /?data=Hello%20Shift -- node server.js
Running 10s test @ http://localhost:8080/?data=Hello%20Shift
200 connections

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Stat    β”‚ 2.5%    β”‚ 50%     β”‚ 97.5%   β”‚ 99%     β”‚ Avg        β”‚ Stdev   β”‚ Max        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Latency β”‚ 1899 ms β”‚ 1951 ms β”‚ 2053 ms β”‚ 2054 ms β”‚ 1964.92 ms β”‚ 99.9 ms β”‚ 3364.03 ms β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Stat      β”‚ 1%  β”‚ 2.5% β”‚ 50%     β”‚ 97.5%  β”‚ Avg    β”‚ Stdev  β”‚ Min     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Req/Sec   β”‚ 0   β”‚ 0    β”‚ 30      β”‚ 199    β”‚ 99.5   β”‚ 94.27  β”‚ 30      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Bytes/Sec β”‚ 0 B β”‚ 0 B  β”‚ 50.7 kB β”‚ 336 kB β”‚ 168 kB β”‚ 159 kB β”‚ 50.7 kB β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Req/Bytes counts sampled once per second.

995 requests in 10.08s, 1.68 MB read

⛅️
Tip 1-bis
Also, find out your ceiling

const { createServer } = require('http')

createServer((req, res) => {
  if (req.method === 'GET' && req.url === '/') {
    res.writeHead(200, { 'Content-Type': 'text/plain' })
    res.end('Hello World\n')
  } else {
    res.statusCode = 404
    res.end()
  }
})
  .listen(8080)
autocannon -c 200 --on-port / -- node server.js
Running 10s test @ http://localhost:8080/
200 connections

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Stat    β”‚ 2.5% β”‚ 50%  β”‚ 97.5% β”‚ 99%   β”‚ Avg     β”‚ Stdev   β”‚ Max      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Latency β”‚ 3 ms β”‚ 5 ms β”‚ 11 ms β”‚ 14 ms β”‚ 5.51 ms β”‚ 2.71 ms β”‚ 80.63 ms β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Stat      β”‚ 1%      β”‚ 2.5%    β”‚ 50%    β”‚ 97.5%   β”‚ Avg     β”‚ Stdev   β”‚ Min     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Req/Sec   β”‚ 21087   β”‚ 21087   β”‚ 34623  β”‚ 35487   β”‚ 33258.4 β”‚ 4107.01 β”‚ 21077   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Bytes/Sec β”‚ 3.29 MB β”‚ 3.29 MB β”‚ 5.4 MB β”‚ 5.54 MB β”‚ 5.19 MB β”‚ 641 kB  β”‚ 3.29 MB β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Req/Bytes counts sampled once per second.

333k requests in 10.1s, 51.9 MB read

🍾
Tip 2.
Find your bottleneck

clinic doctor --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js
clinic flame --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js
clinic bubble --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js

🎳
Tip 3.
Understand your goals

What do we optimize for?

Throughput?

Memory?

Latency?

πŸ‘
Tip 4.
Always "observe"

I mean, in production!

Logs - Metrics - Traces

πŸš€
Tip 5.
Scale your architecture

Performance != Scalability

How can we scale a system
by adding resources?

The "Scale Cube"

x-axis

cloning

z-axis

partitioning

y-axis

functional decomposition

Cloning

Inside the same server

Reverse proxy

Load Balancer

Using multiple server

The cluster module

Master process

Worker process

Worker process

Worker process

const cluster = require('cluster')
const numCPUs = require('os').cpus().length

if (cluster.isMaster) {
  // Fork workers
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork()
  }
} else {
  // Worker code
  require('./server.js')
}

3-4x req/sec

(8 core)

You could also use Worker Threads

Check out piscina!

Cloning is the easiest strategy to scale a service...

Β 

... as long as your application is Stateless

Functional decomposition

a.k.a. "Micro-services"

API Gateway

/products

/cart

products DB

cart DB

Functional decomposition

a.k.a. "Micro-services"

API Gateway

/products

/cart

Functional decomposition can also be combined with cloning!

products DB

cart DB

Node.js is great for microservices

Microservices can also help with
scaling the organisation!

Microservices add complexity

  • Observability
  • Deployments
  • Versioning
  • Integration

Partitioning

Service and Data Partitioning along Customer Boundaries

Shard partitioning

/products/[A-L]/

/products/[M-Z]/

DB 1

DB 2

Partitioning is generally used
to scale databases

and
SaaS software geographically

Summary

πŸ›« Establish a baseline

🍾 Find your bottleneck

🎳 Understand your goals

πŸ‘ Always "observe"

πŸš€ Scale your architecture
(cloning, decomposition & partitioning)

Thank you!