Luciano Mammino (@loige)

Node.js Scalability Tips

Node.js Scalability Tips

Vijayawada - 2021-12-13

Get the slides! ๐Ÿ‘‡

HEllo!

๐Ÿ‘‹ I'm Lucianoย (๐Ÿ‡ฎ๐Ÿ‡น๐Ÿ•๐Ÿ๐ŸคŒ)

๐Ÿ‘จโ€๐Ÿ’ป Senior Architect @ fourTheorem (Dublin ๐Ÿ‡ฎ๐Ÿ‡ช)

๐Ÿ“” Co-Author of Node.js Design Patternsย  ๐Ÿ‘‰

Let's connect!

ย  loige.co (blog)

ย  @loige (twitter)

ย  loige (twitch)

ย  lmammino (github)

We are business focused technologists that deliver.


Accelerated Serverlessย | AI as a Serviceย | Platform Modernisation

We are hiring: do you want to work with us?

"A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added"

โ€” Werner Vogels

๐Ÿ›ซ
Tip 1.
Establish a baseline

/?data=ciao๐Ÿ‘‹
import { createServer } from 'http'
import QRCode from 'qrcode'

createServer(function handler (req, res) {
  const url = new URL(req.url, 'http://localhost:8080')
  const data = url.searchParams.get('data')
  if (!data) {
    res.writeHead(200, {'Content-Type': 'text/html'})
    return res.end(`<html>
      <body>
        <p>To use this app you need to set the <code>data</code> querystring parameter.</p>
        <p>Try for example <a href="/?data=hello">/?data=hello</a></p>
      </body>
    </html>`)
  }
  res.writeHead(200, { 'Content-Type': 'image/png' })
  QRCode.toFileStream(res, data, { width: 300 })
}).listen(8080)
autocannon -c 200 --on-port / -- node server.js
node server.js&
wrk -t8 -c200 -d10s http://localhost:8080/
autocannon -c 200 --on-port /?data=ciao๐Ÿ‘‹ -- node server.js
Running 10s test @ http://localhost:8080/?data=ciao๐Ÿ‘‹
200 connections

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Stat    โ”‚ 2.5%    โ”‚ 50%     โ”‚ 97.5%   โ”‚ 99%     โ”‚ Avg        โ”‚ Stdev   โ”‚ Max        โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Latency โ”‚ 1899 ms โ”‚ 1951 ms โ”‚ 2053 ms โ”‚ 2054 ms โ”‚ 1964.92 ms โ”‚ 99.9 ms โ”‚ 3364.03 ms โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Stat      โ”‚ 1%  โ”‚ 2.5% โ”‚ 50%     โ”‚ 97.5%  โ”‚ Avg    โ”‚ Stdev  โ”‚ Min     โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Req/Sec   โ”‚ 0   โ”‚ 0    โ”‚ 30      โ”‚ 199    โ”‚ 99.5   โ”‚ 94.27  โ”‚ 30      โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Bytes/Sec โ”‚ 0 B โ”‚ 0 B  โ”‚ 50.7 kB โ”‚ 336 kB โ”‚ 168 kB โ”‚ 159 kB โ”‚ 50.7 kB โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Req/Bytes counts sampled once per second.

995 requests in 10.08s, 1.68 MB read

โ›…๏ธ
Tip 1-bis
Also, find out your ceiling

import { createServer } from 'http'

createServer((req, res) => {
  if (req.method === 'GET' && req.url === '/') {
    res.writeHead(200, { 'Content-Type': 'text/plain' })
    res.end('Hello World\n')
  } else {
    res.statusCode = 404
    res.end()
  }
}).listen(8080)
autocannon -c 200 --on-port / -- node server.js
Running 10s test @ http://localhost:8080/
200 connections

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Stat    โ”‚ 2.5% โ”‚ 50%  โ”‚ 97.5% โ”‚ 99%   โ”‚ Avg     โ”‚ Stdev   โ”‚ Max      โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Latency โ”‚ 3 ms โ”‚ 5 ms โ”‚ 11 ms โ”‚ 14 ms โ”‚ 5.51 ms โ”‚ 2.71 ms โ”‚ 80.63 ms โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Stat      โ”‚ 1%      โ”‚ 2.5%    โ”‚ 50%    โ”‚ 97.5%   โ”‚ Avg     โ”‚ Stdev   โ”‚ Min     โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Req/Sec   โ”‚ 21087   โ”‚ 21087   โ”‚ 34623  โ”‚ 35487   โ”‚ 33258.4 โ”‚ 4107.01 โ”‚ 21077   โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Bytes/Sec โ”‚ 3.29 MB โ”‚ 3.29 MB โ”‚ 5.4 MB โ”‚ 5.54 MB โ”‚ 5.19 MB โ”‚ 641 kB  โ”‚ 3.29 MB โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Req/Bytes counts sampled once per second.

333k requests in 10.1s, 51.9 MB read

๐Ÿพ
Tip 2.
Find your bottleneck

clinic doctor --autocannon [ -c 200 '/?data=ciao๐Ÿ‘‹' ] -- node server.js
clinic flame --autocannon [ -c 200 '/?data=ciao๐Ÿ‘‹' ] -- node server.js
clinic bubble --autocannon [ -c 200 '/?data=ciao๐Ÿ‘‹' ] -- node server.js

๐ŸŽณ
Tip 3.
Understand your goals

What do we optimize for?

Throughput?

Memory?

Latency?

๐Ÿ‘
Tip 4.
Always "observe"

I mean, in production!

Logs - Metrics - Traces

๐Ÿš€
Tip 5.
Scale your architecture

Performance != Scalability

How can we scale a system
by adding resources?

The "Scale Cube"

x-axis

cloning

z-axis

partitioning

y-axis

functional decomposition

Cloning

Inside the same server

Load Balancer

Using multiple server

Reverse proxy

The cluster module

Master process

Worker process

Worker process

Worker process

import cluster from 'cluster'
import { cpus } from 'os'

const numCPUs = cpus().length

if (cluster.isMaster) {
  // Fork workers
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork()
  }
} else {
  // Worker code here...
}

3-4x req/sec

(8 core)

You could also use Worker Threads

Check out piscina!

Cloning is the easiest strategy to scale a service...

ย 

... as long as your application is "Stateless"

Functional decomposition

a.k.a. "Micro-services"

API Gateway

/products

/cart

products DB

cart DB

Functional decomposition

a.k.a. "Micro-services"

API Gateway

/products

/cart

Functional decomposition can also be combined with cloning!

products DB

cart DB

Node.js is great for microservices

Microservices can also help with
scaling the organisation!

Microservices add complexity

  • Observability
  • Deployments
  • Versioning
  • Integration

Partitioning

Service and Data Partitioning along Customer Boundaries

Shard partitioning

/products/[A-L]/

/products/[M-Z]/

DB 1

DB 2

Partitioning is generally used
to scale databases

and
SaaS software geographically

Summary

๐Ÿ›ซ Establish a baseline

๐Ÿพ Find your bottleneck

๐ŸŽณ Understand your goals

๐Ÿ‘ Always "observe"

๐Ÿš€ Scale your architecture
(cloning, decomposition & partitioning)

Thank you!

โ˜๏ธ nodejsdp.link

Node.js: scalability tips - Azure Dev Community Vijayawada

By Luciano Mammino

Node.js: scalability tips - Azure Dev Community Vijayawada

You finally built that amazing start-up idea you had in mind for years and you did it using Node.js! That's Great! You just launched it on Hacker News and you are very happy and proud... but now more and more people are using it and you start to have a nasty fear that Node.js won't scale because you now... it's single-threaded! Is your project doomed now? Do you have to invest your time on rewriting it in something like C++ or maybe Rust or even Go? You'd rather invest your time on adding valuable features for your users rather than learning a new language and rewriting everything from scratch, but what if nothing works anymore? And... by the way, what the heck "single-threaded" really means?! Fear no more, dear fellow developer! In this talk, we will discuss the architecture of Node.js going through its strengths and its weaknesses. We will then talk about scalability and I will share some valuable tips and tricks to make your Node.js app scale! Spoiler alert: you probably won't need Go or Rust :)

  • 1,478