Luciano Mammino PRO
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com
JavaScript Iteration Protocols
Luciano Mammino (@loige)
2023-09-20
A new adventure awaits you!
You just joined a new STARTUP.
This is DAY 1!
CTO: Can you help me with troubleshooting something?
CTO: I am debugging a PRODUCTION issue.
CTO: We need to count how many "ERR_SYS_FCKD" there are per customer
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5KGHRV3SNJ71C4E920872","error":"ERR_SYS_FCKD"}
{"requestId":"3e3e49b5...","level":"ERROR","timestamp":"2023-05-26T23:18:50.277Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_CLIENT_TIMEOUT"}
{"requestId":"f28abf98...","level":"ERROR","timestamp":"2023-05-26T23:18:56.575Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_BUSY"}
{"requestId":"608e579f...","level":"ERROR","timestamp":"2023-05-26T23:19:04.529Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_NOT_FOUND"}
{"requestId":"3125588e...","level":"ERROR","timestamp":"2023-05-26T23:19:11.514Z","customerId":"01H1D5MJ4XNJ580DMM8Y7T1HRW","error":"ERR_SERVER_TIMEOUT"}
{"requestId":"40fb0329...","level":"INFO","timestamp":"2023-05-26T23:19:13.536Z","customerId":"01H1D5N6HS2EMA900A2V6NQQ2Q","message":"user logged in"}
{"requestId":"564afe31...","level":"ERROR","timestamp":"2023-05-26T23:19:21.708Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_CLIENT_TIMEOUT"}Let's do it!
import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)Errors by customer:
{
'01H1D5NGA5D689HD57CN5BZSG3': 109,
'01H1D5MZAVPSXSXPTYADF4BDND': 118,
'01H1D5KGHRV3SNJ71C4E920872': 118,
'01H1D5KSG0K81CCAGTSTBXHFA6': 95,
'01H1D5MJ4XNJ580DMM8Y7T1HRW': 103,
'01H1D5N6HS2EMA900A2V6NQQ2Q': 96,
'01H1D5M5RTKKK5SC4ADWAPK7Q0': 130,
'01H1D5K2JDVD941R6H67YEY0G8': 115,
'01H1D5KZ7GV7HK213XE3WV5GQX': 113
}CTO: Great job! Exactly what I needed!
DAY 2
CTO: that script it's now giving "RangeError: Invalid string length"!
import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)import { readFile } from 'node:fs/promises'
const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
if (!acc[error.customerId]) {
acc[error.customerId] = 0
}
acc[error.customerId]++
return acc
}, {})
console.log('Errors by customer:')
console.table(errorsByCustomer)We are loading everything into memory...
If the file is too big this will fail!
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
//{
}Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
}OK
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}NOPE
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1
}OK
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 1,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}OK
Ideal approach
Lazy processing, line by line
Current line:
Current state:
Parse
Filter
Aggregate
{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}{
"01H1D5NGA5D689HD57CN5BZSG3": 2,
"01H1D5MZAVPSXSXPTYADF4BDND": 1
}Ideal approach
Lazy processing, line by line
We can do this using Iterators!
Player 1
I'm Luciano
Senior Architect @ fourTheorem (Dublin)
Co-Author of Node.js Design Patterns
Always re-imagining
We are a pioneering technology consultancy focused on Cloud, AWS & serverless
We can help with:
Cloud Migrations
Training & Cloud enablement
Building serverless applications
Cutting cloud costs
Why Iterators?
-
Lazy abstraction to represent repetition
-
You can consume the collection 1 item at the time
-
Great for large (or endless) datasets
The concept of Iterators exists already in JavaScript...
Good news...
... since ES2015!
const array = ['foo', 'bar', 'baz']
for (const item of array) {
console.log(item)
}
Prints all the items in the array!
Does it need to be an array? 🤔
Output:
foo
bar
baz
const str = 'foo'
for (const item of str) {
console.log(item)
}Output:
f
o
o
const obj = {
foo: 'bar',
baz: 'qux'
}
for (const item of obj) {
console.log(item)
}Output: ⛔️ Uncaught TypeError: obj is not iterable
OMG `for ... of`
does not work with plain objects! 😱
const array = ['foo', 'bar', 'baz']
console.log(...array)Output:
foo bar baz
spread syntax!
Generators
Generator fn & obj
function * myGenerator () {
// generator body
yield 'someValue'
// ... do more stuff
}const genObj = myGenerator()
genObj.next() // -> { done: false, value: 'someValue' }function * fruitGen () {
yield '🍑'
yield '🍉'
yield '🍋'
yield '🥭'
}
const fruitGenObj = fruitGen()
console.log(fruitGenObj.next()) // { value: '🍑', done: false }
console.log(fruitGenObj.next()) // { value: '🍉', done: false }
console.log(fruitGenObj.next()) // { value: '🍋', done: false }
console.log(fruitGenObj.next()) // { value: '🥭', done: false }
console.log(fruitGenObj.next()) // { value: undefined, done: true }function * fruitGen () {
yield '🍑'
yield '🍉'
yield '🍋'
yield '🥭'
}
const fruitGenObj = fruitGen()
// generator objects are iterable!
for (const fruit of fruitGenObj) {
console.log(fruit)
}
// 🍑
// 🍉
// 🍋
// 🥭Iterators & Iterables
Iterator Obj
An object that acts like a cursor to iterate over blocks of data sequentially
Iterable Obj
An object that provides data that can be iterated over sequentially
The iterator protocol
Iterator object:
next() method returns:
-
done (boolean)
-
value (any)
const iterator = {
next () {
return {
done: false,
value: "someValue"
}
}
}function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}A factory function that creates an iterator
returns an object
... which has a next() method
... which returns an object with
done & value
const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true }function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}Generator objects implement the iterator protocol
function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}Equivalent code using generators
The iterable protocol
Iterable object:
Symbol.iterator() method returns:
-
an iterator
const iterable = {
[Symbol.iterator] () {
return {
// iterator
next () {
return {
done: false,
value: "someValue"
}
}
}
}
}function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}[...createCountdown(3)] // [3,2,1,0]
for (const value of createCountdown(3)) {
console.log(value)
}
// 3
// 2
// 1
// 0We can use spread and for ... of with iterable objects!
function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
})
}
}Generator objects implement the iterable protocol
function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}Equivalent code using generators
YES, it's the same code as the iterator example!
const iterableIterator = {
next () {
return { done: false, value: 'hello' }
},
[Symbol.iterator] () {
return this
}
}An object can be both an iterator and an iterable!
Iterator protocol
Iterable protocol
ASYNC
Iterators & Iterables
The async iterator protocol
Async Iterator object:
next() method returns a
Promise that resolves to:
-
done (boolean)
-
value (any)
const asyncIterator = {
async next () {
return {
done: false,
value: "someValue"
}
}
}The async iterable protocol
Async Iterable object:
Symbol.asyncIterator() method returns:
-
an async iterator
const asyncIterable = {
[Symbol.asyncIterator] () {
return {
// iterator
async next () {
return {
done: false,
value: "someValue"
}
}
}
}
}import { setTimeout } from 'node:timers/promises'
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}Async Generator objects implement the async iterator & async iterable protocols
const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value)
}
We can use for await ... of with async iterable objects!
Let's rewrite our log analyser to use iterators!
import { createReadStream } from 'node:fs'
const readable = createReadStream(
'logs.jsonl',
{ encoding: 'utf-8' }
)
// a readable stream is an Async Iterable!
for await (const chunk of readable) {
// do something with a chunk of data
}We can read a file incrementally using Node.js streams!
import { createReadStream } from 'node:fs'
const readable = createReadStream(
'logs.jsonl',
{ encoding: 'utf-8' }
)
// a readable stream is an Async Iterable!
for await (const chunk of readable) {
// do something with a chunk of data
}We can read a file incrementally using Node.js streams!
A chunk is an arbitrary amount of data (not necessarily a line)
// utils/byline.js
export async function * byLine (asyncIterable) {
let remainder = ''
for await (const chunk of asyncIterable) {
const lines = (remainder + chunk).split(/\n+/)
remainder = lines.pop()
yield * lines
}
if (remainder.length > 0) {
yield remainder
}
}
import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'
const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })
const errorsByCustomer = {}
for await (const line of byLine(readable)) {
const message = line === '' ? {} : JSON.parse(line)
if (message.error === 'ERR_SYS_FCKD') {
if (!errorsByCustomer[message.customerId]) {
errorsByCustomer[message.customerId] = 0
}
errorsByCustomer[message.customerId]++
}
}
console.log('Errors by customer:')
console.log(errorsByCustomer)
import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'
const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })
const errorsByCustomer = {}
for await (const line of byLine(readable)) {
const message = line === '' ? {} : JSON.parse(line)
if (message.error === 'ERR_SYS_FCKD') {
if (!errorsByCustomer[message.customerId]) {
errorsByCustomer[message.customerId] = 0
}
errorsByCustomer[message.customerId]++
}
}
console.log('Errors by customer:')
console.log(errorsByCustomer)
CTO: It works with
any file now!
Can't we use .map(), .filter(), .reduce()?
-
you can wait for Iterator Helpers & Async Iterator Helpers TC39 proposals
-
... or you can use core-js to polyfill these features today!
BONUS Material
SLIDES: loige.link/iter-summit
Code: loige.link/iter-summit-code
Original Front cover photo by Jörg Angeli on Unsplash
Original Background photo by Michael Behrens on Unsplash
Thanks!
JavaScript Iteration protocols - JavaScript Global Summit 2023
By Luciano Mammino
JavaScript Iteration protocols - JavaScript Global Summit 2023
How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for…in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!
