Luciano Mammino PRO
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com
Reiterating JavaScript & Node.js iterators
Luciano Mammino (@loige)
RomaJS - June 16, 2021
How many ways do you know to "do iteration" in JavaScript & Node.js? 😰
for i / for...in / for...of
while / do while
Array.forEach / Array.map / Array.flatMap / Array.reduce / Array.reduceRight / Array.filter / Array.find / Array.findIndex / Array.entries / Array.values / Array.every / Array.some
Object.keys / Object.values / Object.entries
Iterators / Generators
Spread operator [...iterable]
Events / Streams
Async iterators / Async generators
for await...of
That was 28 different concepts! 😳
📝 AGENDA
- Iteration protocols... What? Why?
- Syntax review
- Iteration protocols
- Iterator protocol
- Iterable protocol
- Generator functions
- Async iterator protocol
- Async iterable protocol
- Tips & tricks
Let me introduce myself...
I'm Luciano (🇮🇹🍕🍝) 👋
👨💻 Senior Architect
Co-Author of Node.js Design Patterns 👉
We are business focused technologists that deliver.
Accelerated Serverless | AI as a Service | Platform Modernisation
WE ARE HIRING: Do you want to work with us?
Iteration protocols
what? why? 🤔
- An attempt at standardizing "iteration" behaviors providing a consistent and interoperable API
- Use for...of, for await...of and spread operator.
- You can also create lazy iterators
- You can also deal with async iteration
- You can create your own custom iterators/iterables
Syntax Review 🧐
const judokas = [
'Driulis Gonzalez Morales',
'Ilias Iliadis',
'Tadahiro Nomura',
'Anton Geesink',
'Teddy Riner',
'Ryoko Tani'
]
for (const judoka of judokas) {
console.log(judoka)
}for...of
Driulis Gonzalez Morales
Ilias Iliadis
Tadahiro Nomura
Anton Geesink
Teddy Riner
Ryoko Tani
OUTPUT
const judoka = 'Ryoko Tani'
for (const char of judoka) {
console.log(char)
}
for...of (with strings)
R
y
o
k
o
T
a
n
i
OUTPUT
const medals = new Set([
'gold',
'silver',
'bronze'
])
for (const medal of medals) {
console.log(medal)
}
for...of (with Set)
gold
silver
bronze
OUTPUT
const medallists = new Map([
['Teddy Riner', 33],
['Driulis Gonzalez Morales', 16],
['Ryoko Tani', 16],
['Ilias Iliadis', 15]
])
for (const [judoka, medals] of medallists) {
console.log(`${judoka} has won ${medals} medals`)
}for...of (with Map)
Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
'Teddy Riner': 33,
'Driulis Gonzalez Morales': 16,
'Ryoko Tani': 16,
'Ilias Iliadis': 15
}
for (const [judoka, medals] of Object.entries(medallists)) {
console.log(`${judoka} has won ${medals} medals`)
}for...of (with object & Object.entries)
Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
'Teddy Riner': 33,
'Driulis Gonzalez Morales': 16,
'Ryoko Tani': 16,
'Ilias Iliadis': 15
}
for (const [judoka, medals] of medallists) {
console.log(`${judoka} has won ${medals} medals`)
}for...of (with object literals)
for (const [judoka, medals] of medallists) {
^
TypeError: medallists is not iterable
at Object. (.../05-for-of-object.js:8:32)
ERROR
const countdown = [3, 2, 1, 0]
// spread into array
const from5to0 = [5, 4, ...countdown]
console.log(from5to0) // [ 5, 4, 3, 2, 1, 0 ]
// spread function arguments
console.log('countdown data:', ...countdown)
// countdown data: 3 2 1 0Spread operator
import {
DynamoDBClient,
paginateListTables
} from '@aws-sdk/client-dynamodb'
const client = new DynamoDBClient({});
for await (const page of paginateListTables({ client }, {})) {
// page.TableNames is an array of table names
for (const tableName of page.TableNames) {
console.log(tableName)
}
}for await...of (async iterable)
Iteration Protocols 😵💫
Iterator protocol
In JavaScript, an object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.
function createCountdown (start) {
let nextVal = start
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}Countdown iterator
const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }Iterable protocol
An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.
* Symbol.iterator
function createCountdown (start) {
let nextVal = start
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
})
}
}
Countdown iterable
const countdown = createCountdown(3)
for (const value of countdown) {
console.log(value)
}
// 3
// 2
// 1
// 0Can an object be both an iterator and an iterable?! 🤨
const iterableIterator = {
next() {
return { done: false, value: "hello" }
},
[Symbol.iterator]() {
return this
}
}Iterator + Iterable
iterableIterator.next()
// { done: false, value: "hello" }
for (const value of iterableIterator) {
console.log(value)
}
// hello
// hello
// hello
// ...Generators
A generator function "produces" an object that is both an iterator and an iterable! 🤯
function * createCountdown (start) {
for (let i = start; i >= 0; i--) {
yield i
}
}// As iterable
const countdown = createCountdown(3)
for (const value of countdown) {
console.log(value)
}
// 3
// 2
// 1
// 0// As iterator
const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }Well, what about async iteration? 🤌
Async Iterator protocol
An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.
import { setTimeout } from 'timers/promises'
function createAsyncCountdown (start, delay = 1000) {
let nextVal = start
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
}const countdown = createAsyncCountdown(3)
console.log(await countdown.next()) // { done: false, value: 3 }
console.log(await countdown.next()) // { done: false, value: 2 }
console.log(await countdown.next()) // { done: false, value: 1 }
console.log(await countdown.next()) // { done: false, value: 0 }
console.log(await countdown.next()) // { done: true }
Async Iterable protocol
An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.
* Symbol.asyncIterator
import { setTimeout } from 'timers/promises'
function createAsyncCountdown (start, delay = 1000) {
return {
[Symbol.asyncIterator]: function () {
let nextVal = start
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
}
}
}const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value) // 3 ... 2 ... 1 ... 0
}import { setTimeout } from 'timers/promises'
async function * createAsyncCountdown (start, delay = 1000) {
for (let i = start; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value) // 3 ... 2 ... 1 ... 0
}
When to use async iterators
Sequential iteration pattern
Data arriving in order over time
You need to complete processing the current “chunk” before you can request the next one
-
Examples
paginated iteration
consuming tasks from a remote queue
Tips & pitfalls ☠️
import { createReadStream } from 'fs'
const sourceStream = createReadStream('bigdata.csv')
let bytes = 0
for await (const chunk of sourceStream) {
bytes += chunk.length
}
console.log(`bigdata.csv: ${bytes} bytes`)Node.js readable streams are async iterators
What about
backpressure? 😩
import { createReadStream } from 'fs'
import { once } from 'events'
const sourceStream = createReadStream('bigdata.csv')
const destStream = new SlowTransform()
for await (const chunk of sourceStream) {
const canContinue = destStream.write(chunk)
if (!canContinue) {
// backpressure, now we stop and we need to wait for drain
await once(destStream, 'drain')
// ok now it's safe to resume writing
}
}But if you are dealing with streaming pipelines it's probably easier to use pipeline().
import { pipeline } from 'stream/promises'
import { createReadStream, createWriteStream } from 'fs'
import { createBrotliCompress } from 'zlib'
const sourceStream = createReadStream('bigdata.csv')
const compress = createBrotliCompress()
const destStream = createWriteStream('bigdata.csv.br')
await pipeline(
sourceStream,
compress,
destStream
)In Node.js we can convert any Event Emitter to an Async Iterator! 😱
import { on } from 'events'
import glob from 'glob' // from npm
const matcher = glob('**/*.js')
for await (const [filePath] of on(matcher, 'match')) {
console.log(filePath)
}import { on } from 'events'
import glob from 'glob' // from npm
const matcher = glob('**/*.js')
for await (const [filePath] of on(matcher, 'match')) {
console.log(filePath)
}
// ⚠️ DANGER, DANGER (high voltage ⚡️): We'll never get here!
console.log('ALL DONE! :)')import { on } from 'events'
import glob from 'glob'
const matcher = glob('**/*.js')
const ac = new global.AbortController()
matcher.once('end', () => ac.abort())
try {
for await (const [filePath] of on(matcher, 'match', { signal: ac.signal })) {
console.log(`./${filePath}`)
}
} catch (err) {
if (!ac.signal.aborted) {
console.error(err)
process.exit(1)
}
// we ignore the AbortError
}
console.log('NOW WE GETTING HERE! :)') // YAY! 😻NOTE:
If you know ahead of time how many events you need to process you can also use a break in the for...await loop.
LAST TIP:
Can we use async iterators to handle web requests a-la-Deno? 🦕
import { createServer } from 'http'
import { on } from 'events'
const server = createServer()
server.listen(8000)
for await (const [req, res] of on(server, 'request')) {
res.end('hello dear friend')
}EASY PEASY LEMON SQUEEZY! 🍋
But... wait, aren't we processing all requests in series, now? 😱
import { createServer } from 'http'
import { on } from 'events'
const server = createServer()
server.listen(8000)
for await (const [req, res] of on(server, 'request')) {
// ... AS LONG AS WE DON'T USE await HERE, WE ARE FINE!
}import { createServer } from 'http'
import { on } from 'events'
import { setTimeout } from 'timers/promises'
const server = createServer()
server.listen(8000)
for await (const [req, res] of on(server, 'request')) {
await setTimeout(1000)
res.end('hello dear friend')
}
Let's stick to the basics... 😅
import { createServer } from 'http'
import { setTimeout } from 'timers/promises'
const server = createServer(async function (req, res) {
await setTimeout(1000)
res.end('hello dear friend')
})
server.listen(8000)
Conclusion 🤓
- Iterable protocols are a way to standardize iteration in JavaScript and Node.js
- Async iterators are ergonomic tools for sequential asynchronous iteration
- But don't use them for everything!
- Consuming data from paginated APIs or reading messages from a queue are good examples!
- Handling web requests or events from an emitter might not be the best use cases!
Want to learn more? 👩🏫
-
nodejsdesignpatterns.com - possibly a great book 😇
-
nodejsdesignpatterns.com/blog/javascript-async-iterators/ - In-depth article on all things iterators
-
loige.link/async-it - Finding a lost song with Node.js & async iterators
If you enjoyed this talk, you might also enjoy nodejsdp.link 😛
Reiterating JavaScript & Node.js iterators
By Luciano Mammino
Reiterating JavaScript & Node.js iterators
In this talk we will revisit the iteration protocols offered by JavaScript (iterable, iterator, async iterable and async iterator). We will explore generators and async generators as well. We will use this opportunity to discuss some interesting applications of these patterns in Node.js and we will compare them with more traditional approaches like Event Emitters and Node.js Streams. Finally, we will discuss some interesting integrations between traditional approaches and iterators and show some patterns and anti-patterns.
