Reiterating JavaScript & Node.js iterators

Luciano Mammino (@loige)

RomaJS - June 16, 2021

How many ways do you know to "do iteration" in JavaScript & Node.js? 😰

for i / for...in / for...of
while / do while
Array.forEach / Array.map / Array.flatMap / Array.reduce / Array.reduceRight  / Array.filter / Array.find / Array.findIndex / Array.entries / Array.values / Array.every / Array.some
Object.keys / Object.values / Object.entries
Iterators / Generators
 Spread operator

[...iterable]

Events / Streams

Async iterators / Async generators

for await...of

That was 28 different concepts! 😳

📝 AGENDA

  • Iteration protocols... What? Why?
  • Syntax review
  • Iteration protocols
    • Iterator protocol
    • Iterable protocol
    • Generator functions
    • Async iterator protocol
    • Async iterable protocol
  • Tips & tricks

Let me introduce myself...

I'm Luciano (🇮🇹🍕🍝) 👋

👨‍💻  Senior Architect

Co-Author of Node.js Design Patterns  👉

Connect with me:
 

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

We are business focused technologists that deliver.


Accelerated Serverless | AI as a Service | Platform Modernisation

WE ARE HIRING: Do you want to work with us?

Iteration protocols

what? why? 🤔

  • An attempt at standardizing "iteration" behaviors providing a consistent and interoperable API
    • Use for...of, for await...of and spread operator.
  • You can also create lazy iterators
  • You can also deal with async iteration
  • You can create your own custom iterators/iterables

Syntax Review 🧐

const judokas = [
  'Driulis Gonzalez Morales',
  'Ilias Iliadis',
  'Tadahiro Nomura',
  'Anton Geesink',
  'Teddy Riner',
  'Ryoko Tani'
]

for (const judoka of judokas) {
  console.log(judoka)
}

for...of

Driulis Gonzalez Morales
Ilias Iliadis
Tadahiro Nomura
Anton Geesink
Teddy Riner
Ryoko Tani
OUTPUT
const judoka = 'Ryoko Tani'

for (const char of judoka) {
  console.log(char)
}

for...of (with strings)

R
y
o
k
o 

T
a
n
i
OUTPUT
const medals = new Set([
  'gold',
  'silver',
  'bronze'
])

for (const medal of medals) {
  console.log(medal)
}

for...of (with Set)

gold
silver
bronze

 

OUTPUT
const medallists = new Map([
  ['Teddy Riner', 33],
  ['Driulis Gonzalez Morales', 16],
  ['Ryoko Tani', 16],
  ['Ilias Iliadis', 15]
])

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with Map)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of Object.entries(medallists)) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with object & Object.entries)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with object literals)

for (const [judoka, medals] of medallists) {
                              ^
TypeError: medallists is not iterable
  at Object. (.../05-for-of-object.js:8:32)
ERROR
const countdown = [3, 2, 1, 0]

// spread into array
const from5to0 = [5, 4, ...countdown]
console.log(from5to0) // [ 5, 4, 3, 2, 1, 0 ]

// spread function arguments
console.log('countdown data:', ...countdown)
// countdown data: 3 2 1 0

Spread operator

import {
  DynamoDBClient,
  paginateListTables
} from '@aws-sdk/client-dynamodb'

const client = new DynamoDBClient({});

for await (const page of paginateListTables({ client }, {})) {
  // page.TableNames is an array of table names
  for (const tableName of page.TableNames) {
    console.log(tableName)
  }
}

for await...of (async iterable)

Iteration Protocols 😵‍💫

Iterator protocol

In JavaScript, an object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

function createCountdown (start) {
  let nextVal = start
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }
      return {
        done: false,
        value: nextVal--
      }
    }
  }
}

Countdown iterator

const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Iterable protocol

An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.

 

* Symbol.iterator

function createCountdown (start) {
  let nextVal = start
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }
        return {
          done: false, 
          value: nextVal--
        }
      }
    })
  }
}

Countdown iterable

const countdown = createCountdown(3)
for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

Can an object be both an iterator and an iterable?! 🤨

const iterableIterator = {
  next() {
    return { done: false, value: "hello" }
  },
  [Symbol.iterator]() {
    return this
  }
}

Iterator + Iterable

iterableIterator.next()
// { done: false, value: "hello" }

for (const value of iterableIterator) {
 console.log(value)
}

// hello
// hello
// hello
// ...

Generators

A generator function "produces" an object that is both an iterator and an iterable! 🤯

function * createCountdown (start) {
  for (let i = start; i >= 0; i--) {
    yield i
  }
}
// As iterable
const countdown = createCountdown(3)
for (const value of countdown) {
 console.log(value)
}
// 3
// 2
// 1
// 0
// As iterator
const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Well, what about async iteration? 🤌

Async Iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  let nextVal = start
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }
      return { done: false, value: nextVal-- }
    }
  }
}
const countdown = createAsyncCountdown(3)
console.log(await countdown.next()) // { done: false, value: 3 }
console.log(await countdown.next()) // { done: false, value: 2 }
console.log(await countdown.next()) // { done: false, value: 1 }
console.log(await countdown.next()) // { done: false, value: 0 }
console.log(await countdown.next()) // { done: true }

Async Iterable protocol

An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.

 

* Symbol.asyncIterator

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  return {
    [Symbol.asyncIterator]: function () {
      let nextVal = start
      return {
        async next () {
          await setTimeout(delay)
          if (nextVal < 0) {
            return { done: true }
          }
          return { done: false, value: nextVal-- }
        }
      }
    }
  }
}
const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}
import { setTimeout } from 'timers/promises'

async function * createAsyncCountdown (start, delay = 1000) {
  for (let i = start; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}
const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}

When to use async iterators

  • Sequential iteration pattern

  • Data arriving in order over time

  • You need to complete processing the current “chunk” before you can request the next one

  • Examples

    • paginated iteration

    • consuming tasks from a remote queue

Tips & pitfalls ☠️

import { createReadStream } from 'fs'

const sourceStream = createReadStream('bigdata.csv')

let bytes = 0
for await (const chunk of sourceStream) {
  bytes += chunk.length
}

console.log(`bigdata.csv: ${bytes} bytes`)

Node.js readable streams are async iterators

What about
backpressure? 😩

import { createReadStream } from 'fs'
import { once } from 'events'

const sourceStream = createReadStream('bigdata.csv')
const destStream = new SlowTransform()

for await (const chunk of sourceStream) {
  const canContinue = destStream.write(chunk)
  if (!canContinue) {
    // backpressure, now we stop and we need to wait for drain
    await once(destStream, 'drain')
    // ok now it's safe to resume writing
  }
}

But if you are dealing with streaming pipelines it's probably easier to use pipeline().

import { pipeline } from 'stream/promises'
import { createReadStream, createWriteStream } from 'fs'
import { createBrotliCompress } from 'zlib'

const sourceStream = createReadStream('bigdata.csv')
const compress = createBrotliCompress()
const destStream = createWriteStream('bigdata.csv.br')

await pipeline(
  sourceStream,
  compress,
  destStream
)

In Node.js we can convert any Event Emitter to an Async Iterator! 😱

import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}
import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

// ⚠️  DANGER, DANGER (high voltage ⚡️): We'll never get here!
console.log('ALL DONE! :)')
import { on } from 'events'
import glob from 'glob'

const matcher = glob('**/*.js')
const ac = new global.AbortController()

matcher.once('end', () => ac.abort())

try {
  for await (const [filePath] of on(matcher, 'match', { signal: ac.signal })) {
    console.log(`./${filePath}`)
  }
} catch (err) {
  if (!ac.signal.aborted) {
    console.error(err)
    process.exit(1)
  }
  // we ignore the AbortError
}

console.log('NOW WE GETTING HERE! :)') // YAY! 😻

NOTE:
If you know ahead of time how many events you need to process you can also use a break in the for...await loop.

LAST TIP:

Can we use async iterators to handle web requests a-la-Deno? 🦕

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  res.end('hello dear friend')
}

EASY PEASY LEMON SQUEEZY! 🍋

 

But... wait, aren't we processing all requests in series, now? 😱

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  // ... AS LONG AS WE DON'T USE await HERE, WE ARE FINE!
}
import { createServer } from 'http'
import { on } from 'events'
import { setTimeout } from 'timers/promises'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  await setTimeout(1000)
  res.end('hello dear friend')
}

Let's stick to the basics... 😅

import { createServer } from 'http'
import { setTimeout } from 'timers/promises'

const server = createServer(async function (req, res) {
  await setTimeout(1000)
  res.end('hello dear friend')
})

server.listen(8000)

Conclusion 🤓

  • Iterable protocols are a way to standardize iteration in JavaScript and Node.js
  • Async iterators are ergonomic tools for sequential asynchronous iteration
  • But don't use them for everything!
    • Consuming data from paginated APIs or reading messages from a queue are good examples!
    • Handling web requests or events from an emitter might not be the best use cases!

Want to learn more? 👩‍🏫

If you enjoyed this talk, you might also enjoy nodejsdp.link 😛

Let's connect:

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)