Did you know
JavaScript has iterators?

Luciano Mammino (@loige)

loige.link/iter-garda

A new adventure awaits you!

Grab the slides!

loige.link/iter-garda

𝕏 loige

You just joined a new STARTUP.
This is DAY 1!

Grab the slides!

loige.link/iter-garda

𝕏 loige

CTO: Can you help me with troubleshooting something?

Grab the slides!

loige.link/iter-garda

𝕏 loige

CTO: I am debugging a PRODUCTION issue.

Grab the slides!

loige.link/iter-garda

𝕏 loige

CTO: We need to count how many "ERR_SYS_FCKD" there are per customer 

Grab the slides!

loige.link/iter-garda

𝕏 loige

{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}

{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}

{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}

{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5KGHRV3SNJ71C4E920872","error":"ERR_SYS_FCKD"}

{"requestId":"3e3e49b5...","level":"ERROR","timestamp":"2023-05-26T23:18:50.277Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_CLIENT_TIMEOUT"}

{"requestId":"f28abf98...","level":"ERROR","timestamp":"2023-05-26T23:18:56.575Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_BUSY"}

{"requestId":"608e579f...","level":"ERROR","timestamp":"2023-05-26T23:19:04.529Z","customerId":"01H1D5KSG0K81CCAGTSTBXHFA6","error":"ERR_NOT_FOUND"}

{"requestId":"3125588e...","level":"ERROR","timestamp":"2023-05-26T23:19:11.514Z","customerId":"01H1D5MJ4XNJ580DMM8Y7T1HRW","error":"ERR_SERVER_TIMEOUT"}

{"requestId":"40fb0329...","level":"INFO","timestamp":"2023-05-26T23:19:13.536Z","customerId":"01H1D5N6HS2EMA900A2V6NQQ2Q","message":"user logged in"}

{"requestId":"564afe31...","level":"ERROR","timestamp":"2023-05-26T23:19:21.708Z","customerId":"01H1D5K2JDVD941R6H67YEY0G8","error":"ERR_CLIENT_TIMEOUT"}

𝕏 loige

Let's do it!

Grab the slides!

loige.link/iter-garda

𝕏 loige

import { readFile } from 'node:fs/promises'

const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
  if (!acc[error.customerId]) {
    acc[error.customerId] = 0
  }
  acc[error.customerId]++
  return acc
}, {})

console.log('Errors by customer:')
console.table(errorsByCustomer)

𝕏 loige

Errors by customer:

{
  '01H1D5NGA5D689HD57CN5BZSG3': 109,
  '01H1D5MZAVPSXSXPTYADF4BDND': 118,
  '01H1D5KGHRV3SNJ71C4E920872': 118,
  '01H1D5KSG0K81CCAGTSTBXHFA6': 95,
  '01H1D5MJ4XNJ580DMM8Y7T1HRW': 103,
  '01H1D5N6HS2EMA900A2V6NQQ2Q': 96,
  '01H1D5M5RTKKK5SC4ADWAPK7Q0': 130,
  '01H1D5K2JDVD941R6H67YEY0G8': 115,
  '01H1D5KZ7GV7HK213XE3WV5GQX': 113
}

𝕏 loige

CTO: Great job! Exactly what I needed! 

Grab the slides!

loige.link/iter-summit

𝕏 loige

DAY 2

Grab the slides!

loige.link/iter-garda

𝕏 loige

CTO: that script it's now giving "RangeError: Invalid string length"!

Grab the slides!

loige.link/iter-garda

𝕏 loige

import { readFile } from 'node:fs/promises'

const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
  if (!acc[error.customerId]) {
    acc[error.customerId] = 0
  }
  acc[error.customerId]++
  return acc
}, {})

console.log('Errors by customer:')
console.table(errorsByCustomer)

𝕏 loige

import { readFile } from 'node:fs/promises'

const rawData = await readFile('logs.jsonl', 'utf-8')
const lines = rawData.split(/\n+/)
const messages = lines.map(line => line === '' ? {} : JSON.parse(line))
const errors = messages.filter(message => message.error === 'ERR_SYS_FCKD')
const errorsByCustomer = errors.reduce((acc, error) => {
  if (!acc[error.customerId]) {
    acc[error.customerId] = 0
  }
  acc[error.customerId]++
  return acc
}, {})

console.log('Errors by customer:')
console.table(errorsByCustomer)

We are loading everything into memory...

If the file is too big this will fail!

𝕏 loige

Ideal approach

Lazy processing, line by line

Current line:

Current state:

Parse
Filter
Aggregate

//
{
  
}

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  
}

OK

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"d2328daa...","level":"ERROR","timestamp":"2023-05-26T23:18:24.834Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"c4fe5fdd...","level":"INFO","timestamp":"2023-05-26T23:18:29.966Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","message":"transaction in progress"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1
}

NOPE

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1
}

OK

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"69733d2e...","level":"ERROR","timestamp":"2023-05-26T23:18:33.570Z","customerId":"01H1D5MZAVPSXSXPTYADF4BDND","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1,
  "01H1D5MZAVPSXSXPTYADF4BDND": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1,
  "01H1D5MZAVPSXSXPTYADF4BDND": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 1,
  "01H1D5MZAVPSXSXPTYADF4BDND": 1
}

OK

Ideal approach

Lazy processing, line by line

𝕏 loige

Current line:

Current state:

Parse
Filter
Aggregate

{"requestId":"2845fc48...","level":"ERROR","timestamp":"2023-05-26T23:18:40.489Z","customerId":"01H1D5NGA5D689HD57CN5BZSG3","error":"ERR_SYS_FCKD"}
{
  "01H1D5NGA5D689HD57CN5BZSG3": 2,
  "01H1D5MZAVPSXSXPTYADF4BDND": 1
}

Ideal approach

Lazy processing, line by line

𝕏 loige

We can do this using Iterators!

𝕏 loige

Player 1

I'm Luciano

Senior Architect @ fourTheorem (Dublin)

nodejsdp.link

Co-Author of Node.js Design Patterns

Let's connect!

  loige.co (blog)

  @loige (X)

  loige (twitch)

  lmammino (github)

  linktr.ee/loige

Always re-imagining

We are a pioneering technology consultancy focused on Cloud, AWS & serverless

fth.link/careers

We can help with:

Cloud Migrations

Training & Cloud enablement

Building serverless applications

Cutting cloud costs

𝕏 loige

Why Iterators?

  • Lazy abstraction to represent repetition

  • You can consume the collection 1 item at the time

  • Great for large (or endless) datasets

𝕏 loige

The concept of Iterators exists already in JavaScript...

Good news...

... since ES2015!

𝕏 loige

const array = ['foo', 'bar', 'baz']

for (const item of array) {
  console.log(item)
}

 

Prints all the items in the array!

Does it need to be an array? 🤔

Output:
foo
bar
baz

𝕏 loige

const str = 'foo'

for (const item of str) {
  console.log(item)
}
Output:
f
o
o

𝕏 loige

const obj = {
  foo: 'bar',
  baz: 'qux'
}

for (const item of obj) {
  console.log(item)
}
Output:
⛔️ Uncaught TypeError: obj is not iterable

OMG `for ... of`

does not work with plain objects! 😱

𝕏 loige

𝕏 loige

const array = ['foo', 'bar', 'baz']

console.log(...array)
Output:
foo bar baz

spread syntax!

𝕏 loige

𝕏 loige

𝕏 loige

Generators

𝕏 loige

Generator fn & obj

function * myGenerator () {
  // generator body
  yield 'someValue'
  // ... do more stuff
}
const genObj = myGenerator()
genObj.next() // -> { done: false, value: 'someValue' }

𝕏 loige

function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
console.log(fruitGenObj.next()) // { value: '🍑', done: false }
console.log(fruitGenObj.next()) // { value: '🍉', done: false }
console.log(fruitGenObj.next()) // { value: '🍋', done: false }
console.log(fruitGenObj.next()) // { value: '🥭', done: false }
console.log(fruitGenObj.next()) // { value: undefined, done: true }

𝕏 loige

function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
// generator objects are iterable!
for (const fruit of fruitGenObj) {
  console.log(fruit)
}

// 🍑
// 🍉
// 🍋
// 🥭

𝕏 loige

Iterators & Iterables

𝕏 loige

Iterator Obj

An object that acts like a cursor to iterate over blocks of data sequentially

𝕏 loige

Iterable Obj

An object that provides data that can be iterated over sequentially

𝕏 loige

The iterator protocol

Iterator object:

 

next() method returns:

  • done (boolean)

  • value (any)

const iterator = {
  next () {
    return { 
      done: false,
      value: "someValue"
    }
  }
}

𝕏 loige

function createCountdown (from) {
  let nextVal = from
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }

      return { 
        done: false,
        value: nextVal--
      }
    }
  }
}

A factory function that creates an iterator

returns an object

... which has a next() method

... which returns an object with

done & value

𝕏 loige

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true }

𝕏 loige

function createCountdown (from) {
  let nextVal = from
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }

      return { 
        done: false,
        value: nextVal--
      }
    }
  }
}

Generator objects implement the iterator protocol

function * createCountdown (from) {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

Equivalent code using generators

𝕏 loige

The iterable protocol

Iterable object:

 

Symbol.iterator() method returns:

  • an iterator

const iterable = {
  [Symbol.iterator] () {
    return { 
      // iterator
      next () {
        return { 
          done: false,
          value: "someValue"
        }
      }
    }
  }
}

𝕏 loige

function createCountdown (from) {
  let nextVal = from
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }

        return { done: false, value: nextVal-- }
      }
    })
  }
}

𝕏 loige

[...createCountdown(3)] // [3,2,1,0]

for (const value of createCountdown(3)) {
  console.log(value)
}

// 3
// 2
// 1
// 0

We can use spread and for ... of with iterable objects!

𝕏 loige

function createCountdown (from) {
  let nextVal = from
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }

        return {
          done: false, 
          value: nextVal--
        }
      }
    })
  }
}

Generator objects implement the iterable protocol

function * createCountdown (from) {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

Equivalent code using generators

YES, it's the same code as the iterator example!

𝕏 loige

const iterableIterator = {
  next () {
    return { done: false, value: 'hello' }
  },
  [Symbol.iterator] () {
    return this
  }
}

An object can be both an iterator and an iterable!

Iterator protocol

Iterable protocol

𝕏 loige

ASYNC
Iterators & Iterables

𝕏 loige

The async iterator protocol

Async Iterator object:

 

next() method returns a

Promise that resolves to:

  • done (boolean)

  • value (any)

const asyncIterator = {
  async next () {
    return { 
      done: false,
      value: "someValue"
    }
  }
}

𝕏 loige

The async iterable protocol

Async Iterable object:

 

Symbol.asyncIterator() method returns:

  • an async iterator

const asyncIterable = {
  [Symbol.asyncIterator] () {
    return { 
      // iterator
      async next () {
        return { 
          done: false,
          value: "someValue"
        }
      }
    }
  }
}

𝕏 loige

import { setTimeout } from 'node:timers/promises'

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

Async Generator objects implement the async iterator & async iterable protocols

𝕏 loige

const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value)
}

We can use for await ... of with async iterable objects!

𝕏 loige

Let's rewrite our log analyser to use iterators!

𝕏 loige

import { createReadStream } from 'node:fs'

const readable = createReadStream(
  'logs.jsonl',
  { encoding: 'utf-8' }
)

// a readable stream is an Async Iterable!
for await (const chunk of readable) {
  // do something with a chunk of data
}

We can read a file incrementally using Node.js streams!

𝕏 loige

import { createReadStream } from 'node:fs'

const readable = createReadStream(
  'logs.jsonl',
  { encoding: 'utf-8' }
)

// a readable stream is an Async Iterable!
for await (const chunk of readable) {
  // do something with a chunk of data
}

We can read a file incrementally using Node.js streams!

A chunk is an arbitrary amount of data (not necessarily a line)

𝕏 loige

// utils/byline.js

export async function * byLine (asyncIterable) {
  let remainder = ''
  for await (const chunk of asyncIterable) {
    const lines = (remainder + chunk).split(/\n+/)
    remainder = lines.pop()
    yield * lines
  }
  if (remainder.length > 0) {
    yield remainder
  }
}

𝕏 loige

import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'

const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })

const errorsByCustomer = {}

for await (const line of byLine(readable)) {
  const message = line === '' ? {} : JSON.parse(line)
  if (message.error === 'ERR_SYS_FCKD') {
    if (!errorsByCustomer[message.customerId]) {
      errorsByCustomer[message.customerId] = 0
    }
    errorsByCustomer[message.customerId]++
  }
}

console.log('Errors by customer:')
console.log(errorsByCustomer)

𝕏 loige

import { createReadStream } from 'node:fs'
import { byLine } from './utils/byline.js'

const readable = createReadStream('logs.jsonl', { encoding: 'utf-8' })

const errorsByCustomer = {}

for await (const line of byLine(readable)) {
  const message = line === '' ? {} : JSON.parse(line)
  if (message.error === 'ERR_SYS_FCKD') {
    if (!errorsByCustomer[message.customerId]) {
      errorsByCustomer[message.customerId] = 0
    }
    errorsByCustomer[message.customerId]++
  }
}

console.log('Errors by customer:')
console.log(errorsByCustomer)

CTO: It works with

any file now! 

𝕏 loige

Can't we use .map(), .filter(), .reduce()?

𝕏 loige

𝕏 loige

if you need to read a file line by line, there's another option:

loige.link/line-by-line

BONUS Material

loige.link/streams-wrk

loige.link/iter-wrk

SLIDES: loige.link/iter-garda

Code: loige.link/iter-garda-code

𝕏 loige

Original Front cover photo by Jörg Angeli on Unsplash

Original Background photo by Michael Behrens on Unsplash

Grazie!

nodejsdp.link

loige.link/iter-garda

𝕏 loige

Did you know JavaScript has iterators? - Node.js Conference Italy

By Luciano Mammino

Did you know JavaScript has iterators? - Node.js Conference Italy

How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for…in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!

  • 1,595

More from Luciano Mammino