Luciano Mammino PRO
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com
Finding a lost song
with Node.js & async iterators
Luciano Mammino (@loige)
2021-09-30
Get these slides!
Photo by Darius Bashar on Unsplash
A random song you haven't listened to
in years pops into your head...
The Ohrwurm! 👂🐛
It doesn't matter what you do all day...
It keeps coming back to you! 🐛
Photo by Attentie Attentie on Unsplash
And now you want to listen to it!
Photo by Volodymyr Hryshchenko on Unsplash
But, what if you can't remember
the title or the author?!
Photo by Tachina Lee on Unsplash
THERE MUST BE A WAY TO REMEMBER!
Photo by Marius Niveri on Unsplash
Today, I'll tell you how I solved this problem using
- Last.fm API
- Node.js
- Async Iterators
Photo by Quinton Coetzee on Unsplash
Let me introduce myself first...
I'm Luciano (🇮🇹🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨💻
Co-Author of Node.js Design Patterns 👉
We are business focused technologists that deliver.
Accelerated Serverless | AI as a Service | Platform Modernisation
We are hiring: do you want to work with us?
So, there was this ohrwurm... 🐛
I could only remember some random parts and the word "dark" (probably in the title)
Luciano - scrobbling since 12 Feb 2007
~250k scrobbles... that song must be there!
~5k pages of history &
no search functionality! 😓
But there's an API!
https://www.last.fm/api
Let's give it a shot
curl "http://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&user=loige&api_key=${API_KEY}&format=json" | jq .
It works! 🥳
Now let's do this with JavaScript
import querystring from 'querystring'
import axios from 'axios'
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json'
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
We are getting a "paginated" response with 50 tracks per page
but there are 51 here! 🤔
How do we fetch the next pages?
(let's ignore this for now...)
let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
page++
}
Seems good!
Let's look at the tracks...
// ...
for (const track of response.data.recenttracks.track) {
console.log(
track.date?.['#text'],
`${track.artist['#text']} - ${track.name}`
)
}
console.log('--- end page ---')
// ...
* Note that page size here is 10 tracks per page
Every page has a song with undefined time...
This is the song I am currently listening to!
It appears at the top of every page.
Sometimes there are duplicated tracks between pages... 😨
The "sliding windows" problem 😩
...
tracks (newest to oldest)
Page1
Page2
...
Page1
Page2
new track
moved from page 1 to page 2
Time based windows 😎
...*
tracks (newest to oldest)
Page1
before t1
(page 1 "to" t1)
t1
t2
before t2
(page 1 "to" t2)
* we are done when we get an empty page (or num pages is 1)
to ... from
let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
}
The track of the last timestamp becomes the boundary for the next page
We have a working solution! 🎉
Can we generalise it?
We know how to iterate over every page/track.
How do we expose this information?
const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// callbacks
reader.readPages(
(page) => { /* ... */ }, // on page
(err) => { /* ... */} // on completion (or error)
)
const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// event emitter
reader.read()
reader.on('page', (page) => { /* ... */ })
reader.on('completed', (err) => { /* ... */ })
const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// streams <3
reader.pipe(/* transform or writable stream here */)
reader.on('end', () => { /* ... */ })
reader.on('error', () => { /* ... */ })
import { pipeline } from 'stream'
const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// streams pipeline <3 <3
pipeline(
reader,
yourProcessingStream,
(err) => {
// handle completion or err
}
)const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// ASYNC ITERATORS!
for await (const page of reader) {
/* ... */
}
// ... do more stuff when all the data is consumedconst reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// ASYNC ITERATORS WITH ERROR HANDLING!
try {
for await (const page of reader) {
/* ... */
}
} catch (err) {
// handle errors
}
// ... do more stuff when all the data is consumedHow can we build an async iterator? 🧐
Meet the iteration protocols!
The iterator protocol
An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.
function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
}const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true }Generator functions "produce" iterators!
function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true, value: undefined }The iterable protocol
An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.
*Symbol.iterator
function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}function createCountdown (from) {
return {
[Symbol.iterator]: function * () {
for (let i = from; i >= 0; i--) {
yield i
}
}
}
}const countdown = createCountdown(3)
for (const value of countdown) {
console.log(value)
}
// 3
// 2
// 1
// 0OK. So far this is all synchronous iteration.
What about async? 🙄
The async iterator protocol
An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.
import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
let nextVal = from
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
}const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }
console.log(await countdown.next())
// { done: false, value: 2 }
console.log(await countdown.next())
// { done: false, value: 1 }
console.log(await countdown.next())
// { done: false, value: 0 }
console.log(await countdown.next())
// { done: true }
import { setTimeout } from 'timers/promises'
// async generators "produce" async iterators!
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}The async iterable protocol
An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.
*Symbol.asyncIterator
import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
return {
[Symbol.asyncIterator]: async function * () {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
}
}HOT TIP 🔥
With async generators we can create objects that are both async iterators and async iterables!
(We don't need to specify Symbol.asyncIterator explicitly!)
import { setTimeout } from 'timers/promises'
// async generators "produce" async iterators
// (and iterables!)
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value)
}
Now we know how to make our LastFmRecentTracks an Async Iterable 🤩
import querystring from 'querystring'
import axios from 'axios'
async function * createLastFmRecentTracks (apiKey, user) {
let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user,
api_key: apiKey,
format: 'json',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
yield tracks
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
}
}const recentTracks = createLastFmRecentTracks(
process.env.API_KEY,
'loige'
)
for await (const page of recentTracks) {
console.log(page)
}
Let's search for all the songs that contain the word "dark" in their title! 🧐
async function main () {
const recentTracks = createLastFmRecentTracks(
process.env.API_KEY,
'loige'
)
for await (const page of recentTracks) {
for (const track of page) {
if (track.name.toLowerCase().includes('dark')) {
console.log(`${track.artist['#text']} - ${track.name}`)
}
}
}
}
OMG! This is the song! 😱
...from 8 years ago!
For a more serious package that allows you to fetch data from Last.fm:
npm install scrobbles
for await (const _ of createAsyncCountdown(1_000_000)) {
console.log("THANK YOU! 😍")
}Finding a lost song with Node.js and async iterators - EnterJS 2021
By Luciano Mammino
Finding a lost song with Node.js and async iterators - EnterJS 2021
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
