Do You Want to Build a

Build Tool?

Do You Want to Build a

Build Tool?

Hi Everyone, I'm Craig !

twitter/phenomnominal
github/phenomnomnominal

@phenomnominal 2020

Today, I'm going to tell you a fairy tale!

Today, I'm going to tell you a Fairy Tale!

@phenomnominal 2020

Ready?

@phenomnominal 2020

It was Summer in the Kingdom of Arendelle!

@phenomnominal 2020

And Princess Anna was So Excited!

@phenomnominal 2020

The day had finally arrived!

The day had finally arrived!

@phenomnominal 2020

Queen Elsa, Anna's sister and Best friend

@phenomnominal 2020

Had finally agreed to share her secrets!

@phenomnominal 2020

Anna, and her friends Kristoff, Sven and Olaf

@phenomnominal 2020

Had been working hard all winter

@phenomnominal 2020

And now it was time for their reward!

@phenomnominal 2020

Elsa has Amazing powers!

Elsa has Amazing powers!

@phenomnominal 2020

She can sense the water molecules around us,

@phenomnominal 2020

transform those water molecules into ice,

@phenomnominal 2020

And control that ice however she likes!

@phenomnominal 2020

today, Elsa IS going to share her magic!

today, Elsa is going to share her magic!

@phenomnominal 2020

Chapter One

Just like magic

"Anna, do you remember when we went on our trip into the internet?",

asked Elsa.

@phenomnominal 2020

"Of course!",

"We met all the other princesses, it was so much fun!"

replied Anna.

@phenomnominal 2020

"There was that cute, new little princess, Vanellope!"

@phenomnominal 2020

"And her friend Ralph was in trouble!"

@phenomnominal 2020

"And we helped save him!"

@phenomnominal 2020

"That's right! It was such a fun trip!"

"What else do you remember about the internet?"

@phenomnominal 2020

"There was so much to do, so much to see!",

"So much data flying around! l learned all about how to make websites!"

exclaimed Anna.

" l loved it so much l even taught Sven when l got back!"

@phenomnominal 2020

SLURP!

, Sven confirmed.

@phenomnominal 2020

"l learned to write HTML,

and CSS,

and JavaScript!

"lt was just like magic!"

@phenomnominal 2020

It was

just

like

magic

@phenomnominal 2020

Chapter Two

Sufficiently incredible

Anna, Kristoff, Olaf, and Sven waited patiently for Elsa to continue.

@phenomnominal 2020

"I'm going to let you in on my little secret",

Elsa began.

@phenomnominal 2020

"You may have heard that any sufficiently advanced technology is indistinguishable from magic,"

"but actually, its the other way around..."

@phenomnominal 2020

"Any sufficiently incredible magic, is usually actually technology!"

@phenomnominal 2020

"Wow! That's amazing! I think?"

Kristoff was already blown away by this.

@phenomnominal 2020

"I'm just getting started!"

"To begin, we need to talk about data..."

said Elsa, with a grin.

@phenomnominal 2020

"The whole internet is full of data."

"Pictures"

"Memes"

"Infographics"

@phenomnominal 2020

{
  today: {
      temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
  },
  future: [
    { high: 21, low: 13, chanceOfSnow: 0.05 },
    { high: 21, low: 14, chanceOfSnow: 0.03 },
    { high: 21, low: 16, chanceOfSnow: 0.04 },
    { high: 19, low: 16, chanceOfSnow: 0.04 },
    /* ... */
  ]
}

"And that data is typically structured in some way, like this JSON file"

@phenomnominal 2020

"Or like this HTML file"

<h2>Weather result</h2>
<h3>
    <p>Arendelle, Norway</p>
    <p>Sunday 19:00</p>
    <span>Sunny</span>
</h3>
<img alt="Sunny" src="//arendelle.info/weather/64/sunny.png">
<h3>
  19<button aria-label="°Celsius">°C</button>
</h3>
<p>Precipitation: <span>0%</span></p>
<p>Humidity: <span>75%</span></p>
<p>Wind: <span>1 m/s</span></p>
<ol>
    <li>
        <p aria-label="Sunday">Sun</p>
        <img alt="Clear" src="//arendelle.info/weather/48/sunny.png">
        <span>21</span>°
        <span>13</span>°
    </li>
    <!-- ... -->
</ol>

@phenomnominal 2020

"Web developers use programming languages to inspect and manipulate that data"

const response = await fetch('https://arendelle.info/weather.json');
const data = await response.json();

// {
//   today: {
//       temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
//   },
//   future: [
//     { high: 21, low: 13, chanceOfSnow: 0.05 },
//     { high: 21, low: 14, chanceOfSnow: 0.03 },
//     { high: 21, low: 16, chanceOfSnow: 0.04 },
//     { high: 19, low: 16, chanceOfSnow: 0.04 },
//     /* ... */
//   ]
// }

const SATURDAY = 6;
const today = new Date().getDay();
const chanceOfSnowOnSaturday = data.future[SATURDAY - today].chanceOfSnow * 100;

h1 {
  // ...
}

h3 > p {
  // ... 
}

button:active,
button:hover {
  // ...
}

li:first-child {
  // ...
}

"Languages like JavaScript"

"or CSS"

@phenomnominal 2020

"Olaf, what exactly do you imagine when you think about snow?"

@phenomnominal 2020

"I think about skiing, and sledding, and snow fights!"

"It's so bright, and so cold, and so hard, but also so soft!"

"What do you think about Elsa?",

he asked.

@phenomnominal 2020

"I think about all those things too!"

"But my powers allow me to see another level deeper as well..."

"Going outside in the cold, having fun with our friends, ice skating..."

@phenomnominal 2020

"I see the whole system."

"In that system, each water molecule is a little bit of data, following the set of instructions that nature gave it."

@phenomnominal 2020

"Those instructions tell the water molecules when to freeze. "

"Those instructions tell the molecules which way to go when the wind blows."

"I know how to control those instructions."

@phenomnominal 2020

"l've worked it out!",

"You can understand the snow's source code!"

Anna announced suddenly.

@phenomnominal 2020

"Exactly!",

Esla grinned again.

"And do you want to know the best bit?"

@phenomnominal 2020

"The code that contains the instructions for all the snow in the world is written in JavaScript!"

"I can show you how to query code, modify code, and even create code from nothing!!"

"What do you think?"

@phenomnominal 2020

Chapter Three

Chapter THREE

The Snow Code

Olaf was so excited that he could barely stop himself from bursting into song!

@phenomnominal 2020

"Will you teach us to use the snow code, Elsa?"

"Please, please, please, please, PLEASE!"

, he pleaded.

@phenomnominal 2020

"Of course I will!"

, Elsa reassured him.

@phenomnominal 2020

"But first, I need to teach you a few tricks!"

@phenomnominal 2020

"My gifts let me access the snow code directly!"

"You will need to use a slightly more direct approach..."

"Kristoff, have you heard of node.js?"

@phenomnominal 2020

"Yes!"

Kristoff looked pretty proud of himself.

"Anna taught me all about it."

@phenomnominal 2020

"Node.js is a computer program that lets us run JavaScript without a browser!"

@phenomnominal 2020

"We can use it to do things like read and write files, find out stuff about the system, and manipulate data in lots of different ways!"

@phenomnominal 2020

"Exactly right! Let's start by looking at how we can read and write files using node.js"

@phenomnominal 2020

File System APIs

fs.watch

Node has functions that let us do all kinds of things with the file system:

you can check if you can access a file with
change file permissions with
copy a file from one place to another with
create a symlink with
or set up a file watcher with

fs.link

fs.copyFile

fs.chmod

fs.access

@phenomnominal 2020

File System APIs

The most important functions for us to learn today are:

fs.writeFile

fs.mkdir

fs.readFile

- read a file from a path
- make a directory at a path
- write a file to a path

@phenomnominal 2020

File System APIs

// snowflake.txt

                                ()
                                /\
                               //\\
                              <<  >>
                          ()   \\//   ()
                ()._____   /\   \\   /\   _____.()
                   \.--.\ //\\ //\\ //\\ /.--./
                    \\__\\/__\//__\//__\\/__//
                     '--/\\--//\--//\--/\\--'
                        \\\\///\\//\\\////
                    ()-= >>\\< <\\> >\\<< =-()
                        ////\\\//\\///\\\\
                     .--\\/--\//--/\\--/\\--.
                    //""/\\""//\""//\""//\""\\
                   /'--'/ \\// \\// \\// \'--'\
                 ()`"""`   \/   //   \/   `"""`()
                          ()   //\\   ()
                              <<  >>
                               \\//
                                \/
                                ()

Imagine that we have a file like this:

@phenomnominal 2020

import { promises as fs } from 'fs';

async function readSnowflake () {
  return fs.readFile('./snowflake.txt', 'utf8');
}

console.log(await readSnowflake());

File System APIs

We can use the Promise-based fs API to read the file, and print it:

@phenomnominal 2020

File System APIs

Which will give us something like this:

Weaseltown:dywbabt queenelsa$ node index.js

                              ()
                              /\
                             //\\
                            <<  >>
                        ()   \\//   ()
              ()._____   /\   \\   /\   _____.()
                 \.--.\ //\\ //\\ //\\ /.--./
                  \\__\\/__\//__\//__\\/__//
                   '--/\\--//\--//\--/\\--'
                      \\\\///\\//\\\////
                  ()-= >>\\< <\\> >\\<< =-()
                      ////\\\//\\///\\\\
                   .--\\/--\//--/\\--/\\--.
                  //""/\\""//\""//\""//\""\\
                 /'--'/ \\// \\// \\// \'--'\
               ()`"""`   \/   //   \/   `"""`()
                        ()   //\\   ()
                            <<  >>
                             \\//
                              \/
                              ()

@phenomnominal 2020

import { promises as fs } from 'fs';

async function readSnowflake () {
  return fs.readFile('./snowflake.txt', 'utf8');
}

async function saveSnowflake (data: string) {
  await fs.mkdir('./saved/snowflakes', { recursive: true });
  await fs.writeFile('./saved/snowflakes/snowflake.txt', data);
}

const snowflake = await readSnowflake();
await saveSnowflake(snowflake);

File System APIs

We can combine that with making a directory and writing the file to disk:

@phenomnominal 2020

Path APIs

It is also important to understand how the path API works:

path.resolve

- resolve a path relative to the given path
- split a path string into its constituent parts

path.parse

@phenomnominal 2020

import * as path from 'path';

const relativeToThisFile = path.resolve(__dirname, './snowflake.txt');
const parsed = path.parse(relativeToThisFile);

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

Path APIs

import { promises as fs } from 'fs';
import * as path from 'path';

export async function readSnowflake () {
  const relativePath = path.resolve(__dirname, './snowflake.txt');
  const relativePath = path.resolve(process.cwd(), './snowflake.txt');
  return fs.readFile(relativePath, 'utf8');
}

@phenomnominal 2020

, Sven snorted.

snort

@phenomnominal 2020

"That's right Sven! You must always test your file structure & path operations on all the different operating systems you care about! It's very easy to get it wrong!"

@phenomnominal 2020

"Now, we've seen how we can read a plain text file, what about something a bit different..."

@phenomnominal 2020

import { promises as fs } from 'fs';
import * as path from 'path';

async function readSnowflakes () {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const json = await fs.readFile(snowflakePath, 'utf8');
  return JSON.parse(json);
}

console.log(await readSnowflakes());

We can read different types of data from different files:

JSON data

@phenomnominal 2020

"Wow! So we took some data that was in a file, and parsed it into JavaScript objects that we can manipulate with code!"

Anna was really excited!

@phenomnominal 2020

"If you think that is cool, check this out..."

Elsa was just getting started!

@phenomnominal 2020

"If you think that is cool, check this out..."

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

Elsa was just getting started!

@phenomnominal 2020

"THE PATH TO THE SNOW CODE!"

Olaf instantly knew what it was!

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

@phenomnominal 2020

"Yes! The snow code is just another file!"

"It's just text and we can manipulate it and control it!"

@phenomnominal 2020

import { promises as fs } from 'fs';

export async function readSnowCode () {
  return fs.readFile(SNOW_CODE_PATH, 'utf8');
}

console.log(await readSnowCode());

Code as data

We don't need to treat a code file any differently:

@phenomnominal 2020

Weaseltown:dywbabt queenelsa$ node secret_snow_code_file_dont_touch.js 

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

Code as data

@phenomnominal 2020

"Now we have the snow code as a string, so we can change it however we want!"

"This is amazing!"

Kristoff chimed in:

@phenomnominal 2020

// kristoff.js

import { promises as fs } from 'fs';
import { readSnowCode } from './read-snow-code';

const code = await readSnowCode();
  
code += `; console.log('Kristoff is the best!');`
  
await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

"I'm going to make it always print out 'Kristoff is the best!'"

, yelled Kristoff.

@phenomnominal 2020

// olaf.js

import { promises as fs } from 'fs';

const code = await fs.readFile('./kristoffs-cool-new-snow-code.js');

code = code.replace(/Kristoff/, 'Olaf');

await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

"Not if I can help it!"

, replied Olaf

@phenomnominal 2020

// sven.js

import { promises as fs } from 'fs';

const code = await fs.readFile('./kristoffs-cool-new-snow-code.js');

code = code.replace(/Olaf/, 'Sven');
code = code.replace(/best!/, 'bestest!!!!');
code = code.replace(/console.log\((.*)\)/, 'console.log($1.toUpperCase())');

await fs.writeFile('./kristoffs-cool-new-snow-code.js', code);

Even Sven had his own ideas...

@phenomnominal 2020

"Hold on everyone, l think we're getting ahead of ourselves!"

"Anna, can you think of any reasons why working with the string directly might not be the best approach?"

@phenomnominal 2020

"Strings aren't exactly structured!"

"You're just changing a bunch of characters, and that could break easily!"

"Regular Expressions are hard to get right, and hard to understand later!"

"You have to fight with comments, and whitespace!"

@phenomnominal 2020

"So how can we get the snow code into a more useful structure?"

@phenomnominal 2020

import * as path from 'path';

const parsed = path.parse('/some/path/to/some/file.txt');

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

console.log(parsed.ext);

// read-snowflakes.js

import { promises as fs } from 'fs';
import * as path from 'path';

export async function readSnowflakes () {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const data = await fs.readFile(snowflakePath, 'utf8');
  const snowflakes = JSON.parse(data);
  
  console.log(snowFlakes[0].velocity.x)
}

Parsing

Let's look back at how we handled the other strings:

@phenomnominal 2020

import { parseScript } from 'esprima';
import { readSnowCode } from './read-snow-code';

const code = await readSnowCode();
const ast = parseScript(code);
console.log(ast);

Parsing

We also need to parse our string of code!

@phenomnominal 2020

"An ast, I LOVE IT!"

"What is an ast?"

@phenomnominal 2020

"Good try Olaf..."

"It's actually an A S T "

"It's an Abstract Syntax Tree..."

@phenomnominal 2020

Chapter Four

The root of all problems

"Oh yeah of course, I know all about those kinds of trees..."

"But I think Sven needs a bit of a refresher..."

Kristoff sounded uncertain.

@phenomnominal 2020

"Sure! Let's start with a literal meaning."

@phenomnominal 2020

Abstract Syntax Trees

Tree

data structure made up of vertices and edges without any cycles

SYNTAX

the way in which linguistic elements are put together

Abstract

Not associated with any specific instance

Abstract Syntax Trees

@phenomnominal 2020

????

@phenomnominal 2020

Abstract Syntax Trees

An Abstract Syntax Tree is a data structure that represents the structure of code, but without any actual syntax.

let it = go('let it go');

it = go 'let it go'

it <= go ['let it go']

'let it go' -> go => it

Abstract Syntax Trees

@phenomnominal 2020

{
  type: 'Program',
  body: [{
    type: 'VariableDeclaration',
    declarations: [{
      type: 'VariableDeclarator',
      id: {
        type: 'Identifier',
        name: 'it'
      },
      init: {
        type: 'CallExpression',
        callee: {
          type: 'Identifier',
          name: 'go'
        },
        arguments: [{
          type: Literal',
          value: 'let it go'
        }]
      }
    }],
    kind: 'let'
  }]
}

@phenomnominal 2020

            'let it go'

Literal

Identifier

CallExpression

Identifier

VariableDeclarator

VariableDeclaration

go

         go('let it go')

it

    it = go('let it go')

let it = go('let it go');

@phenomnominal 2020

Identifier

IfStatement

ImportDeclaration

Literal

CallExpression

"Learning these abstract names and this new data structure is tricky, but there are tools to help!"

ClassDeclaration

FunctionDeclaration

VariableDeclaration

ExpressionStatement

AwaitExpression

ForStatement

WhileStatement

DoWhileStatement

@phenomnominal 2020

"AST Explorer lets you look at the parser output for a huge number of languages!"

"It is very useful for seeing the structure of the tree and what properties are present!"

AST Explorer

@phenomnominal 2020

import { parseScript } from 'esprima';

const code = `let it = go('let it go');`
const ast = parseScript(code);

const arg = ast.body[0].declarations[0].init.arguments[0];

console.log(arg);

"Seeing the whole tree helps to write out long complex property access chains"

@phenomnominal 2020

"So Anna, what do you think? Is this better than the string version?"

@phenomnominal 2020

"l think so!"

"We have a real structure now with the AST"

"We access data exactly like we would with JSON, so it's easier to understand"

"We don't have to think about comments, or whitespace, we just access what we want to!"

@phenomnominal 2020

"lt's still kind of clunky though..."

"Do you think we can do better?",

"And it will throw an error if the structure doesn't line up with the code!"

Elsa asked.

@phenomnominal 2020

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

"Let's look at another data structure..."

@phenomnominal 2020

"Olaf, what do you think this data structure represents?"

@phenomnominal 2020

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

@phenomnominal 2020

"IT LOOKS LIKE HTML!"

@phenomnominal 2020

"That's an approximation of what the browser creates when it downloads an HTML file and parses it!"

"Very good!"

"We call it the Document Object Model, or the DOM."

"It is also a tree!"

@phenomnominal 2020

"Now, wouldn't it be strange if we queried the DOM like this?"

import { parse } from './parse-html';

const HTML = `
    <h3>...</h3>
    ...
`;

const dom = parse(HTML);

const node = dom.children[1].children[3].children[0];

console.log(node);

@phenomnominal 2020

"It's much more likely that we would do something like this:"

const dom = parse(HTML);

const node = $(dom, 'body > h3 > span:last-child');

console.log(node);

"CSS selectors are a very powerful way to navigate a tree!"

@phenomnominal 2020

"So why don't we just use CSS selectors with the JavaScript AST?"

"Does JavaScript let us do that?"

@phenomnominal 2020

"Unfortunately no, not out of the box..."

"Fortunately, there's a huge JavaScript open-source community, and some
wonderful people made
it so we can do this!

@phenomnominal 2020

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = `let it = go('let it go');`;
const ast = parseScript(code);
const query = 'CallExpression:has(Identifier[name="go"]) > Literal';
const nodes = query(ast, query);

console.log(nodes);

"The query helper means that the chained property access vanishes!"

ESQuery

@phenomnominal 2020

"Woooow"

Olaf was impressed

@phenomnominal 2020

"I get it! These tools are amazing!"

"We can find out if a file has an identifier with a certain name!"

"Or all the names of all the exported functions!"

"Or if it matches some syntactical pattern!"

@phenomnominal 2020

"Like a lint rule!?"

"Yes! AST queries are perfect for writing lint rules!"

@phenomnominal 2020

"We can write a custom lint rule in only a few lines!"

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = `Object.freeze()`;
const ast = parseScript(code);
const query = 
  'CallExpression:has(MemberExpression[object.name="Object"][property.name="freeze"])';
const nodes = query(ast, query);

if (nodes.length !== 0) {
  throw new Error(`Don't use Object.freeze!`);
}

@phenomnominal 2020

"But wait. What do these queries have to do with the snow code?"

@phenomnominal 2020

"In order to change how the snow code works, we need to be able to add our code in the right place!"

"This is where the real magic begins..."

@phenomnominal 2020

Chapter Five

Real Magic

"My magic works by changing how physics works in real-time."

"Let's look at the snow code again..."

@phenomnominal 2020

import { doPhysics } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

"We need to wrap the "doPhysics" function call with some magic!"

@phenomnominal 2020

import { parseScript } from 'esprima';
import { query } from 'esquery';
import { readSnowCode } from './read-snow-code';

const PHYSICS_CALL_QUERY = 'CallExpression:has(Identifier[name="doPhysics"])';

const snowCode = await readSnowCode();
const ast = parseScript(snowCode);
const doPhysicsCalls = query(ast, PHYSICS_CALL_QUERY);

doPhysicsCalls.forEach(doPhysicsCall => {
  doPhysicsCall.expression.name = 'doMagic';
  const doMagicCall = doPhysicsCall;
  doMagicCall.arguments = [{
    type: 'CallExpression',
    callee: {
      type: 'Identifier',
      name: 'doPhysics'
    },
    arguments: [{
      type: 'Identifier',
      name: 'molecule'
    }]
  }];
});

"We can do that like this"

@phenomnominal 2020

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat, long, alt) {
  const molecule = global.waterMolecules[lat][long][alt];
  return doMagic(doPhysics(molecule));
}

"That gives us something like this"

"Can you spot the problem, Kristoff?"

@phenomnominal 2020

"We didn't create the "doMagic" function yet!"

"I think that it should freeze all the molecules!"

@phenomnominal 2020

"That's right! And what a great idea!"

"Generating code is pretty verbose when you manually create AST tokens..."

@phenomnominal 2020

// function doMagic (molecule) {
//   molecule.setTemperature(-10);
//   return molecule;
// }

function createMagicFunction (ast) {
  return {
    type: "FunctionDeclaration",
    id: {
      type: "Identifier",
      name: "doMagic"
    },
    params: [{
      type: "Identifier",
      name: "molecule",
    }],
    body: {
      type: "BlockStatement",
      body: [{
        type: "ExpressionStatement",
        expression: {
          type: "CallExpression",
          callee: {
            type: "MemberExpression",
            object: {
              type: "Identifier",
              name: "molecule"
            },
            property: {
              type: "Identifier",
              name: "setTemperature",
            },
          },
          arguments: [{
            type: "UnaryExpression",
            operator: "-",
            argument: {
              type: "Literal",
              value: 10
            }
          }]
      }
    }, {
    type: "ReturnStatement",
      argument: {
        type: "Identifier",
        name: "molecule"
      }
    }]
  }
}

"But it would look something like this..."

@phenomnominal 2020

"Yeeesh"

"You bet we can! We've one last trick up our sleeves!"

"We can do better than that can't we Elsa?"

@phenomnominal 2020

const h2 = document.createElement('h2');
h2.innerText = 'Weather result';
const h3 = document.createElement('h3');
const p1 = document.createElement('p');
p1.innerText = 'Arendelle, Norway';
const p2 = document.createElement('p');
p2.innerText = 'Sunday 19:00';
const span = document.createElement('span');
span.innerText = 'Sunny';
h3.appendChild(p1);
h3.appendChild(p2);
h3.appendChild(span);
const img = document.createElement('img');
img.setAttribute('alt', 'Sunny');
img.setAttribute('src', '//arendelle.info/weather/64/sunny.png');

"Let's go back to the DOM "

"Wouldn't it be quite odd to create elements like this?"

@phenomnominal 2020

const template = `
  <h2>Weather result</h2>
  <h3>
    <p>{{ location }}</p>
    <p>{{ time }}</p>
  <span>{{ weather }}</span>
  </h3>
  <img
    alt="{{ weather }}"
    src="//arendelle.info/weather/64/{{ weather }}.png">
`;

render(template, {
  location: 'Arendelle, Norway',
  time: 'Sunday 19:00',
  weather: 'Sunny'
});

"We're more likely to do something like this!"

@phenomnominal 2020

"That'd be pretty amazing! Does something like that exist for JavaScript?"

clip clop

@phenomnominal 2020

"It sure does!"

@phenomnominal 2020

The template helper means that all that code squashes down to just this:

import { query } from 'esquery';
import { compile } from 'estemplate';

const DO_MAGIC_TEMPLATE = tstemplate.compile(`
  function doMagic (molecule) {
    molecule.setTemperature(-<%= temperature %>);
    return molecule;
  }
`);

function createMagicFunction () {
  const doMagicAst = DO_MAGIC_TEMPLATE({
      temperature: createNumericLiteral('10')
  });
  const [doMagicFunction] = query(doMagicAst, 'FunctionDeclaration');
  return doMagicFunction;
}

ESTemplate

@phenomnominal 2020

"That is SO MUCH BETTER!!"

@phenomnominal 2020

"It sure is!"

"We have much less code to maintain!"

"Doing AST-based templating like this also makes it much easier to transplant and clone parts of the tree around!

@phenomnominal 2020

"Now all we need to do is convert the AST back into text!"

@phenomnominal 2020

"For that, we get to use one more open-source library!"

@phenomnominal 2020

The generate helper takes a valid AST and turns it into JavaScript!

import { parseScript } from 'esprima';
import { query } from 'esquery';
import { generate } from 'escodegen';

const code = `let it = go('let it go');`;
const ast = parseScript(code);
const query = 'CallExpression:has(Identifier[name="go"]) > Literal';
const [literal] = query(ast, query);
literal.value = 'LET IT GO!';
const updatedCode = generate(ast);

console.log(updatedCode); // let it = go('LET IT GO!');

ESCodeGen

@phenomnominal 2020

"Now everyone, we're ready to put all the bits together!"

"I'm going to need all your help!"

@phenomnominal 2020

import { readSnowCode } from './read-snow-code';

const snowCode = await readSnowCode();

import { parseScript } from 'esprima';

const ast = parseScript(snowCode);

import { query } from 'esquery';

const doPhysicsCalls = query(ast, 
  'CallExpression:has(Identifier[name="doPhysics"])'
);

import { template } from 'estemplate';

doPhysicsCalls.forEach(oldDoPhysicsCall => {
  oldDoPhysicsCall.expression.name = 'doMagic';
  const doMagicCall = oldDoPhysicsCall;
  const newDoPhysicsAst = template('doPhysics(molecule)');
  const [newDoPhysicsCall] = query(newDoPhysicsAst, 'CallExpression');
  doMagicCall.arguments = [newDoPhysicsCall];
});
  
const doMagicAst = template(`
  function doMagic (molecule) {
    molecule.setTemperature(-<%= temperature %>);
    return molecule;
  }
`, {
    temperature: createNumericLiteral('10')
});

const [doMagicFunction] = query(doMagicAst, 'FunctionDeclaration');

ast.statements.push(doMagicFunction);

import { generate } from 'escodegen';

const updatedSnowCode = generate(ast);

import { writeSnowCode } from './write-snow-code';

await writeSnowCode(updatedSnowCode);

"Are you ready for this?"

@phenomnominal 2020

"We just did a lot of things!"

Read the "Snow Code" with the "fs" and "path" APIs
Use ESPrima to convert the code into an AST
Use ESQuery to query the AST for the "doPhysics" CallExpression
Modify the AST to pass the result of "doPhysics" to "doMagic"
Use ESTemplate to create the code for the "doMagic" function
Add the new FunctionDeclaration back to the AST
Use ESCodeGen to print the AST back to a string
Save it to disk

"Let's recap!"

@phenomnominal 2020

"That was a lot of information"

@phenomnominal 2020

"These ideas are the foundation of all tools that help build the internet!"

"From Webpack to ESLint, from Sass to Create React App"

"And now we can make our own!!"

@phenomnominal 2020

"Remember you can use these ideas to modify any source code in any programming language!"

@phenomnominal 2020

"Elsa, thank you so much for showing us your magic!"

, said "Sven".

@phenomnominal 2020

"Don't you mean 'technology'?"

Everyone laughed, and they started heading back towards Arendelle.

, said Anna.

The End

Okay, Bye

twitter/phenomnominal
github/phenomnomnominal

Do You Want to Build a

Build Tool?

Do You Want to Build a

Build Tool?

Hi Everyone, I'm Craig !

Hi Everyone, I'm Craig !

Today, I'm going to tell you a fairy tale!

Today, I'm going to tell you a Fairy Tale!

Ready?

Ready?

It was Summer in the Kingdom of Arendelle!

It was Summer in the Kingdom of Arendelle!

And Princess Anna was So Excited!

And Princess Anna was So Excited!

The day had finally arrived!

The day had finally arrived!

Queen Elsa, Anna's sister and Best friend

Queen Elsa, Anna's sister and Best friend

Had finally agreed to share her secrets!

Had finally agreed to share her secrets!

Anna, and her friends Kristoff, Sven and Olaf

Anna, and her friends Kristoff, Sven and Olaf

Had been working hard all winter

Had been working hard all winter

And now it was time for their reward!

And now it was time for their reward!

Elsa has Amazing powers!

Elsa has Amazing powers!

She can sense the water molecules around us,

She can sense the water molecules around us,

transform those water molecules into ice,

transform those water molecules into ice,

And control that ice however she likes!

And control that ice however she likes!

today, Elsa IS going to share her magic!

today, Elsa is going to share her magic!

Chapter One

Chapter One

Just like magic

Just like magic

"Anna, do you remember when we went on our trip into the internet?",

asked Elsa.

"Of course!",

"We met all the other princesses, it was so much fun!"

replied Anna.

"There was that cute, new little princess, Vanellope!"

"And her friend Ralph was in trouble!"

"And we helped save him!"

"That's right! It was such a fun trip!"

"What else do you remember about the internet?"

"There was so much to do, so much to see!",

"So much data flying around! l learned all about how to make websites!"

exclaimed Anna.

" l loved it so much l even taught Sven when l got back!"

*SLURP! *

, Sven confirmed.

"l learned to write HTML,

and CSS,

and JavaScript!

"lt was just like magic!"

It was

just

like

magic

Chapter Two

Chapter Two

Sufficiently incredible

Sufficiently incredible

Anna, Kristoff, Olaf, and Sven waited patiently for Elsa to continue.

"I'm going to let you in on my little secret",

Elsa began.

"You may have heard that any sufficiently advanced technology is indistinguishable from magic,"

"but actually, its the other way around..."

"Any sufficiently incredible magic, is usually actually technology!"

"Wow! That's amazing! I think?"

Kristoff was already blown away by this.

"I'm just getting started!"

"To begin, we need to talk about data..."

said Elsa, with a grin.

"The whole internet is full of data."

SLURP!