Do You Want to Build a

Build Tool?

Do You Want to Build a

Build Tool?

Hi Everyone, I'm Craig !

Hi Everyone, I'm Craig !

@phenomnominal 2019

  • twitter/phenomnominal
  • github/phenomnomnominal

Today, I'm going to tell you a fairy tale!

Today, I'm going to tell you a Fairy Tale!

@phenomnominal 2019

Ready?

Ready?

@phenomnominal 2019

It was Summer in the Kingdom of Arendelle!

@phenomnominal 2019

It was Summer in the Kingdom of Arendelle!

And Princess Anna was So Excited!

@phenomnominal 2019

And Princess Anna was So Excited!

The day had finally arrived!

@phenomnominal 2019

The day had finally arrived!

@phenomnominal 2019

Queen Elsa, Anna's sister and Best friend

Queen Elsa, Anna's sister and Best friend

@phenomnominal 2019

Had finally agreed to share her secrets!

Had finally agreed to share her secrets!

Anna, and her friends Kristoff, Sven and Olaf

@phenomnominal 2019

Anna, and her friends Kristoff, Sven and Olaf

@phenomnominal 2019

Had been working hard all winter

Had been working hard all winter

And now it was time for their reward!

@phenomnominal 2019

And now it was time for their reward!

Elsa has Amazing powers!

@phenomnominal 2019

Elsa has Amazing powers!

She can sense the water molecules around us,

@phenomnominal 2019

She can sense the water molecules around us,

transform those water molecules into ice,

@phenomnominal 2019

transform those water molecules into ice,

And control that ice however she likes!

@phenomnominal 2019

And control that ice however she likes!

@phenomnominal 2019

today, Elsa IS going to share her magic!

today, Elsa is going to share her magic!

Chapter One

Chapter One

Just like magic

Just like magic

@phenomnominal 2019

"Anna, do you remember when we went on our trip into the internet?",

asked Elsa.

@phenomnominal 2019

"Of course!",

"We met all the other princesses, it was so much fun!"

replied Anna.

@phenomnominal 2019

"There was that cute, new  little princess, Vanellope!"

@phenomnominal 2019

"And her friend Ralph was in trouble!"

@phenomnominal 2019

"And we helped save him!"

@phenomnominal 2019

"That's right! It was such a fun trip!"

"What else do you remember about the internet?"

@phenomnominal 2019

"There was so much to do, so much to see!",

"So much data flying around! l learned all about how to make websites!"

exclaimed Anna.

" l loved it so much l even taught Sven when l got back!"

@phenomnominal 2019

*SLURP! *

, Sven confirmed.

@phenomnominal 2019

"l learned to write HTML,

and CSS,

and TypeScript!

"lt was just like magic!"

@phenomnominal 2019

It was

just

like

magic

Chapter Two

Chapter Two

Sufficiently incredible

Sufficiently incredible

@phenomnominal 2019

Anna, Kristoff, Olaf, and Sven waited patiently for Elsa to continue.

@phenomnominal 2019

"I'm going to let you in on my little secret",

Elsa began.

@phenomnominal 2019

"You may have heard that any sufficiently advanced technology is indistinguishable from magic,"

"but actually, its the other way around..."

@phenomnominal 2019

"Any sufficiently incredible magic, is usually actually technology!"

@phenomnominal 2019

"Wow! That's amazing! I think?"

Kristoff was already blown away by this.

@phenomnominal 2019

"I'm just getting started!"

"To begin, we need to talk about data..."

said Elsa, with a grin.

@phenomnominal 2019

"The whole internet is full of data."

"Pictures"

"Memes"

"Infographics"

@phenomnominal 2019

{
  today: {
      temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
  },
  future: [
    { high: 21, low: 13, chanceOfSnow: 0.05 },
    { high: 21, low: 14, chanceOfSnow: 0.03 },
    { high: 21, low: 16, chanceOfSnow: 0.04 },
    { high: 19, low: 16, chanceOfSnow: 0.04 },
    /* ... */
  ]
}

"And that data is typically structured in some way, like this JSON  file"

@phenomnominal 2019

"Or like this HTML   file"

<h2>Weather result</h2>
<h3>
    <p>Arendelle, Norway</p>
    <p>Sunday 19:00</p>
    <span>Sunny</span>
</h3>
<img alt="Sunny" src="//arendelle.info/weather/64/sunny.png">
<h3>
  19<button aria-label="°Celsius">°C</button>
</h3>
<p>Precipitation: <span>0%</span></p>
<p>Humidity: <span>75%</span></p>
<p>Wind: <span>1 m/s</span></p>
<ol>
    <li>
        <p aria-label="Sunday">Sun</p>
        <img alt="Clear" src="//arendelle.info/weather/48/sunny.png">
        <span>21</span>°
        <span>13</span>°
    </li>
    <!-- ... -->
</ol>

@phenomnominal 2019

"Web developers use programming languages to inspect and manipulate that data"

const response = await fetch('https://arendelle.info/weather.json');
const data = await response.json();

// {
//   today: {
//       temperatures: [19, 18.5, 18, 18.5, 17, /* ... */]
//   },
//   future: [
//     { high: 21, low: 13, chanceOfSnow: 0.05 },
//     { high: 21, low: 14, chanceOfSnow: 0.03 },
//     { high: 21, low: 16, chanceOfSnow: 0.04 },
//     { high: 19, low: 16, chanceOfSnow: 0.04 },
//     /* ... */
//   ]
// }

const SATURDAY = 6;
const today = new Date().getDay();
const chanceOfSnowOnSaturday = data.future[SATURDAY - today].chanceOfSnow * 100;
h1 {
  // ...
}

h3 > p {
  // ... 
}

button:active,
button:hover {
  // ...
}

li:first-child {
  // ...
}

"Languages like JavaScript"

"or CSS"

@phenomnominal 2019

"Olaf, what exactly do you imagine when you think about snow?"

@phenomnominal 2019

"I think about skiing, and sledding, and snow fights!"

"It's so bright, and so cold, and so hard, but also so soft!"

"What do you think about Elsa?",

he asked.

@phenomnominal 2019

"I think about all those things too!"

"But my powers allow me to see another level deeper as well..."

"Going outside in the cold, having fun with our friends, ice skating..."

@phenomnominal 2019

"I see the whole system."

"In that system, each water molecule is a little bit of data, following the set of instructions that nature gave it."

@phenomnominal 2019

"Those instructions tell the water molecules when to freeze. "

"Those instructions tell the molecules which way to go when the wind blows."

"I know how to control those instructions."

@phenomnominal 2019

"l've worked it out!",

"You can understand the snow's source code!"

Anna announced suddenly.

@phenomnominal 2019

"Exactly!",

Esla grinned again.

"And do you want to know the best bit?"

@phenomnominal 2019

"The code that contains the instructions for all the snow in the world is written in TypeScript!"

"I can show you how to query code, modify code, and even create code from nothing!!"

"What do you think?"

Chapter Three

Chapter THREE

The Snow Code

The Snow Code

@phenomnominal 2019

Olaf was so excited that he could barely stop himself from bursting into song!

@phenomnominal 2019

"Will you teach us to use the snow code, Elsa?"

"Please, please, please, please, PLEASE!"

, he pleaded.

@phenomnominal 2019

"Of course I will!"

, Elsa reassured him.

@phenomnominal 2019

"But first, I need to teach you a few tricks!"

@phenomnominal 2019

"My gifts let me access the snow code directly!"

"You will need to use a slightly more direct approach..."

"Kristoff, have you heard of node.js?"

@phenomnominal 2019

"Yes!"

Kristoff looked pretty proud of himself.

"Anna taught me all about it."

@phenomnominal 2019

"Node.js is a computer program that lets us run JavaScript without a browser!"

@phenomnominal 2019

"We can use it to do things like read and write   files, find out stuff about the system, and manipulate data in lots of different ways!"

@phenomnominal 2019

"Exactly right! Let's start by looking at how we can read and write files using node.js"

File System APIs

File System APIs

@phenomnominal 2019

fs.watch

Node has functions that let us do all kinds of things with the file system:

  • you can check if you can access a file with
  • change file permissions with 
  • copy a file from one place to another with 
  • create a symlink with 
  • or set up a file watcher with 
fs.link
fs.copyFile
fs.chmod
fs.access

File System APIs

File System APIs

@phenomnominal 2019

The most important functions for us to learn today are:

fs.writeFile
fs.mkdir
fs.readFile
  •                                     - read a file from a path
  •                        - make a directory at a path
  •                                   - write a file to a path

File System APIs

File System APIs

@phenomnominal 2019

// snowflake.txt

                                ()
                                /\
                               //\\
                              <<  >>
                          ()   \\//   ()
                ()._____   /\   \\   /\   _____.()
                   \.--.\ //\\ //\\ //\\ /.--./
                    \\__\\/__\//__\//__\\/__//
                     '--/\\--//\--//\--/\\--'
                        \\\\///\\//\\\////
                    ()-= >>\\< <\\> >\\<< =-()
                        ////\\\//\\///\\\\
                     .--\\/--\//--/\\--/\\--.
                    //""/\\""//\""//\""//\""\\
                   /'--'/ \\// \\// \\// \'--'\
                 ()`"""`   \/   //   \/   `"""`()
                          ()   //\\   ()
                              <<  >>
                               \\//
                                \/
                                ()

Imagine that we have a file like this:

@phenomnominal 2019

import * as fs from 'fs';

async function main () {
  console.log(await readSnowflake());
};

async function readSnowflake (): Promise<string> {
  return fs.promises.readFile('./snowflake.txt', 'utf8');
}

main();

File System APIs

File System APIs

We can use the Promise-based fs API to read the file, and print it:

@phenomnominal 2019

File System APIs

File System APIs

Which will give us something like this:

Weaseltown:dywbabt queenelsa$ ts-node index.ts 

                              ()
                              /\
                             //\\
                            <<  >>
                        ()   \\//   ()
              ()._____   /\   \\   /\   _____.()
                 \.--.\ //\\ //\\ //\\ /.--./
                  \\__\\/__\//__\//__\\/__//
                   '--/\\--//\--//\--/\\--'
                      \\\\///\\//\\\////
                  ()-= >>\\< <\\> >\\<< =-()
                      ////\\\//\\///\\\\
                   .--\\/--\//--/\\--/\\--.
                  //""/\\""//\""//\""//\""\\
                 /'--'/ \\// \\// \\// \'--'\
               ()`"""`   \/   //   \/   `"""`()
                        ()   //\\   ()
                            <<  >>
                             \\//
                              \/
                              ()

@phenomnominal 2019

import * as fs from 'fs';

async function main () {
  const snowflake = await readSnowflake();
  await saveSnowflake(snowflake);
};

async function readSnowflake (): Promise<string> {
  return fs.promises.readFile('./snowflake.txt', 'utf8');
}

async function saveSnowflake (data: string): Promise<void> {
  await fs.promises.mkdir('./saved/snowflakes', { recursive: true });
  await fs.promises.writeFile('./saved/snowflakes/snowflake.txt', data);
}

main();

File System APIs

File System APIs

We can combine that with making a directory and writing the file to disk:

Path APIs

@phenomnominal 2019

Path APIs

It is also important to understand how the path API works:

path.resolve
  •                                       - resolve a path relative to the given path
  •                                 - split a path string into its constituent parts
path.parse

@phenomnominal 2019

import * as path from 'path';

const relativeToThisFile = path.resolve(__dirname, './snowflake.txt');
const parsed = path.parse(relativeToThisFile);

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

Path APIs

Path APIs

import * as fs from 'fs';
import * as path from 'path';

export async function readSnowflake (): Promise<string> {
  const relativePath = path.resolve(__dirname, './snowflake.txt');
  const relativePath = path.resolve(process.cwd(), './snowflake.txt');
  return fs.promises.readFile(relativePath, 'utf8');
}

@phenomnominal 2019

import * as glob from 'fast-glob';
 
(async function () {
  const entries = await glob('./src/**/*.ts');
  console.log(entries);
  // ['./src/path/to/a/file.ts', './src/path/to/another/file.ts']
})();

Path APIs

Path APIs

Node does not provide globing by default, but it very useful for finding all matching paths:

@phenomnominal 2019

, Sven snorted.

*snort*

@phenomnominal 2019

"That's right Sven! You must always test your file structure & path operations on all the different operating systems you care about! It's very easy to get it wrong!"

@phenomnominal 2019

"Now, we've seen how we can read a plain text file, what about something a bit different..."

@phenomnominal 2019

import * as fs from 'fs';
import * as path from 'path';

async function main () {
  console.log(await readSnowflakes());
}

async function readSnowflakes (): Promise<Array<WaterMolecule>> {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const json = await fs.promises.readFile(snowflakePath, 'utf8');
  return JSON.parse(json);
}

main();

We can read different types of data from different files:

JSON data

JSON data

@phenomnominal 2019

"Wow! So we took some data that was in a file, and parsed it into JavaScript objects that we can manipulate with code! Cool!"

Anna was really excited!

@phenomnominal 2019

"If you think that is cool, check this out..."

Elsa was just getting started!

@phenomnominal 2019

"If you think that is cool, check this out..."

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

Elsa was just getting started!

@phenomnominal 2019

"THE PATH TO THE SNOW CODE!"

Olaf instantly knew what it was!

const SNOW_CODE_PATH = path.resolve(
  __dirname,
  './elsa',
  process.env.SECRET_SNOW_FILE
);

@phenomnominal 2019

"Yes! The snow code is just another file!"

"It's just text and we can manipulate it and control it!"

@phenomnominal 2019

import * as fs from 'fs';

async function main () {
  console.log(await readSnowCode());
};

export async function readSnowCode (): Promise<string> {
  return fs.promises.readFile(SNOW_CODE_PATH, 'utf8');
}

main();

Code as data

Code as data

We don't need to treat a code file any differently:

@phenomnominal 2019

Weaseltown:dywbabt queenelsa$ ts-node index.ts 

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat: number, long: number, alt: number): WaterMolecule {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

Code as data

Code as data

@phenomnominal 2019

"Now we have the snow code as a string, so  we can change it however we want!"

"This is amazing!"

Kristoff chimed in:

@phenomnominal 2019

// kristoff.ts

import * as fs from 'fs';
import { readSnowCode } from './read-snow-code';

(async function () {
  const code = await readSnowCode();
  
  code += `; console.log('Kristoff is the best!');`
  
  return fs.writeFile('./kristoffs-cool-new-snow-code.ts', code);
})();

"I'm going to make it always  print out 'Kristoff is  the best!'"

, yelled Kristoff.

@phenomnominal 2019

// olaf.ts

import * as fs from 'fs';

(async function () {
  const code = await fs.readFile('./kristoffs-cool-new-snow-code.ts');
  
  code = code.replace(/Kristoff/, 'Olaf');
  
  return fs.writeFile('./kristoffs-cool-new-snow-code.ts', code);
})();

"Not if I can help it!"

, replied Olaf

@phenomnominal 2019

// sven.ts

import * as fs from 'fs';

(async function () {
  const code = await fs.readFile('./kristoffs-cool-new-snow-code.ts');
  
  code = code.replace(/Olaf/, 'Sven');
  code = code.replace(/best!/, 'bestest!!!!');
  code = code.replace(/console.log\((.*)\)/, 'console.log($1.toUpperCase())');
  
  return fs.writeFile('./kristoffs-cool-new-snow-code.ts', code);
})();

Even Sven had his own ideas...

@phenomnominal 2019

"Hold on everyone, l think we're getting ahead of ourselves!"

"Anna, can you think of any reasons why working with the string directly might not be the best approach?"

@phenomnominal 2019

"Strings aren't exactly structured!"

"You're just changing a bunch of characters, and that could break easily!"

"Regular Expressions are hard to get right, and hard to understand later!"

"You have to fight with comments, and whitespace!"

@phenomnominal 2019

"So how can we get the snow code into a more useful structure?"

@phenomnominal 2019

import * as path from 'path';

const parsed = path.parse('/some/path/to/some/file.txt');

// interface ParsedPath {
//   root: string;
//   dir: string;
//   base: string;
//   ext: string;
//   name: string;
// }

console.log(parsed.ext);
// read-snowflakes.ts

import * as fs from 'fs';
import * as path from 'path';

export async function readSnowflakes (): Promise<Array<WaterMolecule>> {
  const snowflakePath = path.resolve(__dirname, './snowflakes.json');
  const data = await fs.promises.readFile(snowflakePath, 'utf8');
  const snowflakes = JSON.parse(data);
  
  console.log(snowFlakes[0].velocity.x)
}

Parsing

Parsing

Let's look back at how we handled the other strings:

@phenomnominal 2019

import { createSourceFile, ScriptKind, ScriptTarget, SourceFile } from 'typescript';
import { readSnowCode } from './read-snow-code';

async function main () {
  const code = await readSnowCode();
  const ast = parse(code);
  console.log(ast);
}

function parse (source: string): SourceFile {
    return createSourceFile('', source, ScriptTarget.Latest, true, ScriptKind.TS);
}

main();

Parsing

Parsing

We also need to parse our string of code!

@phenomnominal 2019

"An ast, I LOVE IT!"

"What is an ast?"

@phenomnominal 2019

"Good try Olaf..."

"It's actually an A S T "

"It's an Abstract Syntax Tree..."

Chapter Four

Chapter Four

The root of all problems

The root of all problems

@phenomnominal 2019

"Oh yeah of course, I know all about those kinds of trees..."

"But I think Sven needs a bit of a refresher..."

Kristoff sounded uncertain.

@phenomnominal 2019

"Sure! Let's start with a literal meaning."

@phenomnominal 2019

Abstract Syntax Trees

Tree

data structure made up of vertices and edges without any cycles

SYNTAX

the way in which linguistic elements are put together

Abstract

Not associated with any specific instance

Abstract Syntax Trees

@phenomnominal 2019

"????"

????

????

????

????

????

????

????

????

@phenomnominal 2019

Abstract Syntax Trees

An Abstract Syntax Tree is a data structure that represents the structure of code, but without any actual syntax.

let it = go('let it go');
it = go 'let it go'
it <= go ['let it go']
'let it go' -> go => it

Abstract Syntax Trees

@phenomnominal 2019

{
  "kind": "Program",
  "statements": [{
    "kind": "VariableStatement",
    "declarationList": {
      "kind": "VariableDeclarationList",
      "declarations": [{
        "kind": "VariableDeclaration",
        "name": {
          "kind": "Identifier",
          "text": "it"
        },
        "initializer": {
          "kind": "CallExpression",
          "expression": {
            "kind": "Identifier",
            "text": "go"
          },
          "arguments": [{
            "kind": "StringLiteral",
            "text": "let it go"
          }]
        }
      }]
    }
  }],
}

@phenomnominal 2019

            'let it go'

StringLiteral

Identifier

CallExpression

Identifier

VariableDeclaration

VariableDeclarationList

         go
         go('let it go')
    it
    it = go('let it go')
let it = go('let it go')

VariableStatement

let it = go('let it go');

@phenomnominal 2019

Identifier

IfStatement

ImportDeclaration

NumericLiteral

TrueKeyword

CallExpression

"Learning these abstract names and this new data structure is tricky, but there are tools to help!"

StringLiteral

ClassDeclaration

FunctionDeclaration

VariableDeclaration

FalseKeyword

NullKeyword

AnyKeyword

ForStatement

WhileStatement

DoStatement

"This tool lets you look at the parser output for a huge number of languages!"

@phenomnominal 2019

"It is very useful for seeing the structure of the tree and what properties are present!"

AST Explorer

AST Explorer

@phenomnominal 2019

@phenomnominal 2019

import { createSourceFile, ScriptKind, ScriptTarget, SourceFile } from 'typescript';

function parse (source: string): SourceFile {
  return createSourceFile('', source, ScriptTarget.Latest, true, ScriptKind.TS);
}

const code = `let it = go('let it go');`
const ast = parse(code);

const arg = ast.statements[0].declarationsList.declarations[0].initializer.arguments[0];

console.log(arg);

"Seeing the whole tree helps to write out long complex property access chains"

@phenomnominal 2019

"So Anna, what do you think? Is this better than the string version?"

@phenomnominal 2019

"l guess so!"

"We have a real structure now with the AST"

"We access data exactly like we would with JSON, so it's easier to understand"

"We don't have to think about comments, or whitespace, we just access what we want to!"

@phenomnominal 2019

"lt's still kind of clunky though..."

"Do you think we can do better?",

" And it will throw an error if the structure doesn't line up with the code!"

Elsa asked.

@phenomnominal 2019

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

"Let's look at another data structure..."

@phenomnominal 2019

"Olaf, what do you think this data structure represents?"

@phenomnominal 2019

{
  "kind": "document",
  "children": [{
    "kind": "h2",
    "children": [{
      "kind": "text",
      "text": "Weather result"
    }]
  }, {
    "kind": "h3",
    "children": [{
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Arendelle, Norway"
      }]
    }, {
      "kind": "p",
      "children": [{
        "kind": "text",
        "text": "Sunday 19:00"
      }]
    }, {
      "kind": "span",
      "children": [{
        "kind": "text",
        "text": "Sunny"
      }]
    }]
  }, {
    "kind": "img",
    "alt": "Sunny",
    "src": "//arendelle.info/weather/64/sunny.png"
  }]
}

@phenomnominal 2019

"IT LOOKS LIKE HTML!"

@phenomnominal 2019

"That's close to what the browser creates when it downloads an index.html file and parses it!"

"Very good!"

"We call it the Document Object Model, or the DOM."

"It is also a tree!"

@phenomnominal 2019

"Now, wouldn't it be strange if we queried the DOM  like this?"

import { parse } from './parse-html';

const HTML = `
    <h3>...</h3>
    ...
`;

const dom = parse(HTML);

const node = dom.children[1].children[3].children[0];

console.log(node);

@phenomnominal 2019

"It's much more likely that we would do something like this:"

const dom = parse(HTML);

const node = $(dom, 'body > h3 > span:last-child');

console.log(node);

"CSS selectors are a very powerful way to navigate a tree!"

@phenomnominal 2019

"So why don't we just use CSS selectors with the TypeScript AST?"

"Does TypeScript let us do that?"

@phenomnominal 2019

"Unfortunately no, not out of the box."

"Fortunately, there's a huge TypeScript community, and some kind, generous, hilarious, brilliant, wonderful person made it so we can do this!

@phenomnominal 2019

@phenomnominal 2019

import { tsquery } from '@phenomnomnominal/tsquery';

const code = `let it = go('let it go');`;
const ast = tsquery.ast(code);
const query = 'CallExpression:has(Identifier[name="go"]) > StringLiteral';
const nodes = tsquery.query(ast, query);

console.log(nodes);

"The query helper means that the chained property access vanishes!

TSQuery

TSQuery

@phenomnominal 2019

"Woooow"

Olaf was impressed

@phenomnominal 2019

"I don't know, these queries look kind of confusing..."

"Yeah, is there some way we can practise?"

Kristoff was a bit stuck

@phenomnominal 2019

"Of course!"

" You can experiment with the TSQuery playground, made by another brilliant community member,
Uri Shaked!"

@phenomnominal 2019

TSQuery Playground

TSQuery Playground

"This tool is like a stripped back version of AST Explorer, just for TypeScript"

"It surfaces the AST node names, which helps you see patterns and understand how they work together."

@phenomnominal 2019

@phenomnominal 2019

"I get it! These tools are amazing!"

"We can find out if a file has an identifier with a certain name!"

"Or all the names of all the exported functions!"

"Or if it matches some syntactical pattern!"

@phenomnominal 2019

"Like a lint rule!?"

"Yes! AST queries are *perfect* for writing lint rules!"

@phenomnominal 2019

"We can write a custom lint rule in only a few lines!"

import { tsquery } from '@phenomnomnominal/tsquery';

const code = `Object.freeze()`;
const ast = tsquery.ast(code);

const query = 
  'CallExpression:has(PropertyAccessExpression[expression.text="Object"][name.text="freeze"])';
const nodes = tsquery.query(ast, query);

if (nodes.length !== 0) {
  throw new Error(`Don't use Object.freeze!`);
}

@phenomnominal 2019

"But wait. What do these queries have to do with the snow code?"

@phenomnominal 2019

"In order to change how the snow code works, we need to be able to add our code in the right place!"

"This is where the real magic begins..."

Chapter Five

Chapter Five

Real Magic

Real Magic

@phenomnominal 2019

"My magic works by changing how physics works in real-time."

"Let's look at the snow code again..."

@phenomnominal 2019

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat: number, long: number, alt: number): WaterMolecule {
  const molecule = global.waterMolecules[lat][long][alt];
  return doPhysics(molecule);
}

"We need to wrap the "doPhysics" function call with some magic!"

@phenomnominal 2019

import { tsquery } from '@phenomnomnominal/tsquery';
import { createCall, createIdentifier, createPrinter } from 'typescript';
import { readSnowCode } from './read-snow-code';

const PHYSICS_CALL_QUERY = 'CallExpression:has(Identifier[name="doPhysics"])';

(async function () {
  const snowCode = await readSnowCode());
  const ast = tsquery.ast(snowCode);
  const doPhysicsCalls = tsquery.query(ast, PHYSICS_CALL_QUERY);
 
  doPhysicsCalls.forEach(doPhysicsCall => {
    doPhysicsCall.expression.name = 'doMagic';
    doMagicCall = doPhysicsCall;
    doMagicCall.arguments = [
      createCall(
        createIdentifier('doPhysics'),
      	null,
        [createIdentifier('molecule')]
      )
    ];
  });
})();

"We can do that like this"

@phenomnominal 2019

import { doPhysics, WaterMolecule } from './physics';

requestAnimationFrame(() => {
  for (let lat = -90; lat < 90; lat += 0.000000001) {
    for (let long = -180; long < 180; long += 0.000000001) {
      for (let alt = 0; alt < 80000; alt += 0.000000001) {
        const updated = update(lat, long, alt);
        render(updated);
      }
    }
  }
});

function update (lat: number, long: number, alt: number): WaterMolecule {
  const molecule = global.waterMolecules[lat][long][alt];
  return doMagic(doPhysics(molecule));
}

"That gives us something like this"

"Can you spot the problem, Kristoff?"

@phenomnominal 2019

"We didn't create the "doMagic" function  yet!"

"I think that it  should freeze all the molecules!"

@phenomnominal 2019

"That's right! And what a great idea!"

"Generating code is pretty tricky with the normal TypeScript APIs..."

@phenomnominal 2019

// function doMagic (molecule) {
//   molecule.setTemperature(-10);
//   return molecule;
// }

function createMagicFunction (ast) {
  return createFunctionDeclaration(null, null, null,
    createIdentifier('doMagic'), null,
    [createParameter(null, null, null, createIdentifier('molecule'))],
    null,
    createBlock([
      createExpressionStatement(
        createCall(
          createPropertyAccess(
            createIdentifier('molecule'),
            createIdentifier('setTemperature')
          ), null,
          [createPrefix(SyntaxKind.MinusToken,createNumericLiteral('10'))]
        )
      ),
      createReturn(createIdentifier('molecule'))
    ])
  )
}

"But it would look something like this..."

@phenomnominal 2019

"Yeesh"

"You bet we can! We've one last trick up our sleeves!"

"We can do better than that can't we Elsa?"

@phenomnominal 2019

const h2 = document.createElement('h2');
h2.innerText = 'Weather result';
const h3 = document.createElement('h3');
const p1 = document.createElement('p');
p1.innerText = 'Arendelle, Norway';
const p2 = document.createElement('p');
p2.innerText = 'Sunday 19:00';
const span = document.createElement('span');
span.innerText = 'Sunny';
h3.appendChild(p1);
h3.appendChild(p2);
h3.appendChild(span);
const img = document.createElement('img');
img.setAttribute('alt', 'Sunny');
img.setAttribute('src', '//arendelle.info/weather/64/sunny.png');

"Let's go back to the DOM "

"Wouldn't it be quite odd to create elements like this?"

@phenomnominal 2019

const template = `
  <h2>Weather result</h2>
  <h3>
    <p>{{ location }}</p>
    <p>{{ time }}</p>
  <span>{{ weather }}</span>
  </h3>
  <img
    alt="{{ weather }}"
    src="//arendelle.info/weather/64/{{ weather }}.png">
`;

render(template, {
  location: 'Arendelle, Norway',
  time: 'Sunday 19:00',
  weather: 'Sunny'
});

"We're more likely to do something like this!"

@phenomnominal 2019

"That'd be pretty amazing! Does something like that exist for TypeScript?"

*clip clop*

@phenomnominal 2019

"It sure does! The same absolutely incredible person who made TSQuery developed another tool "

@phenomnominal 2019

@phenomnominal 2019

The template helper means that all that code squashes down to just this:

import { tsquery } from '@phenomnomnomninal/tsquery';
import { tstemplate } from '@phenomnomnominal/tstemplate';

const DO_MAGIC_TEMPLATE = tstemplate.compile(`
  function doMagic (molecule) {
    molecule.setTemperature(-<%= temperature %>);
    return molecule;
  }
`);

function createMagicFunction () {
  const doMagicAst = DO_MAGIC_TEMPLATE({
      temperature: createNumericLiteral('10')
  });
  const [doMagicFunction] = tsquery.query(doMagicAst, 'FunctionDeclaration');
  return doMagicFunction;
}

TSTemplate

TSTemplate

@phenomnominal 2019

"That is SO MUCH BETTER!!"

@phenomnominal 2019

"It sure is!"

"We have so much less code!"

"Doing AST-based templating like this also makes it much easier to transplant and clone parts of the tree around!

@phenomnominal 2019

"Now everyone, we're ready to put all the bits together!"

"I'm going to need all your help!"

@phenomnominal 2019

(async function () {
  const snowCode = await readSnowCode();
  const ast = tsquery.ast(snowCode);
  
  const doPhysicsCalls = tsquery.query(ast, 
    'CallExpression:has(Identifier[name="doPhysics"])'
  );
  doPhysicsCalls.forEach(doPhysicsCall => {
    doPhysicsCall.expression.name = 'doMagic';
    doPhysicsCall.arguments = [createCall(
      createIdentifier('doPhysics'), null, [createIdentifier('molecule')]
    )];
  });
  
  const doMagicAst = tstemplate(`
    function doMagic (molecule) {
      molecule.setTemperature(-<%= temperature %>);
      return molecule;
    }
  `, {
      temperature: createNumericLiteral('10')
  });
  const [doMagicFunction] = tsquery.query(doMagicAst, 'FunctionDeclaration');
  ast.statements.push(doMagicFunction);
  
  await writeSnowCode(createPrinter().print(ast));
})();

"Are you ready for this?"

@phenomnominal 2019

"We just did a lot of things!"

  • Read the "Snow Code" with the "fs" and "path" APIs
  • Used TSQuery to convert the code into an AST
  • Used TSQuery to query the AST for the "doPhysics" CallExpression
  • Modified the AST to pass the result of "doPhysics" to "doMagic"
  • Used TSTemplate to create the code for the "doMagic" function
  • Added the new Function Declaration back to the AST
  • Printed the AST back to a string
  • And saved it to disk!

"Let's recap!"

@phenomnominal 2019

"That was a lot of information"

@phenomnominal 2019

"I know, thank you all for sticking with me!"

"And remember you can use these ideas to modify any source code in any language!"

@phenomnominal 2019

"These ideas are the foundation of all tools that help build the internet!"

"From Webpack to ESLint, from Babel to the Angular CLI "

"And now we can make our own!!"

@phenomnominal 2019

"Elsa, thank you so much for showing us your magic!"

, said "Sven".

@phenomnominal 2019

"Don't you mean 'technology'?"

Everyone laughed, and they started heading back towards Arendelle.

, said Anna.

The End

The End

Okay, Bye

Okay, Bye

  • twitter/phenomnominal
  • github/phenomnomnominal