Reimplementing Meteor in TypeScript

Ben Newman

Meteor Night

9 July 2019

{ github,
  twitter,
  instagram,
  facebook
}.com/benjamn

Programmers love talking about how they would do things differently if they could start over

O, the code I’d write!

Myles Borins calls these stories “developer fan-fiction”

If only Dr. Seuss had been an O’Reilly author…

Ask yourself:

Do I have a time machine?

No?

Too bad for you and your great idea, then

I stand before you as someone who has, in fact, built a time machine or two

Time machines I have built:

Collaborations

Code that rewrites other code can bend the code-time continuum

Once a good idea finds a time machine, it can spread faster than any disease

This talk is all about my vision for how Meteor could have been implemented, and how I believe we can get there

🧐

☝️

We’re talking about the meteor/tools codebase, not Meteor application code

Important clarification:

But first…

An even bigger moonshot?

Meteor’s data system should be database-independent, rather than tightly coupled to Mongo
Making everything reactive and real-time by default is great for rapid prototyping, but also a source of scaling problems for Meteor applications
If Meteor becomes more agnostic about the sources of your data, we can’t keep calling its client-side query language “Minimongo”
As a company, we should get out of the business of shipping monolithic all-or-nothing frameworks

GraphQL?

A beautiful query language, far more declarative than the Mongo query API
Mutations feel like an afterthought, compared to Meteor’s latency-compensated, isomorphic methods
Not a replacement for Meteor’s Livedata system, at least not in 2015

Apollo!

Just a set of libraries published to npm
Server and client independently adoptable
To differentiate itself from Relay:
- Much more open to configuration
- Less opinionated about IDs, fragments, etc.
- No build step required
Declarative data fetching, without pretense of automatic updates
TypeScript instead of JavaScript

Apollo!

A different story, told to a different audience

Apollo!

The very future of the company was at stake! 😰

Apollo!

Would everything we learned from Meteor remain trapped inside that monolith, or not?

Meanwhile, Meteor

A JavaScript framework implemented in JavaScript
Real-time reactivity by default (sometimes to a fault)
A great way to write rich client applications using any view layer, but especially React
The build tool and module system of my dreams
A package ecosystem more powerful than npm
As little configuration as possible
A commitment to backwards compatibility and smooth upgrades
Open source, though somewhat centralized
A consistent source of revenue for Meteor Development Group

Zero Configuration

We take it very seriously, perhaps too seriously?

Success:

Dynamic `import()` and “exact code splitting”

Failure?

Selective recompilation of `node_modules`

What about TypeScript?

A statically typed language that compiles to JavaScript
Writing Apollo in plain JavaScript seemed irresponsible and risky
Not a choice that I would have made, at the time
Fortunately not my decision!

I was deeply skeptical of TypeScript

Every static type system I have ever used felt like a straitjacket

Noisy compilers make people hate programming

What’s the point of a static type system if following the rules is optional?

Do we really want to keep transpiling code for browsers forever?

JavaScript, like Meteor, avoids solving problems with configuration

TypeScript,

by contrast, comes from a culture of configurability

Meteor has evolved with JavaScript, and bears indelible traces of that history

TypeScript would have been a hipster choice at best for Meteor when it was first introduced in October 2012

JavaScript emerged from intense competition between browser vendors

And it represents a truly remarkable consensus among those stakeholders

TypeScript is a Microsoft project

JavaScript can be written in any text editor

TypeScript really shines within development environments like Visual Studio Code

Will something like TypeScript syntax ever become a standard feature of JavaScript?

As someone who attends TC39 meetings, I am very doubtful
However, as JavaScript improves, so does TypeScript

Better questions:

What would TypeScript gain from standardization?
Does TypeScript really need to be so configurable? Maybe so!
Can JavaScript stakeholders ever reach consensus on the hard tradeoffs inherent in static type systems?

Should TypeScript share equal status with JavaScript in Meteor, at least?

npm’s 2018 analysis

Something of a surprise, however, was TypeScript, with 46% of survey respondents reporting they use Microsoft’s… type-checked JavaScript variant. This is major adoption for a tool of this kind and might signal a sea change in how developers write JavaScript.

2018.stateofjs.com

npm’s 2019 analysis

Here’s the point:

Even an unsound optional type system bolted on top of JavaScript has this benefit

Static analysis lets you find out about problems sooner

TypeScript is a straitjacket that you can take off

And it’s a remarkably comfortable jacket when you choose to wear it!

TypeScript is a straitjacket that you can take off

… thanks to fantastic editor integration, structural subtypes, union types, the occasional `any` type, etc.

TypeScript has to earn your consent to yell at you about your code

… whereas most other static type systems enjoy a total monopoly over compiling and running your code

In fact, if you prefer, you can keep on writing pure dynamic JavaScript, and provide TypeScript `.d.ts` type declarations merely as documentation for external consumers

Every @types/* package published to npm is another proof of this point

With significant help and inspiration from Brie Bunge (of Airbnb)

Finding out about problems sooner was not our primary goal, however

I recently rewrote Recast in TypeScript

The real goal was to take advantage of TypeScript + VSCode to make writing AST transforms substantially easier and faster

I never want to go back to writing AST transforms without auto-completion

And I say that as someone who has all but memorized the AST formats of JavaScript and TypeScript

I never want to go back to writing AST transforms without auto-completion

I can’t imagine how anyone else does it

Every significant change I make to Meteor requires rereading tons and tons of potentially affected code

There’s no way to tell at a glance what kinds of data are expected, what data are available, or how the resulting data will be consumed

Every significant change I make to Meteor requires rereading tons and tons of potentially affected code

I know this codebase better than anyone else alive, so I know how intimidating it must be to anyone else

Why I’m convinced Meteor itself should have been implemented in TypeScript:

TypeScript will make the codebase more approachable for contributors
Changes to shared data types can be made more confidently
Hacking Meteor in VSCode is an absolute joy compared to EMACS
We might even find some bugs!

But… can we actually pull this off?

Can we choose to go to the moon before this decade is out?

“We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are [worthwhile]; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one we intend to win”

But… can we actually pull this off?

Do we have a time machine, or am I writing developer fan-fiction?

Here’s how:

We will convert the codebase one module at a time, reaping incremental benefits at every step
- The infrastructure has already been implemented in Meteor 1.8.2 betas
- Possible because .ts modules can coexist alongside .js modules
- Order of conversion can be decided according to impact
- No need to convert modules that are “too dynamic”
Emphasis on “we”

Here’s how:

The meteor/tools codebase is already compiled with meteor-babel
- This is how we originally validated the ecmascript and babel-compiler packages that almost every Meteor application uses
While Babel now supports its own implementation of the TypeScript compiler, we use the official TS compiler
- More control over compilation
- Fewer caveats than @babel/preset-typescript

Example: modules

export {
  appendFile as appendFile,
  appendFile as appendFileSync,
  chmod as chmod,
  chmod as chmodSync,
  close as close,
  close as closeSync,
  createReadStream as createReadStream,
  createWriteStream as createWriteStream,
  lstat as lstat,
  lstat as lstatSync,
  mkdir as mkdir,
  mkdir as mkdirSync,
  open as open,
  open as openSync,
  read as read,
  readFile as readFile,
  readFile as readFileSync,
  read as readSync,
  readdir as readdir,
  readdir as readdirSync,
  readlink as readlink,
  readlink as readlinkSync,
  realpath as realpath,
  realpath as realpathSync,
  rmdir as rmdir,
  rmdir as rmdirSync,
  stat as stat,
  stat as statSync,
  symlink as symlink,
  symlink as symlinkSync,
  unlink as unlink,
  unlink as unlinkSync,
  unwatchFile as unwatchFile,
  watchFile as watchFile,
  write as write,
  writeFile as writeFile,
  writeFile as writeFileSync,
  write as writeSync,
} from "./files";

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
Object.defineProperty(exports, "appendFile", {
  enumerable: true,
  get: function get() {
    return _files.appendFile;
  }
});
Object.defineProperty(exports, "appendFileSync", {
  enumerable: true,
  get: function get() {
    return _files.appendFile;
  }
});
Object.defineProperty(exports, "chmod", {
  enumerable: true,
  get: function get() {
    return _files.chmod;
  }
});
Object.defineProperty(exports, "chmodSync", {
  enumerable: true,
  get: function get() {
    return _files.chmod;
  }
});
Object.defineProperty(exports, "close", {
  enumerable: true,
  get: function get() {
    return _files.close;
  }
});
Object.defineProperty(exports, "closeSync", {
  enumerable: true,
  get: function get() {
    return _files.close;
  }
});
Object.defineProperty(exports, "createReadStream", {
  enumerable: true,
  get: function get() {
    return _files.createReadStream;
  }
});
Object.defineProperty(exports, "createWriteStream", {
  enumerable: true,
  get: function get() {
    return _files.createWriteStream;
  }
});
Object.defineProperty(exports, "lstat", {
  enumerable: true,
  get: function get() {
    return _files.lstat;
  }
});
Object.defineProperty(exports, "lstatSync", {
  enumerable: true,
  get: function get() {
    return _files.lstat;
  }
});
Object.defineProperty(exports, "mkdir", {
  enumerable: true,
  get: function get() {
    return _files.mkdir;
  }
});
Object.defineProperty(exports, "mkdirSync", {
  enumerable: true,
  get: function get() {
    return _files.mkdir;
  }
});
Object.defineProperty(exports, "open", {
  enumerable: true,
  get: function get() {
    return _files.open;
  }
});
Object.defineProperty(exports, "openSync", {
  enumerable: true,
  get: function get() {
    return _files.open;
  }
});
Object.defineProperty(exports, "read", {
  enumerable: true,
  get: function get() {
    return _files.read;
  }
});
Object.defineProperty(exports, "readFile", {
  enumerable: true,
  get: function get() {
    return _files.readFile;
  }
});
Object.defineProperty(exports, "readFileSync", {
  enumerable: true,
  get: function get() {
    return _files.readFile;
  }
});
Object.defineProperty(exports, "readSync", {
  enumerable: true,
  get: function get() {
    return _files.read;
  }
});
Object.defineProperty(exports, "readdir", {
  enumerable: true,
  get: function get() {
    return _files.readdir;
  }
});
Object.defineProperty(exports, "readdirSync", {
  enumerable: true,
  get: function get() {
    return _files.readdir;
  }
});
Object.defineProperty(exports, "readlink", {
  enumerable: true,
  get: function get() {
    return _files.readlink;
  }
});
Object.defineProperty(exports, "readlinkSync", {
  enumerable: true,
  get: function get() {
    return _files.readlink;
  }
});
Object.defineProperty(exports, "realpath", {
  enumerable: true,
  get: function get() {
    return _files.realpath;
  }
});
Object.defineProperty(exports, "realpathSync", {
  enumerable: true,
  get: function get() {
    return _files.realpath;
  }
});
Object.defineProperty(exports, "rmdir", {
  enumerable: true,
  get: function get() {
    return _files.rmdir;
  }
});
Object.defineProperty(exports, "rmdirSync", {
  enumerable: true,
  get: function get() {
    return _files.rmdir;
  }
});
Object.defineProperty(exports, "stat", {
  enumerable: true,
  get: function get() {
    return _files.stat;
  }
});
Object.defineProperty(exports, "statSync", {
  enumerable: true,
  get: function get() {
    return _files.stat;
  }
});
Object.defineProperty(exports, "symlink", {
  enumerable: true,
  get: function get() {
    return _files.symlink;
  }
});
Object.defineProperty(exports, "symlinkSync", {
  enumerable: true,
  get: function get() {
    return _files.symlink;
  }
});
Object.defineProperty(exports, "unlink", {
  enumerable: true,
  get: function get() {
    return _files.unlink;
  }
});
Object.defineProperty(exports, "unlinkSync", {
  enumerable: true,
  get: function get() {
    return _files.unlink;
  }
});
Object.defineProperty(exports, "unwatchFile", {
  enumerable: true,
  get: function get() {
    return _files.unwatchFile;
  }
});
Object.defineProperty(exports, "watchFile", {
  enumerable: true,
  get: function get() {
    return _files.watchFile;
  }
});
Object.defineProperty(exports, "write", {
  enumerable: true,
  get: function get() {
    return _files.write;
  }
});
Object.defineProperty(exports, "writeFile", {
  enumerable: true,
  get: function get() {
    return _files.writeFile;
  }
});
Object.defineProperty(exports, "writeFileSync", {
  enumerable: true,
  get: function get() {
    return _files.writeFile;
  }
});
Object.defineProperty(exports, "writeSync", {
  enumerable: true,
  get: function get() {
    return _files.write;
  }
});

var _files = require("./files");

Source code:

Babel CJS plugin:

Example: modules

export {
  appendFile as appendFile,
  appendFile as appendFileSync,
  chmod as chmod,
  chmod as chmodSync,
  close as close,
  close as closeSync,
  createReadStream as createReadStream,
  createWriteStream as createWriteStream,
  lstat as lstat,
  lstat as lstatSync,
  mkdir as mkdir,
  mkdir as mkdirSync,
  open as open,
  open as openSync,
  read as read,
  readFile as readFile,
  readFile as readFileSync,
  read as readSync,
  readdir as readdir,
  readdir as readdirSync,
  readlink as readlink,
  readlink as readlinkSync,
  realpath as realpath,
  realpath as realpathSync,
  rmdir as rmdir,
  rmdir as rmdirSync,
  stat as stat,
  stat as statSync,
  symlink as symlink,
  symlink as symlinkSync,
  unlink as unlink,
  unlink as unlinkSync,
  unwatchFile as unwatchFile,
  watchFile as watchFile,
  write as write,
  writeFile as writeFile,
  writeFile as writeFileSync,
  write as writeSync,
} from "./files";

module.link("./files", {
  appendFile: ["appendFile", "appendFileSync"],
  chmod: ["chmod", "chmodSync"],
  close: ["close", "closeSync"],
  createReadStream: "createReadStream",
  createWriteStream: "createWriteStream",
  lstat: ["lstat", "lstatSync"],
  mkdir: ["mkdir", "mkdirSync"],
  open: ["open", "openSync"],
  read: ["read", "readSync"],
  readFile: ["readFile", "readFileSync"],
  readdir: ["readdir", "readdirSync"],
  readlink: ["readlink", "readlinkSync"],
  realpath: ["realpath", "realpathSync"],
  rmdir: ["rmdir", "rmdirSync"],
  stat: ["stat", "statSync"],
  symlink: ["symlink", "symlinkSync"],
  unlink: ["unlink", "unlinkSync"],
  unwatchFile: "unwatchFile",
  watchFile: "watchFile",
  write: ["write", "writeSync"],
  writeFile: ["writeFile", "writeFileSync"]
}, 0);

Source code:

Meteor/Reify:

What about automating parts of the conversion?

I’m open to speeding up the process, but not if we miss the opportunity to specify meaningful types
Most automated tools simply default to the any type, which is not what we want
No value in TypeScript without meaningful types
Might as well just stick to JavaScript!

What about automating parts of the conversion?

I can’t think of a better way for the community to learn about the codebase than to participate in its conversion to TypeScript

Next.js 9 thinks alike

What about all the ECMAScript modernization we’ve already done?

Code is much easier to convert to TypeScript if it starts out as modern JavaScript

This work can happen in parallel, as long as we coordinate who’s converting which modules

All future contributors will benefit from the presence of type information

This kind of time machine travels into the future faster than real time, because we work together

Source: Meteor forums

DEMO

Some good options:

Where should we begin?

DEMO

Some good options:

Where should we begin?

One more thing…

Questions?

@inspector_beckett

Reimplementing Meteor in TypeScript

By Ben Newman

Reimplementing Meteor in TypeScript

6,237

Ben Newman

benjamn

Reimplementing Meteor in TypeScript

Ben Newman

Programmers love talking about how they would do things differently if they could start over

O, the code I’d write!

Myles Borins calls these stories “developer fan-fiction”

If only Dr. Seuss had been an O’Reilly author…

Ask yourself:

Do I have a time machine?

No?

Too bad for you and your great idea, then

I stand before you as someone who has, in fact, built a time machine or two

Time machines I have built:

Recast

Regenerator

Reify

Code that rewrites other code can bend the code-time continuum

Once a good idea finds a time machine, it can spread faster than any disease

This talk is all about my vision for how Meteor could have been implemented, and how I believe we can get there

🧐

☝️

We’re talking about the meteor/tools codebase, not Meteor application code

Important clarification:

But first…

An even bigger moonshot?

GraphQL?

A beautiful query language, far more declarative than the Mongo query API

Mutations feel like an afterthought, compared to Meteor’s latency-compensated, isomorphic methods

Not a replacement for Meteor’s Livedata system, at least not in 2015

Apollo!

Just a set of libraries published to npm

Server and client independently adoptable

To differentiate itself from Relay:

Declarative data fetching, without pretense of automatic updates

TypeScript instead of JavaScript

Apollo!

A different story, told to a different audience

Apollo!

The very future of the company was at stake! 😰

Apollo!

Would everything we learned from Meteor remain trapped inside that monolith, or not?

Meanwhile, Meteor

Zero Configuration

We take it very seriously, perhaps too seriously?

Success:

Dynamic import() and “exact code splitting”

Failure?

Selective recompilation of node_modules

What about TypeScript?

A statically typed language that compiles to JavaScript

Writing Apollo in plain JavaScript seemed irresponsible and risky

Not a choice that I would have made, at the time

Fortunately not my decision!

I was deeply skeptical of TypeScript

Every static type system I have ever used felt like a straitjacket

Noisy compilers make people hate programming

What’s the point of a static type system if following the rules is optional?

Do we really want to keep transpiling code for browsers forever?

JavaScript, like Meteor, avoids solving problems with configuration

TypeScript,

by contrast, comes from a culture of configurability

Meteor has evolved with JavaScript, and bears indelible traces of that history

TypeScript would have been a hipster choice at best for Meteor when it was first introduced in October 2012

JavaScript emerged from intense competition between browser vendors

And it represents a truly remarkable consensus among those stakeholders

TypeScript is a Microsoft project

JavaScript can be written in any text editor

TypeScript really shines within development environments like Visual Studio Code

Will something like TypeScript syntax ever become a standard feature of JavaScript?

As someone who attends TC39 meetings, I am very doubtful

However, as JavaScript improves, so does TypeScript

Better questions:

What would TypeScript gain from standardization?

Does TypeScript really need to be so configurable? Maybe so!

Can JavaScript stakeholders ever reach consensus on the hard tradeoffs inherent in static type systems?

Should TypeScript share equal status with JavaScript in Meteor, at least?

npm’s 2018 analysis

2018.stateofjs.com

npm’s 2019 analysis

Here’s the point:

Even an unsound optional type system bolted on top of JavaScript has this benefit

Dynamic `import()` and “exact code splitting”

Selective recompilation of `node_modules`

… thanks to fantastic editor integration, structural subtypes, union types, the occasional `any` type, etc.

In fact, if you prefer, you can keep on writing pure dynamic JavaScript, and provide TypeScript `.d.ts` type declarations merely as documentation for external consumers