Unbundling the JavaScript module bundler

{

Luciano Mammino (@loige)

ROME - APRIL 13/14 2018

Luciano... who

Find me online:

- Twitter (@loige)

- GitHub (lmammino)

- Linkedin

- Blog (loige.co)

Solution Architect at

  1. A sample app
  2. what is a dependency
  3. JavaScript module systems
  4. How a module bundler works
  5. Webpack in 2 minutes!

Agenda

App features

Dynamic DOM manipulation

React, Angular & Vue

are so... overrated! 😆

Dynamic Favicon

favico.js

Custom animated tooltips

tippy.js

🎉 Confetti rainfall 😱

dom-confetti

Persist state through Local Storage + UUIDs

store2 + uuidjs

7 Requests only for the JS code!

current scenario

zepto@1.2.0/dist/zepto.min.js
uuidjs@4.0.3/dist/uuid.core.js
store2@2.7.0/dist/store2.min.js
 tippy.js@2.2.3/dist/tippy.all.min.js
confetti@0.0.10/lib/main.js
 favico.js@0.3.10/favico-0.3.10.min.js

Ideal scenario

zepto@1.2.0/dist/zepto.min.js
uuidjs@4.0.3/dist/uuid.core.js
store2@2.7.0/dist/store2.min.js
 tippy.js@2.2.3/dist/tippy.all.min.js
confetti@0.0.10/lib/main.js
 favico.js@0.3.10/favico-0.3.10.min.js
vendors.js
+
+
+
+
+
=

How to do this?

./buildVendors.sh > vendors.js
buildVendors.sh
$
npm install --global lumpy
$
# lumpy.txt
https://unpkg.com/zepto@1.2.0/dist/zepto.min.js
https://unpkg.com/uuidjs@4.0.3/dist/uuid.core.js
https://unpkg.com/store2@2.7.0/dist/store2.min.js
https://unpkg.com/tippy.js@2.2.3/dist/tippy.all.min.js
https://unpkg.com/confetti-js@0.0.11/dist/index.min.js
https://unpkg.com/dom-confetti@0.0.10/lib/main.js
https://unpkg.com/favico.js@0.3.10/favico-0.3.10.min.js

Lumpy allows you to define

all the vendors in a text file

lumpy build
$
  1. Downloads the files from lumpy.txt (and caches them)
  2. Concatenates the content of the files
  3. Minifies the resulting source code (using babel-minify)
  4. Saves the resulting content in vendors.js

7 requests

2 requests

Concatenation
+
Minification

This is good...

... in 2008

was

Today...

We can do better!

  • Updating dependencies should be easy

  • We don't want to worry about dependencies of dependencies

  • We don't have to worry about the order of imports

Today...

Dependency

(or coupling)

a state in which one object uses a function of another object

Wikipedia

Reusable dependencies...

Modules!

Modules

The bricks for structuring non-trivial applications, but also the main mechanism to enforce information hiding by keeping private all the functions and variables that are not explicitly marked to be exported

Node.js Design Patterns (Second Edition)*

* yeah, I quite like quoting my stuff... 😅

Meet my friend I.I.F.E.

(Immediately Invoked Function Expression)

We generally define a function this way

const sum = (a, b) => a + b
sum(a, b) {
  return a + b
}

or

then, at some point, we execute it...

const four = sum(2, 2)

A function in JS creates an isolated scope

(a, b) => {
  const secretString = "Hello"
  return a + b
}

console.log(secretString) // undefined

secretString is not visible outside the function

IIFE allows you to define an isolated scope that executes itself


    (arg1, arg2) => {
       // do stuff here
       const iAmNotVisibleOutside = true
    }
(

A function with its own scope

This wrapper executes the function immediately and passes arguments from the outer scope

)(arg1, arg2)

IIFE is a recurring pattern in
JavaScript modules

Let's implement a module
that provides:

 

  • Information hiding

  • exported functionalities

const myModule = (() => {
  const privateFoo = () => { /* ... */ }
  const privateBar = [ /* ... */ ]

  const exported = {
    publicFoo: () => { /* ... */ },
    publicBar: [ /* ... */ ]
  };
	
  return exported
})()

A module

myModule.publicFoo()
myModule.publicBar[0]

myModule.privateFoo // undefined
myModule.privateBar // undefined
privateFoo // undefined
privateBar // undefined

IIFE

Creates an isolated scope

and executes it

information hiding

non-exposed functionality

defines exported functionalities

propagates the exports

to the outer scope (assigning it to myModule)

Can access exported functionalities

No visibility for the
non-exported ones

We want modules to be
 reusable across different apps and organisations...

...we need

A STANDARD MODULE format!

Module system features

Must have

  • Simple syntax for import / export
  • Information hiding
  • Allows to define modules in separate files
  • Modules can import from other modules
    (nested dependencies)​

Module system features

Nice to have

  • Ability to import module subsets
  • Avoid naming collision
  • Asynchronous module loading
  • Seamless support for Browsers & Server-side

JavaScript module systems

  • globals

  • CommonJS (Node.js)

  • AMD (Require.js / Dojo)

  • UMD

  • ES2015 Modules (ESM)

  • Many others (SystemJS, ...)

Globals

var $, jQuery
$ = jQuery = (() => { 
  return { /* ... */ }
})()


// ... use $ or jQuery in the global scope

$.find('.button').remove()

Globals

👎 Might generate naming collisions

(e.g. $ overrides browser global variable)

 

👎 Import order is important

 

👎 Cannot import parts of modules

CommonJS

// loDash.js
const loDash = {
  /* ... */
}

module.exports = loDash

module

// app.js

// import full module
const _ = require('./loDash')
_.concat([1], [2], [3])
// or import single functionality
const { concat } = require('./loDash')
concat([1], [2], [3])

app using module

CommonJS

👍 No naming collisions

(imported modules can be renamed)

👍 Huge repository of modules through NPM

 

👎 Synchronous import only

👎 Works natively on the server side only (Node.js)

AMD (Require.js)

Asynchronous Module Definition

// jquery-1.9.0.js

define(
  'jquery',
  ['sizzle', 'jqueryUI'],
  function (sizzle, jqueryUI) {
    // Returns the exported value
    return function () {
     // ...
    }
  }
)

module

module name

dependencies

factory function used to construct the module,

receives the dependencies as arguments

exported value

AMD (Require.js)

Asynchronous Module Definition

// app.js

// define paths
requirejs.config({
  baseUrl: 'js/lib',
  paths: {
    jquery: 'jquery-1.9.0'
  }
})

define(['jquery'], function ($) {
  // this is executed only when jquery
  // and its deps are loaded
});


app

Require.js config

jquery will be loaded from

://<currentDomain>/js/lib/jquery-1.9.0.js

app main function

Has jquery as dependency

AMD (Require.js)

Asynchronous Module Definition

👍 Asynchronous modules

👍 Works on Browsers and Server side

 

👎 Very verbose and convoluted syntax (my opinion™)

UMD

Universal Module Definition

A module definition that is compatible with

Global modules, CommonJS & AMD

 

👉 github.com/umdjs/umd 👈

(function (root, factory) {
  if (typeof exports === 'object') {
    // CommonJS
    module.exports = factory(require('dep'))
  } else if (typeof define === 'function' && define.amd) {
    // AMD
    define(['dep'], function (dep) {
      return (root.returnExportsGlobal = factory(dep))
    })
  } else {
    // Global Variables
    root.myModule = factory(root.dep)
  }
}(this, function (dep) {
  // Your actual module
  return {}
}))

IIFE with arguments:

 - Current scope (this) and the module factory function.

 - "dep" is a sample dependency of the module.

👍 Allows you to define modules that

can be used by almost any module loader

 

👎 Complex, the wrapper code

is almost impossible to write manually

UMD

Universal Module Definition

ES2015 modules

// calculator.js
const add = (num1, num2) => num1 + num2
const sub = (num1, num2) => num1 - num2
const div = (num1, num2) => num1 / num2
const mul = (num1, num2) => num1 * num2

export { add, sub, div, mul }



// app.js
import { add } from './calculator'
console.log(add(2,2)) // 4

module

exported functionalities

app

import specific functionality

ES2015 modules

// index.html
<html>
  <body>
    <!-- ... -->
    <script type="module">
      import { add } from 'calculator.js'
      console.log(add(2,2)) // 4
    </script>
  </body>
</html>

"works" in some modern browsers

ES2015 modules

🙄 Syntactically very similar to CommonJS...

BUT

👍 import & export are static
(allow static analysis of dependencies)

 

👍 It is a (still work in progress) standard format

 

👍 Works (almost) seamlessly in browsers & servers

ES2015 modules

Cool & broad subject, it would deserve it's own talk

Wanna know more?

🔗 ​import / export syntax reference

🔗 ECMAScript modules in browsers

🔗 ES modules: A cartoon deep-dive

🔗 ES Modules in Node Today!

So many options...

Current most used practice:

Use CommonJS or ES2015 & create "compiled bundles"

Let's try to use CommonJS in the browser

const $ = require('zepto')
const tippy = require('tippy.js')
const UUID = require('uuidjs')
const { confetti } = require('dom-confetti/src/main')
const store = require('store2')
const Favico = require('favico.js')

!(function () {
  const colors = ['#a864fd', '#29cdff', '#78ff44', '#ff718d', '#fdff6a']

  const todoApp = (rootEl, opt = {}) => {
    const todos = opt.todos || []
    let completedTasks = opt.completedTasks || 0
    const onChange = opt.onChange || (() => {})

    const list = rootEl.find('.todo-list')
    const footer = rootEl.find('.footer')
    const todoCount = footer.find('.todo-count')
    const insertInput = rootEl.find('.add-todo-box input')
    const insertBtn = rootEl.find('.add-todo-box button')

    const render = () => {
      let tips
      list.html('')
      if (todos.length === 0) {
        list.html('<div class="nothing"><p>Nothing to do</p><p><small>It\'s good to be lazy 🎉</small></p></div>')
      } else {
        const listContainer = $('<ul></ul>')
        todos.forEach(item => {
          const el = $(`<li id="${item.id}"><a title="Click to mark as done" href="#">${item.text}</a></li>`)
          el.on('click', (e) => {
            e.preventDefault()
            const elId = $(e.target).parent().attr('id')
            if (tips) tips.destroyAll()
            remove(elId)
          })
          el.on('keyup', (e) => {
            if (e.code === 'Enter' || e.code === 'Space') {
              e.preventDefault()
              const elId = $(e.target).parent().attr('id')
              if (tips) tips.destroyAll()
              remove(elId)
            }
          })
          const link = el.find('a')
          link.on('focus', (e) => {
            $(e.target).attr('title', 'Press [space] or [enter] to mark as done')
          })
          link.on('blur', (e) => {
            $(e.target).attr('title', 'Click to mark as done')
          })
          el.appendTo(listContainer)
        })
        listContainer.appendTo(list)
        tips = tippy(list.find('li').toArray(), {
          dynamicTitle: true,
          placement: 'top',
          animation: 'shift-away',
          arrow: 'up',
          target: 'a',
          trigger: 'focus mouseenter',
          theme: 'poo'
        })
      }

      let footerText = todos.length ? `<strong>${todos.length}</strong> task${todos.length !== 1 ? 's' : ''} to get done!` : ''
      footerText += completedTasks ? ` <strong>${completedTasks}</strong> completed task${completedTasks !== 1 ? 's' : ''}</strong>` : ''
      todoCount.html(footerText)
    }

    const add = (text) => {
      const item = { id: UUID.generate(), text }
      todos.push(item)
      render()
      onChange(todos, completedTasks)
      return item
    }

    const remove = (id) => {
      const elIndex = todos.findIndex((item) => item.id === id)
      const removedItem = todos.splice(elIndex, 1)[0]
      completedTasks++
      // every 10 todos completed spread some confetti!
      if (completedTasks % 10 === 0) {
        confetti(rootEl.get(0), {colors, angle: 45})
      }
      render()
      onChange(todos, completedTasks)
      return removedItem
    }

    const getAll = () => todos
    const getRootEl = () => rootEl

    insertInput.on('keyup', (e) => {
      if (insertInput.val() && e.key === 'Enter') {
        e.preventDefault()
        add(insertInput.val())
        insertInput.val('')
      }
    })

    insertBtn.on('click', (e) => {
      e.preventDefault()
      if (insertInput.val()) {
        add(insertInput.val())
        insertInput.val('')
      }
    })

    render()
    return ({ add, remove, getAll, getRootEl })
  }

  const returning = store.get('init')
  const storedTodos = store.get('todos', [])
  const storedCompletedTasks = store.get('completedTasks', 0)

  const favicon = new Favico({
    animation: 'popFade', position: 'up'
  })
  favicon.badge(storedTodos.length)

  const app = todoApp($('#app'), {
    todos: storedTodos,
    completedTasks: storedCompletedTasks,
    onChange: (todos, completedTasks) => {
      favicon.badge(todos.length)
      store('todos', todos)
      store('completedTasks', completedTasks)
    }
  })
  store.set('init', true)

  if (!returning) {
  // for the first init
    app.add('Buy soy milk')
    app.add('Do homeworks')
    app.add('Prepare material for the talk')
    app.add('Read SPAM emails')
    app.add('Write a module bundler')
    app.add('Tick some todos')
    app.add('Find a better name for this example')
  }
})()

The browser doesn't know how to process require.

It doesn't support CommonJS!

Module Bundler

A tool that takes modules with dependencies and emits static assets representing those modules

 

Those static assets can be processed by browsers!

Dependency graph

A graph built by connecting every module with its direct dependencies.

app

dependency A

dependency B

dependency A2

shared
dependency

A module bundler has to:

  • Construct the dependency graph (Dependency Resolution)
  • Assemble the modules in the graph into a single executable asset (Packing)
// app.js

const calculator = require('./calculator')
const log = require('./log')

log(calculator('2 + 2 / 4'))


// log.js

module.exports = console.log


// calculator.js

const parser = require('./parser')
const resolver = require('./resolver')

module.exports = (expr) => resolver(parser(expr))


// parser.js

module.exports = (expr) => { /* ... */ }


// resolver.js

module.exports = (tokens) => { /* ... */ }

Dependency resolution

app

calculator

log

parser

resolver

(entrypoint)

(1)
(2)
(3)
(4)
(5)

During dependency resolution,

the bundler creates a modules map

{







}
'./app': (module, require) => {  },
'./calculator': (module, require) => {  },
'./log': (module, require) => {  },
'./parser': (module, require) => {  },
'./resolver': (module, require) => {  },
const parser = require('./parser');const resolver = require('./resolver');module.exports = (expr) => resolver(parser(expr))

require path

module factory function

Packing

.js

{
  --- : ---
  --- : ----
  --- : --
}

Modules map

Single executable
JS file

Packed executable file

((modulesMap) => {
  const require = (name) => {
    const module = { exports: {} }
    modulesMap[name](module, require)

    return module.exports
  }

  require('./app')
})(
  {
    './app': (module, require) => { … },
    './calculator': (module, require) => { … },
    './log': (module, require) => { … },
    './parser': (module, require) => { … },
    './resolver': (module, require) => { … }
  }
)

IIFE passing the modules map as argument

Custom require function:

it will load the modules by evaluating the code from the modules map

A reference to a module with an empty module.exports. 

This will be filled at evaluation time

Invoking the factory function for the given module name.

(Service locator pattern)

The current reference module is passed, the factory function will modify this object by adding the proper exported values.

The custom require function is passed so, modules can recursively require other modules

The resulting module.exports is returned

The entrypoint module is required, triggering the actual execution of the business logic

Now you know how

Module bundlers work!

 

And how to convert code written using CommonJS to a single file that works in the browser

A challenge for you!

If you do, let me know... I'll have a prize for you!

 

TIP: you can use acorn to parse JavaScript files (look for require and module.exports) and resolve to map relative module paths to actual files in the filesystem.

Can you build a (simple) module bundler from scratch?

A state of the art module bundler for the web

npm install webpack webpack-cli
webpack app.js

yep, recent versions of Webpack work without config! 😎

webpack --mode=development app.js

Do not compress the code and add annotations!

Webpack concepts

  • Entry point: the starting file for dependency resolution.
  • Output: the destination file (bundled file).
  • Loaders: algorithms to parse different file types and convert them into executable javascript (e.g. babel, typescript, but also CSS, images or other static assets)
  • Plugins: do extra things (e.g. generate a wrapping HTML or analysis tools)
const { resolve, join } = require('path')
const CompressionPlugin = require('compression-webpack-plugin')

module.exports = {
  entry: './app.js',
  output: {
    path: resolve(join(__dirname, 'build')),
    filename: 'app.js'
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules|bower_components)/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: [
              ['@babel/preset-env']
            ]
          }
        }
      }
    ]
  },
  plugins: [
    new CompressionPlugin()
  ]
}

Webpack.config.js

Entrypoint
Build the dependency graph starting from ./app.js

Output
Save the resulting bundled file in ./build/app.js

Loaders
All the files matching "*.js" are processed with babel and converted to ES5 Javascript

Plugins

Uses a plugin that generates a gzipped copy of every emitted file.

...And you can do more!

Module bundlers are your friends

  • Now you know how they work,
    they are not (really) magic!
  • Start small and add more when needed
  • If you try to build your own you'll learn a lot more!

Grazie!

Special thanks to:

@Podgeypoos79, @andreaman87, @mariocasciaro (reviewers) and @MarijnJH (inspiration - check out his amazing book and his workshop on JS modules)

Unbundling the JavaScript module bundler - Codemotion Rome 2018

By Luciano Mammino

Unbundling the JavaScript module bundler - Codemotion Rome 2018

Today we all use Webpack (right?), but I remember a time when you had to manually copy-paste JavaScript files to create a package of libraries you could use in your frontend application. Many years have passed since then and the landscape of module bundlers evolved significantly along with the evolution of JavaScript and Node.js. In this talk, I will try to uncover some JavaScript module history and illustrate how a module bundler actually works, so that the next time you will use Webpack you will be able to understand what's going on behind the scenes.

  • 4,076