Unbundling the JavaScript

module bundler

Luciano Mammino - @loige

21/11/2018

Webpack == PAIN!?

❤️

💔💔

It's not Webpack!

Module bundling is actually complicated!

Luciano... who

Find me online:

- Twitter (@loige)

- GitHub (lmammino)

- Linkedin

- Blog (loige.co)

Solution Architect at

  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

App features

Dynamic DOM manipulation

React, Angular, Vue

are so... overrated! 😆

Dynamic Favicon

favico.js

Custom animated tooltips

tippy.js

🎉 Confetti rainfall 😱

dom-confetti

Persist state through Local Storage + UUIDs

store2 + uuidjs

7 Requests only for the JS code!

Current scenario

zepto@1.2.0/dist/zepto.min.js
uuidjs@4.0.3/dist/uuid.core.js
store2@2.7.0/dist/store2.min.js
 tippy.js@2.2.3/dist/tippy.all.min.js
confetti@0.0.10/lib/main.js
 favico.js@0.3.10/favico-0.3.10.min.js

Ideal scenario

zepto@1.2.0/dist/zepto.min.js
uuidjs@4.0.3/dist/uuid.core.js
store2@2.7.0/dist/store2.min.js
 tippy.js@2.2.3/dist/tippy.all.min.js
confetti@0.0.10/lib/main.js
 favico.js@0.3.10/favico-0.3.10.min.js
vendors.js
+
+
+
+
+
=

How to do this?

./buildVendors.sh > vendors.js
buildVendors.sh
$
npx lumpy build
$
# lumpy.txt
https://unpkg.com/zepto@1.2.0/dist/zepto.min.js
https://unpkg.com/uuidjs@4.0.3/dist/uuid.core.js
https://unpkg.com/store2@2.7.0/dist/store2.min.js
https://unpkg.com/tippy.js@2.2.3/dist/tippy.all.min.js
https://unpkg.com/confetti-js@0.0.11/dist/index.min.js
https://unpkg.com/dom-confetti@0.0.10/lib/main.js
https://unpkg.com/favico.js@0.3.10/favico-0.3.10.min.js

Lumpy allows you to define

all the vendors in a text file

lumpy build
$
  1. Downloads the files from lumpy.txt (and caches them)
  2. Concatenates the content of the files
  3. Minifies the resulting source code (using babel-minify)
  4. Saves the resulting content in vendors.js

7 requests

2 requests

Even better if you "minify" these!

Concatenation
+
Minification

This is good...

... in 2008

was

Today...

We can do better!

  • Updating them should be easy

  • We shouldn't worry about transitive dependencies
    (dependencies of dependencies)

  • Order of imports shouldn't really matter

We rely on dependencies!

Dependency

(or coupling)

a state in which one object uses a function of another object

Wikipedia

Reusable dependencies...

Modules!

Modules

The bricks for structuring non-trivial applications, but also the main mechanism to enforce information hiding by keeping private all the functions and variables that are not explicitly marked to be exported

Node.js Design Patterns (Second Edition)*

* yeah, I quite like quoting my stuff... 😅

  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

Meet my friend I.I.F.E.

(Immediately Invoked Function Expression)

We generally define a function this way

const sum = (a, b) => a + b
function sum(a, b) {
  return a + b
}

or

then, at some point, we execute it...

const four = sum(2, 2)

A function in JS creates an isolated scope

(a, b) => {
  const secretString = "Hello"
  return a + b
}

console.log(secretString) // undefined

secretString is not visible outside the function

IIFE allows you to define an isolated scope that executes itself


    (arg1, arg2) => {
       // do stuff here
       const iAmNotVisibleOutside = true
    }
(

A function with its own scope

This wrapper executes the function immediately and passes arguments from the outer scope

)(someArg1, someArg2)

IIFE is a recurring pattern in
JavaScript modules

Let's implement a module
that provides:

 

  • Information hiding

  • exported functionalities

const myModule = (() => {
  const privateFoo = () => { /* ... */ }
  const privateBar = [ /* ... */ ]

  const exported = {
    publicFoo: () => { /* ... */ },
    publicBar: [ /* ... */ ]
  };
	
  return exported
})()

A module

myModule.publicFoo()
myModule.publicBar[0]

myModule.privateFoo // undefined
myModule.privateBar // undefined
privateFoo // undefined
privateBar // undefined

IIFE

Creates an isolated scope

and executes it

information hiding

non-exposed functionality

defines exported functionalities

propagates the exports

to the outer scope (assigning it to myModule)

Can access exported functionalities

No visibility for the
non-exported ones

We want modules to be
 reusable across different apps and organisations...

...we need

A STANDARD MODULE format!

Module system features

Must have

  • Simple syntax for import / export
  • Information hiding
  • Allows to define modules in separate files
  • Modules can import from other modules
    (nested dependencies)​

Module system features

Nice to have

  • Ability to import module subsets
  • Avoid naming collision
  • Asynchronous module loading
  • Seamless support for Browsers & Server-side

JavaScript module systems

  • globals

  • CommonJS (Node.js)

  • AMD (Require.js / Dojo)

  • UMD

  • ES2015 Modules (ESM)

  • Many others (SystemJS, ...)

Globals

var $, jQuery
$ = jQuery = (() => { 
  return { /* ... */ }
})()


// ... use $ or jQuery in the global scope

$.find('.button').remove()

Globals

👎 Might generate naming collisions

(e.g. $ overrides browser global variable)

 

👎 Modules needs to be "fully loaded" in the right order

 

👎 Cannot import parts of modules

CommonJS

// loDash.js
const loDash = {
  /* ... */
}

module.exports = loDash

module

// app.js

// import full module
const _ = require('./loDash')
_.concat([1], [2], [3])
// or import single functionality
const { concat } = require('./loDash')
concat([1], [2], [3])

app using module

CommonJS

👍 No naming collisions

(imported modules can be renamed)

👍 Huge repository of modules through NPM

 

👎 Synchronous import only

👎 Works natively on the server side only (Node.js)

AMD (Require.js)

Asynchronous Module Definition

// jquery-1.9.0.js

define(
  'jquery',
  ['sizzle', 'jqueryUI'],
  function (sizzle, jqueryUI) {
    // Returns the exported value
    return function () {
     // ...
    }
  }
)

module

module name

dependencies

factory function used to construct the module,

receives the dependencies as arguments

exported value

AMD (Require.js)

Asynchronous Module Definition

// app.js

// define paths
requirejs.config({
  baseUrl: 'js/lib',
  paths: {
    jquery: 'jquery-1.9.0'
  }
})

define(['jquery'], function ($) {
  // this is executed only when jquery
  // and its deps are loaded
});


app

Require.js config

jquery will be loaded from

://<currentDomain>/js/lib/jquery-1.9.0.js

app main function

Has jquery as dependency

AMD (Require.js)

Asynchronous Module Definition

👍 Asynchronous modules

👍 Works on Browsers and Server side

 

👎 Very verbose and convoluted syntax (my opinion™)

UMD

Universal Module Definition

A module definition that is compatible with

Global modules, CommonJS & AMD

 

👉 github.com/umdjs/umd 👈

(function (root, factory) {
  if (typeof exports === 'object') {
    // CommonJS
    module.exports = factory(require('dep'))
  } else if (typeof define === 'function' && define.amd) {
    // AMD
    define(['dep'], function (dep) {
      return (root.returnExportsGlobal = factory(dep))
    })
  } else {
    // Global Variables
    root.myModule = factory(root.dep)
  }
}(this, function (dep) {
  // Your actual module
  return {}
}))

IIFE with arguments:

 - Current scope (this) and the module factory function.

 - "dep" is a sample dependency of the module.

👍 Allows you to define modules that

can be used by almost any module loader

 

👎 Complex, the wrapper code

is almost impossible to write manually

UMD

Universal Module Definition

ES2015 modules

Cool & broad subject, it would deserve it's own talk

Wanna know more?

🔗 ​import / export syntax reference

🔗 ECMAScript modules in browsers

🔗 ES modules: A cartoon deep-dive

🔗 ES Modules in Node Today!

ES2015 modules

// calculator.js
const add = (num1, num2) => num1 + num2
const sub = (num1, num2) => num1 - num2
const div = (num1, num2) => num1 / num2
const mul = (num1, num2) => num1 * num2

export { add, sub, div, mul }



// app.js
import { add } from './calculator'
console.log(add(2,2)) // 4

module

exported functionalities

app

import specific functionality

ES2015 modules

// index.html
<html>
  <body>
    <!-- ... -->
    <script type="module">
      import { add } from 'calculator.js'
      console.log(add(2,2)) // 4
    </script>
  </body>
</html>

"works" in some modern browsers

ES2015 modules

🙄 Syntactically very similar to CommonJS...

BUT

👍 import & export are static
(allow static analysis of dependencies)

 

👍 It is a (still work in progress) standard format

 

👍 Works (almost) seamlessly in browsers & servers

So many options...

Current most used practice:

Use CommonJS or ES2015 & create "compiled bundles"

  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

Let's try to use CommonJS in the browser

const $ = require('zepto')
const tippy = require('tippy.js')
const UUID = require('uuidjs')
const { confetti } = require('dom-confetti/src/main')
const store = require('store2')
const Favico = require('favico.js')

!(function () {
  const colors = ['#a864fd', '#29cdff', '#78ff44', '#ff718d', '#fdff6a']

  const todoApp = (rootEl, opt = {}) => {
    const todos = opt.todos || []
    let completedTasks = opt.completedTasks || 0
    const onChange = opt.onChange || (() => {})

    const list = rootEl.find('.todo-list')
    const footer = rootEl.find('.footer')
    const todoCount = footer.find('.todo-count')
    const insertInput = rootEl.find('.add-todo-box input')
    const insertBtn = rootEl.find('.add-todo-box button')

    const render = () => {
      let tips
      list.html('')
      if (todos.length === 0) {
        list.html('<div class="nothing"><p>Nothing to do</p><p><small>It\'s good to be lazy 🎉</small></p></div>')
      } else {
        const listContainer = $('<ul></ul>')
        todos.forEach(item => {
          const el = $(`<li id="${item.id}"><a title="Click to mark as done" href="#">${item.text}</a></li>`)
          el.on('click', (e) => {
            e.preventDefault()
            const elId = $(e.target).parent().attr('id')
            if (tips) tips.destroyAll()
            remove(elId)
          })
          el.on('keyup', (e) => {
            if (e.code === 'Enter' || e.code === 'Space') {
              e.preventDefault()
              const elId = $(e.target).parent().attr('id')
              if (tips) tips.destroyAll()
              remove(elId)
            }
          })
          const link = el.find('a')
          link.on('focus', (e) => {
            $(e.target).attr('title', 'Press [space] or [enter] to mark as done')
          })
          link.on('blur', (e) => {
            $(e.target).attr('title', 'Click to mark as done')
          })
          el.appendTo(listContainer)
        })
        listContainer.appendTo(list)
        tips = tippy(list.find('li').toArray(), {
          dynamicTitle: true,
          placement: 'top',
          animation: 'shift-away',
          arrow: 'up',
          target: 'a',
          trigger: 'focus mouseenter',
          theme: 'poo'
        })
      }

      let footerText = todos.length ? `<strong>${todos.length}</strong> task${todos.length !== 1 ? 's' : ''} to get done!` : ''
      footerText += completedTasks ? ` <strong>${completedTasks}</strong> completed task${completedTasks !== 1 ? 's' : ''}</strong>` : ''
      todoCount.html(footerText)
    }

    const add = (text) => {
      const item = { id: UUID.generate(), text }
      todos.push(item)
      render()
      onChange(todos, completedTasks)
      return item
    }

    const remove = (id) => {
      const elIndex = todos.findIndex((item) => item.id === id)
      const removedItem = todos.splice(elIndex, 1)[0]
      completedTasks++
      // every 10 todos completed spread some confetti!
      if (completedTasks % 10 === 0) {
        confetti(rootEl.get(0), {colors, angle: 45})
      }
      render()
      onChange(todos, completedTasks)
      return removedItem
    }

    const getAll = () => todos
    const getRootEl = () => rootEl

    insertInput.on('keyup', (e) => {
      if (insertInput.val() && e.key === 'Enter') {
        e.preventDefault()
        add(insertInput.val())
        insertInput.val('')
      }
    })

    insertBtn.on('click', (e) => {
      e.preventDefault()
      if (insertInput.val()) {
        add(insertInput.val())
        insertInput.val('')
      }
    })

    render()
    return ({ add, remove, getAll, getRootEl })
  }

  const returning = store.get('init')
  const storedTodos = store.get('todos', [])
  const storedCompletedTasks = store.get('completedTasks', 0)

  const favicon = new Favico({
    animation: 'popFade', position: 'up'
  })
  favicon.badge(storedTodos.length)

  const app = todoApp($('#app'), {
    todos: storedTodos,
    completedTasks: storedCompletedTasks,
    onChange: (todos, completedTasks) => {
      favicon.badge(todos.length)
      store('todos', todos)
      store('completedTasks', completedTasks)
    }
  })
  store.set('init', true)

  if (!returning) {
  // for the first init
    app.add('Buy soy milk')
    app.add('Do homeworks')
    app.add('Prepare material for the talk')
    app.add('Read SPAM emails')
    app.add('Write a module bundler')
    app.add('Tick some todos')
    app.add('Find a better name for this example')
  }
})()

The browser doesn't know how to process require.

It doesn't support CommonJS!

Module Bundler

A tool that takes modules with dependencies and emits static assets representing those modules

 

Those static assets can be processed by browsers!

Dependency graph

A graph built by connecting every module with its direct dependencies.

app

dependency A

dependency B

dependency A2

shared
dependency

A module bundler has to:

  1. Construct the dependency graph (Dependency Resolution)
  2. Assemble the modules in the graph into a single executable asset (Packing)
// app.js

const calculator = require('./calculator')
const log = require('./log')

log(calculator('2 + 2 / 4'))


// log.js

module.exports = console.log


// calculator.js

const parser = require('./parser')
const resolver = require('./resolver')

module.exports = (expr) => resolver(parser(expr))


// parser.js

module.exports = (expr) => { /* ... */ }


// resolver.js

module.exports = (tokens) => { /* ... */ }

Dependency resolution

app

calculator

log

parser

resolver

(entrypoint)

(1)
(2)
(3)
(4)
(5)

During dependency resolution,

the bundler creates a modules map

{







}
'./app': (module, require) => {  },
'./calculator': (module, require) => {  },
'./log': (module, require) => {  },
'./parser': (module, require) => {  },
'./resolver': (module, require) => {  },
const parser = require('./parser');const resolver = require('./resolver');module.exports = (expr) => resolver(parser(expr))

require path

module factory function

Packing

.js

{
  --- : ---
  --- : ----
  --- : --
}

Modules map

Single executable
JS file

Packed executable file

((modulesMap) => {
  const require = (name) => {
    const module = { exports: {} }
    modulesMap[name](module, require)

    return module.exports
  }

  require('./app')
})(
  {
    './app': (module, require) => { … },
    './calculator': (module, require) => { … },
    './log': (module, require) => { … },
    './parser': (module, require) => { … },
    './resolver': (module, require) => { … }
  }
)

IIFE passing the modules map as argument

Custom require function:

it will load the modules by evaluating the code from the modules map

A reference to a module with an empty module.exports. 

This will be filled at evaluation time

Invoking the factory function for the given module name.

(Service locator pattern)

The current reference module is passed, the factory function will modify this object by adding the proper exported values.

The custom require function is passed so, modules can recursively require other modules

The resulting module.exports is returned

The entrypoint module is required, triggering the actual execution of the business logic

Now you know how

Module bundlers work!

 

And how to convert code written using CommonJS to a single file that works in the browser

A challenge for you!

If you do, let me know... I'll arrange a prize for you!

 

TIP: you can use acorn or babel-parser to parse JavaScript files (look for require and module.exports) and resolve to map relative module paths to actual files in the filesystem.

 

Need an inspiration?
Check the awesome minipack and @adamisnotdead's w_bp_ck!

Can you build a (simple) module bundler from scratch?

  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

A state of the art module bundler for the web

npm install webpack webpack-cli
webpack app.js

yep, recent versions of Webpack work without config! 😎

webpack --mode=development app.js

Do not compress the code and add annotations!

Webpack concepts

  • Entry point: the starting file for dependency resolution.
  • Output: the destination file (bundled file).
  • Loaders: algorithms to parse different file types and convert them into executable javascript (e.g. babel, typescript, but also CSS, images or other static assets)
  • Plugins: do extra things (e.g. generate a wrapping HTML or analysis tools)
const { resolve, join } = require('path')
const CompressionPlugin = require('compression-webpack-plugin')

module.exports = {
  entry: './app.js',
  output: {
    path: resolve(join(__dirname, 'build')),
    filename: 'app.js'
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /(node_modules|bower_components)/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: [
              ['@babel/preset-env']
            ]
          }
        }
      }
    ]
  },
  plugins: [
    new CompressionPlugin()
  ]
}

Webpack.config.js

Entrypoint
Build the dependency graph starting from ./app.js

Output
Save the resulting bundled file in ./build/app.js

Loaders
All the files matching "*.js" are processed with babel and converted to ES5 Javascript

Plugins

Uses a plugin that generates a gzipped copy of every emitted file.

Everything is a module

import React, { Component } from 'react'
import logo from './logo.svg'
import './App.css'

class App extends Component {
  render() {
    return (
      <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          <h1 className="App-title">Welcome to React</h1>
        </header>
      </div>
    );
  }
}

export default App

Webpack can load any type of file

As long as you can provide a "loader" that tells how to convert the file into something the browser understands.

This is how Webpack allows you to use Babel, TypeScript, Clojure, Elm, Imba but also to load CSSs, Images and other assets.

{
  test: /\.css$/,
  use: [
    require.resolve('style-loader'),
    {
      loader: require.resolve('css-loader'),
      options: {
        importLoaders: 1,
      },
    },
    {
      loader: require.resolve('postcss-loader'),
      options: {
        ident: 'postcss',
        plugins: () => [
          require('postcss-flexbugs-fixes'),
          autoprefixer({
            browsers: [
              '>1%',
              'last 4 versions',
              'Firefox ESR',
              'not ie < 9',
            ],
            flexbox: 'no-2009',
          }),
        ],
      },
    }
  ]
}

// Defines how to load .css files (uses a pipeline of loaders)

// parses the file with post-css

// process @import and url()
// statements

// inject the resulting code with a <style> tag

...Webpack can do (a lot) more!

  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

Bundle cache busting

Device

CDN

Origin

Server

example.com

bundle.js

bundle.js

bundle.js

(original)

bundle.js

(cache copy)

bundle.js

(cache copy)

Bundle cache busting

Device

CDN

Origin

Server

example.com

bundle.js

bundle.js

bundle.js

(original)

bundle.js

(cache copy)

bundle.js

(cache copy)

NEW VERSION

STALE!

STALE!

Bundle cache busting

(Manual) Solution 1

bundle.js?v=1

bundle.js?v=2

bundle.js?v=3

Doesn't play nice with some CDNs and Proxies
(they won't consider different query parameters to be different resources)

Bundle cache busting

(Manual) Solution 2

bundle-v1.js

bundle-v2.js

bundle-v3.js

...

Better, but still a lot of diligence and manual effort needed...

Bundle cache busting

Webpack Solution

bundle.9f61f58dd1cc3bb82182.js
bundle.aacdf58ef1aa12382199.js
bundle.ed61f68defef3bb82221.js

...

output: {
  filename: '[name].[contenthash].js'
}

Bundle cache busting

Webpack Solution

contenthash + webpack-html-plugin

 

Every new asset version will generate a new file

cache is automatically cleaned up on every release

(if content actually changed)

html-plugin will update the reference to the new file

A good example for dev/prod config

(with create-react-app)

npm i -g create-react-app

create-react-app new-project
cd new-project
npm run eject
cat config/webpack*.js
  1. Why we need modules
  2. JavaScript module systems
  3. How a module bundler works
  4. Webpack in 2 minutes!
  5. Advanced module bundling

Agenda

Module bundlers are your friends

  • Now you know how they work,
    they are not (really) magic!
  • Start small and add more when needed
  • If you try to build your own, you'll learn a lot more!

Webpack is not
the only possibility!

THE PRIZE!

npx twaffle --keywords '#oredev #IWANTTOWINAPRIZE'

Checkout twaffle if you want to organise Twitter-based raffles.

Tack så mycket!

     Special thanks:

Tadeas from Øredev (for reviewing my stuff thoroughly)

@Podgeypoos79, @andreaman87, @mariocasciaro, @eugenserbanescu (reviewers) and @MarijnJH (inspirations: his amazing book and his workshop on JS modules)

Unbundling the JavaScript module bundler - Oredev 21 Nov 2018

By Luciano Mammino

Unbundling the JavaScript module bundler - Oredev 21 Nov 2018

The landscape of module bundlers has evolved significantly since the days you would manually copy-paste your libraries to create a package for your frontend app. Like many parts of the JS world, the evolution has happened somewhat haphazardly, and the pace of change can feel overwhelming. Has Webpack ever felt like magic to you? How well do you understand what’s really going on under the hood? In this talk, I will uncover the history of JS module bundlers and illustrate how they actually work. Once we have the basics down, I will dive deeper into some of the more advanced topics, such as bundle cache boost and resolving cycling dependencies. At the end of this session, you will have a much more profound understanding of what’s going on behind the scenes.

  • 4,048