multi-process-dependencies-graph

/salestrader [ct5-post-conversion|✔] 14:27 $ npm start

Version: webpack 2.3.3
Time: 57358ms
webpack: Compiled successfully.

/salestrader [ct5-post-conversion|✔] 14:28 $ npm start

Time: 16931ms
webpack: Compiled successfully.

Given

"/Users/briand/dev/salestrader/apps/salestrader/src/index.js"

1. The file is processed by Webpack loaders.
2. Their output is parsed to extract the files dependencies.
3. The dependencies are resolved to file paths.
4. Back to 1. with those file paths.

What does Webpack do?

Webpack is a single core application.

What happens if we parallelise the parsing, extracting and resolving of dependencies?

Multi-process node.js

// PARENT PROCESS.

const { fork } = require("child_process"); // 1

const childPath = join(__dirname, "../child/child-process-module");

const childProcess = fork(childPath); // 2

childProcess.on("message", (message) => handleChildProcessMessage(message)); // 3

// CHILD PROCESS.

process.on("message", handleParentMessage); // 3

process.send(message); // 3

Testing with a real code base

Salestrader: 1006 files.

1 core: 2.6s

2 core: 1.7s

about 30-35% speed up

Windows

1 core: 12s

2 core: 7.5s

4 core: 6s

Let's add babel-loader*

1 core: 19s

2 core: 12.5s

Conclusion

Multi process support could speed up Webpack builds by 30%+ and it does seem to scale with the # of physical cores.

Spike adding it to Webpack/creating loader/plugin?

https://github.com/webpack/webpack/issues/914

What can we do right now?

Another way to improve performance is to cache work (transforms).

https://github.com/mzgoddard/hard-source-webpack-plugin

With this applied sales trader goes from 17s to 11.5s on second build.

Parallel loaders

There is a plugin that allows parallel loader execution

https://github.com/webpack/webpack/issues/1810

https://github.com/amireh/happypack

Using it on babel-loader

First build: 18s!

Subsequent: 11.5s...

Concluding Conclusions

Multi-process can help.

Caching helps.

Babel is a slow blob monster.

There is nothing new under the sun.

https://github.com/webpack/webpack/issues/1810

https://github.com/webpack/webpack/issues/250

https://github.com/webpack/webpack/issues/1905