multi-process-dependencies-graph

/salestrader [ct5-post-conversion|✔] 14:27 $ npm start

Version: webpack 2.3.3
Time: 57358ms
webpack: Compiled successfully.

 

/salestrader [ct5-post-conversion|✔] 14:28 $ npm start

Time: 16931ms
webpack: Compiled successfully.

Given

 

"/Users/briand/dev/salestrader/apps/salestrader/src/index.js"

 

  1. The file is processed by Webpack loaders.
  2. Their output is parsed to extract the files dependencies.
  3. The dependencies are resolved to file paths.
  4. Back to 1. with those file paths.

What does Webpack do?

Webpack is a single core application.

What happens if we parallelise the parsing, extracting and resolving of dependencies?

Multi-process node.js

// PARENT PROCESS.

const { fork } = require("child_process"); // 1

const childPath = join(__dirname, "../child/child-process-module");

const childProcess = fork(childPath); // 2

childProcess.on("message", (message) => handleChildProcessMessage(message)); // 3

// CHILD PROCESS.

process.on("message", handleParentMessage); // 3

process.send(message); // 3

Testing with a real code base

Salestrader: 1006 files.

 

1 core: 2.6s

2 core: 1.7s

 

about 30-35% speed up

 

Windows

 

1 core: 12s

2 core: 7.5s

4 core: 6s

Let's add babel-loader*

1 core: 19s

2 core: 12.5s

Conclusion

Multi process support could speed up Webpack builds by 30%+ and it does seem to scale with the # of physical cores.

 

Spike adding it to Webpack/creating loader/plugin?

 

https://github.com/webpack/webpack/issues/914

What can we do right now?

Another way to improve performance is to cache work (transforms).

 

https://github.com/mzgoddard/hard-source-webpack-plugin

 

With this applied sales trader goes from 17s to 11.5s on second build.

Parallel loaders

There is a plugin that allows parallel loader execution

 

https://github.com/webpack/webpack/issues/1810

 

https://github.com/amireh/happypack

 

Using it on babel-loader

 

First build: 18s!

Subsequent: 11.5s...

Concluding Conclusions

Multi-process can help.

 

Caching helps.

 

Babel is a slow blob monster.

 

There is nothing new under the sun.

 

https://github.com/webpack/webpack/issues/1810

https://github.com/webpack/webpack/issues/250

https://github.com/webpack/webpack/issues/1905

Made with Slides.com