F1LT3R

https://F1LT3R.io (my blog)

Alistair MacDonald

Senior Software Architect

           @ Houghton Mifflin Harcourt

HMH / What we do

Interested in working with us?

mailto: alistair.macdonald@hmhco.com

What is a Monorepo?

 A monorepo (syllabic abbreviation of monolithic repository) is a software development strategy where code for many projects are stored in the same repository. (Wikipeda)

But why?

Deps Get Complex, Quick

React Monorepos

Why is HMH adopting a monorepo?

  • One way of developing react apps as a company
  • Shared, reusable components
  • Shared, reusable architecture
  • Shared, reusable configuration
  • No cross referenced PRs
  • Simplify upgrade path across many apps

Killing Innovation?

Killing Innovation?

No.

Less time

  • Setting up new apps, pipelines, capabilities etc.

 

🚀 More time

  • ​Creating new experiences
  • Contributing to common architecture
    (any engineer can commit contribute to the architecture to the monorepo)

Best Practices

  • Best Practices are great ideals
    • Under stress we fall back to complacency
  • Tie ourselves to the ship
    • Codify best practices
    • Tooling/gating around best practices
    • Lock iteration of product to quality gates
  • Continuous Integration
    • "Move fast with stable infra" - Zuck

... are great ideals

Where are we?

Since March 2019

  •       1x  🐉   Lerna  +  🐈  Yarn Workspaces Monorepo
  •       7x  ⚛️   CRA App's (Create-React-App)
  •   100x  🔬   Components
  •       5x  🏫   Component Libraries
  •       5x  📖   Storybooks - rollup the component libraries
  •     18x  👷   Engineers - actively contributing
  •       4x  👥   Teams

React Workspaces

React Workspaces

How does

it work?

Answer:

Turtles

(sort of)

React-Scripts

Fork

Insight: we can use the same create-react-app config across every React part of our Monorepo.
(Its turtles all the way down).

--scripts-version

create-react-app

create-react-app --scripts-version @your-org/react-scripts your-app

A little-known feature of create-react-app lets you specify your own custom version of react-scripts.

package.json

<workspaces-root>

{
  "workspaces": {
    "packages": [
      "packages/apps/*",
      "packages/components",
      "packages/storybook"
    ],
    "production": true,
    "development": true,
    "package-entry": "main:src"
  }
}

webpack.config.js

(react-scripts fork)

--- a/./facebook/react-scripts/config/webpack.config.js
+++ b/react-workspaces/react-scripts/config/webpack.config.js
@@ -9,7 +9,6 @@
'use strict';

const fs = require('fs');
const isWsl = require('is-wsl');
const path = require('path');
const webpack = require('webpack');
const resolve = require('resolve');
@@ -28,15 +27,14 @@ const WatchMissingNodeModulesPlugin = require('react-dev-utils/WatchMissingNodeM
const ModuleScopePlugin = require('react-dev-utils/ModuleScopePlugin');
const getCSSModuleLocalIdent = require('react-dev-utils/getCSSModuleLocalIdent');
const paths = require('./paths');
const modules = require('./modules');
+const workspaces = require('./workspaces');
const getClientEnvironment = require('./env');
const ModuleNotFoundPlugin = require('react-dev-utils/ModuleNotFoundPlugin');
const ForkTsCheckerWebpackPlugin = require('react-dev-utils/ForkTsCheckerWebpackPlugin');
const typescriptFormatter = require('react-dev-utils/typescriptFormatter');
// @remove-on-eject-begin
const getCacheIdentifier = require('react-dev-utils/getCacheIdentifier');
// @remove-on-eject-end

// Source maps are resource heavy and can cause out of memory issue for large source files.
const shouldUseSourceMap = process.env.GENERATE_SOURCEMAP !== 'false';
@@ -53,12 +51,22 @@ const cssModuleRegex = /\.module\.css$/;
const sassRegex = /\.(scss|sass)$/;
const sassModuleRegex = /\.module\.(scss|sass)$/;

+const workspacesConfig = workspaces.init(paths);
+
// This is the production and development configuration.
// It is focused on developer experience, fast rebuilds, and a minimal bundle.
module.exports = function(webpackEnv) {
  const isEnvDevelopment = webpackEnv === 'development';
  const isEnvProduction = webpackEnv === 'production';

+  const workspacesMainFields = [workspacesConfig.packageEntry, 'main'];
+  const mainFields =
+    isEnvDevelopment && workspacesConfig.development
+      ? workspacesMainFields
+      : isEnvProduction && workspacesConfig.production
+        ? workspacesMainFields
+        : undefined;
+
  // Webpack uses `publicPath` to determine where the app is being served from.
  // It requires a trailing slash, or the file assets will get an incorrect path.
  // In development, we always serve from the root. This makes config easier.
@@ -279,6 +282,7 @@ module.exports = function(webpackEnv) {
      extensions: paths.moduleFileExtensions
        .map(ext => `.${ext}`)
        .filter(ext => useTypeScript || !ext.includes('ts')),
+      mainFields,
      alias: {
        // Support React Native Web
        // https://www.smashingmagazine.com/2016/08/a-glimpse-into-the-future-with-react-native-for-web/
@@ -330,7 +335,11 @@ module.exports = function(webpackEnv) {
              loader: require.resolve('eslint-loader'),
            },
          ],
-          include: paths.appSrc,
+          include: isEnvDevelopment && workspacesConfig.development
+          ? [paths.appSrc, workspacesConfig.paths]
+          : isEnvProduction && workspacesConfig.production
+            ? [paths.appSrc, workspacesConfig.paths]
+            : paths.appSrc,
        },
        {
          // "oneOf" will traverse all following loaders until one will
@@ -352,7 +361,12 @@ module.exports = function(webpackEnv) {
            // The preset includes JSX, Flow, TypeScript, and some ESnext features.
            {
              test: /\.(js|mjs|jsx|ts|tsx)$/,
-              include: paths.appSrc,
+              include:
+                isEnvDevelopment && workspacesConfig.development
+                  ? [paths.appSrc, workspacesConfig.paths]
+                  : isEnvProduction && workspacesConfig.production
+                    ? [paths.appSrc, workspacesConfig.paths]
+                    : paths.appSrc,
              loader: require.resolve('babel-loader'),
              options: {
                customize: require.resolve(

yarn-workspaces.js

(react-scripts fork)

'use strict';

const fse = require('fs-extra');
const path = require('path');
const findUp = require('find-up');
const glob = require('glob');

const loadPackageJson = packagePath => {
	try {
		const packageObj = fse.readJsonSync(packagePath);
		return packageObj;
	} catch (err) {
		throw err;
	}
};

const getWorkspacesRootConfig = dir => {
	const packageJsonUp = findUp.sync('package.json', {cwd: dir});

	if (packageJsonUp === null) {
		return false;
	}

	const packageObj = loadPackageJson(packageJsonUp);

	if (Reflect.has(packageObj, 'workspaces')) {
		const workspacesRootConfig = {
			root: path.dirname(packageJsonUp),
			workspaces: packageObj.workspaces
		};
		return workspacesRootConfig;
	}

	const dirUp = path.dirname(dir);
	return getWorkspacesRootConfig(dirUp);
};

const getPackagePaths = (root, workspacesList) => {
	const packageList = [];

	workspacesList.forEach(workspace => {
		const workspaceDir = path.dirname(workspace);
		const workspaceAbsDir = path.join(root, workspaceDir);
		const packageJsonGlob = path.join('**!(node_modules)', 'package.json');
		const packageJsonAbsPaths = glob
			.sync(packageJsonGlob, {cwd: workspaceAbsDir})
			.map(pkgPath => path.join(workspaceAbsDir, pkgPath));

		packageList.push(...packageJsonAbsPaths);
	});

	return packageList;
};

const getDeep = (obj, keyChain) => {
	const nextKey = keyChain.shift();
	const has = Reflect.has(obj, nextKey);
	const val = obj[nextKey];

	if (keyChain.length === 0) {
		return val;
	}

	if (has) {
		return getDeep(val, keyChain);
	}

	return false;
};

const resolveBabelLoaderPaths = ({root, workspacesList}, packageEntry) => {
	const packageJsonPaths = getPackagePaths(root, workspacesList);
	const babelLoaderPaths = [];

	packageJsonPaths.map(absPkgPath => {
		const packageJson = loadPackageJson(absPkgPath);
		const mainSrcFile = getDeep(packageJson, [packageEntry]);

		if (mainSrcFile) {
			const mainSrcPath = path.dirname(mainSrcFile);
			const packageAbsDir = path.dirname(absPkgPath);
			const absSrcPath = path.join(packageAbsDir, mainSrcPath);
			babelLoaderPaths.push(absSrcPath);
		}
	});

	return babelLoaderPaths;
};

const loadAppSettings = appPackageJson => {
	const result = {workspaces: {}, dependencies: {}};

	const appPackageObj = loadPackageJson(appPackageJson);

	const dependencies = getDeep(appPackageObj, ['dependencies']);
	const devDependencies = getDeep(appPackageObj, ['devDependencies']);

	if (!dependencies && !devDependencies) return result;

	if (dependencies) {
		result.dependencies = Object.assign(result.dependencies, dependencies);
	}

	if (devDependencies) {
		result.dependencies = Object.assign(
			result.dependencies,
			devDependencies
		);
	}

	const reactScripts = getDeep(appPackageObj, ['react-scripts']);
	if (!reactScripts) return result;

	const workspaces = getDeep(reactScripts, ['workspaces']);
	result.workspaces = workspaces;
	if (!workspaces) return result;

	return workspaces;
};

const guard = (appDirectory, appPackageJson) => {
	if (!appDirectory) {
		throw new Error('appDirectory not provided');
	}

	if (typeof appDirectory !== 'string') {
		throw new Error('appDirectory should be a string');
	}

	if (!appPackageJson) {
		throw new Error('appPackageJson not provided');
	}

	if (typeof appPackageJson !== 'string') {
		throw new Error('appPackageJson should be a string');
	}
};

const getPkg = path => {
	const pkgPath = findUp.sync('package.json', {cwd: path});
	const pkg = loadPackageJson(pkgPath);
	return pkg;
};

const getDeps = pkg => {
	const deps = getDeep(pkg, ['dependencies']);
	const devDeps = getDeep(pkg, ['devDependencies']);

	let dependencies = {};

	if (deps) {
		dependencies = Object.assign(dependencies, deps);
	}

	if (devDeps) {
		dependencies = Object.assign(dependencies, devDeps);
	}

	return dependencies;
};

const depsTable = {};

const filterDeps = deps =>
	Reflect.ownKeys(deps).filter(dep => Reflect.has(depsTable, dep));

const filterDepsTable = () => {
	Reflect.ownKeys(depsTable).forEach(depName => {
		const depsList = depsTable[depName].deps;
		const workspacesOnlyDeps = filterDeps(depsList);
		depsTable[depName].deps = workspacesOnlyDeps;
	});
};

const buildDepsTable = srcPaths => {
	srcPaths.forEach(path => {
		const pkg = getPkg(path);
		const name = pkg.name;
		const deps = getDeps(pkg);
		depsTable[name] = {path, deps};
	});
};

const filterSrcPaths = (srcPaths, dependencies) => {
	const filteredPaths = [];

	srcPaths.forEach(path => {
		const pkg = getPkg(path);

		if (dependencies && Reflect.has(dependencies, pkg.name)) {
			filteredPaths.push(path);

			const subDeps = depsTable[pkg.name].deps;
			const subPaths = filterSrcPaths(srcPaths, subDeps);
			filteredPaths.push(...subPaths);
		}
	});

	return filteredPaths;
};

const init = paths => {
	guard(paths.appPath, paths.appPackageJson);

	const config = {
		root: null,
		paths: [],
		packageEntry: 'main:src',
		development: true,
		production: true
	};

	const {root, workspaces} = getWorkspacesRootConfig(paths.appPath);
	const workspacesList = [];

	// Normally "workspaces" in package.json is an array
	if (Array.isArray(workspaces)) {
		workspacesList.push(...workspaces);
	}

	// Sometimes "workspaces" in package.json is an object
	// with a ".packages" sub-array, eg: when used with "nohoist"
	// See: https://yarnpkg.com/blog/2018/02/15/nohoist
	if (workspaces && !Array.isArray(workspaces) && Reflect.has(workspaces, 'packages')) {
		workspacesList.push(...workspaces.packages);
	}

	if (workspacesList.length === 0) {
		return config;
	}
	console.log('Yarn Workspaces paths detected.');
	config.root = root;

	const appSettings = loadAppSettings(paths.appPackageJson);

	if (Reflect.has(appSettings.workspaces, 'development')) {
		config.development = appSettings.workspaces.development ? true : false;
	}

	if (Reflect.has(appSettings.workspaces, 'production')) {
		config.production = appSettings.workspaces.production ? true : false;
	}

	if (Reflect.has(appSettings.workspaces, 'package-entry')) {
		config.packageEntry = appSettings.workspaces['package-entry'];
	}

	const babelSrcPaths = resolveBabelLoaderPaths(
		{root, workspacesList},
		config.packageEntry
	);

	buildDepsTable(babelSrcPaths);

	const applicableSrcPaths = [...new Set(filterSrcPaths(
		babelSrcPaths,
		appSettings.dependencies
	))];

	console.log(
		`Found ${babelSrcPaths.length} path(s) with "${
			config.packageEntry
		}" entry.`
	);

	if (applicableSrcPaths.length > 0) {
		config.paths.push(...applicableSrcPaths);
	}

	console.log('Exporting Workspaces config to Webpack.');
	console.log(config);
	return config;
};

module.exports = {
	init
};

Building on

custom react-scripts

at HMH

React-Client-Common

  • Tour of our React monorepo structure

    • codeowners

    • ​rcc-config in package.json

    • settings, scripts, media

    • packages: suites, apps, components, storybooks

    • build pipes: jenkins, maven

  • RCC Menu: yarn menu

  • SGM Storybook

  • Demo app: sgm/app-ri

React-Workspaces

patterns & features

@HMH

No Local Builds

features

  • All builds happen in Jenkins.

  • No transpiled code lives on developer machines.

  • Components are only built when transpiled with an app.

CRA Developer Experience

features

  • For all components and apps:

    • CRA Hot reloading

    • CRA Linting

    • CRA Open browser bugs/warnings in your IDE

Relevant Source Only

features

  • Babel-Loader Paths
    @react-workspaces/react-scripts builds your app's dependency tree and adds *only* the relevant paths to babel-loader in Webpack; so you only lint and transpile source your app uses.

     

  • Tree-Shaking
    Only the code in your app's dependency tree will be transpiled in CI when your app is built and deployed.

Trunk Based Development

features

Trunk Based Development

features

Say

"No!"

to

Git-Flow

Fix Forward

features

don't roll back

fix forward!

React Workspaces: Roadmap

 

  • Continue to stay current with create-react-app

  • Dependency tree mapping -
        tree of component & app dependencies

  • Know what changed?  glorified git diff

  • Only test what changed

  • Only build & deploy changed apps in CI

    • yarn-workspaces upgrade required

  • Only build & deploy changed apps in CI

    • ​(currently we build all jobs on PR/merge &  promote to prod. manually)

  • Support TypeScript, Flow, etc.

  • PR react-scripts fork back into create-react-app (& kill this project)

[the END]

questions?

React Workspaces

By Alistair MacDonald (f1lt3r)