Monorepo

One ring repo to rule them all

MASMOVIL TECHNOLOGY

What?

monorepo !== monolith

Several projects stored in the same repository

Who?

TB

  • 4 million commits
  • ~1.3 million files
  • + 17M LOCs
  • +1000 commits per week
  • Linear history
  • .git ~15GB
  • index file ~191MB

Facebook monorepo in numbers

  • Ultra-large-scale system
  • +17 years
  • 35 Millions of commits
  • One billion files
  • Nine million unique source files
  • Two billion lines of code
  • 86TB of data
  • 25.000 engineers around the globe
  • 16,000 changes plus 24,000 by automated systems (since 2012)

Google monorepo in numbers

January 2015

Why?

MYRIAD OF REPOS

DEPENDENCY MANAGEMENT

Artifactory, nexus, private npm, ...

Communicate, update and test dependency releases

RELEASE MANAGEMENT

Release order

DEPRECATION PLANS

Backward support effort

Deprecation and cleanup plan

Advantages

Shared code

Models

Services

Automations

feat/branch-2

x.y.z-sta.1

x.y.z

master

feat/branch-1

hotfix/branch-1

release/1.y.z

PULL REQUEST!!!

x.y.z-sta.2

squash+merge

Every commit at master could become a release!

ATOMIC CHANGES

Front

BFF

BSS

Front

Cross refactors

Product Value defined by just 1 PR

Continuous integration

Risks

Permissions

Which permissions for in-house developers

Which permissions for external developers

Code Owners

CI | CD

Feature branches 2.0?

Value-focused testing

How to version

How to release

BUILDS

Incremental

Fast

TESTS

Incremental

Fast

Full regression?

TOOLING

Large scale builder

Code repository performance

Distributed Builder+cache?

Offline support?

FEATURE FLAGS

Culture

Tooling

Two-steps roll-outs

http://featureflags.io/

Cross teams

Maintain a healthy repo

Cross refactors/tasks

Tooling

Bazelfarian

  • Makefile as a contract between projects-devops
  • Bazel over makefiles
  • Less advantages
  • More feasible

Bazelful

  • Build+test process fully defined in bazel
  • Build performance
  • Boil the ocean

https://book.restafarians.com/

Partial Clone

https://www.git-scm.com/docs/partial-clone

git clone --depth 1 --no-checkout --filter=blob:none \
  "file://$(pwd)/server_repo" local_repo
cd local_repo
git checkout master -- mydir/
'--filter=blob:none' # omits all blobs.
--filter=blob:limit=<n>[kmg]' # omits blobs larger than n bytes
or units.  n may be zero.  The suffixes k, m, and g can be used to name
units in KiB, MiB, or GiB.  For example, 'blob:limit=1k' is the same
as 'blob:limit=1024'.

'--filter=sparse:oid=<blob-ish>' # uses a sparse-checkout
specification contained in the blob (or blob-expression) '<blob-ish>'
to omit blobs that would not be not required for a sparse checkout on
the requested refs.

'--filter=sparse:path=<path>' # similarly uses a sparse-checkout
specification contained in <path>.

'--filter=tree:<depth>' # omits all blobs and trees whose depth
from the root tree is >= <depth>

Roadmap

Whe are here

SERIALIZED

Makefile everything

Deterministic builds

Code Review culture

Trunk based development

Stack convergence

Performant builds

Performant tests

Incremental tests

New CI/CD paradigm

Merge repos

Feature branches 2.0

Monorepo party hard!!!

Bazelify everything

Take aways

open your mind

challenge status quo

References

https://www.perforce.com/products/helix-teamhub-enterprise

https://www.perforce.com/products/helix4git

https://github.com/korfuri/awesome-monorepo

https://github.com/jin/awesome-bazel

https://github.com/hectim/create-react-app-bazel

https://medium.com/@Jakeherringbone/you-too-can-love-the-monorepo-d95d1d6fcebe

http://www.syntaxsuccess.com/viewarticle/scalable-react-build-with-bazel

https://medium.com/@jorgeucano/entendiendo-bazel-377ef4063fa4

https://itsvit.com/blog/monorepo-google-way-ci-cd/

https://codeburst.io/develop-and-deploy-your-own-react-monorepo-app-in-under-2-hours-using-lerna-travis-and-now-2b140d647238

https://blog.box.com/blog/how-we-improved-webpack-build-performance-95

https://slack.engineering/keep-webpack-fast-a-field-guide-for-better-build-performance-f56a5995e8f1

https://webpack.js.org/configuration/other-options/#recordspath

https://webpack.js.org/guides/caching/

https://webpack.js.org/plugins/commons-chunk-plugin/#manifest-file

https://medium.com/onfido-tech/speed-up-webpack-ff53c494b89c

https://hackernoon.com/one-vs-many-why-we-moved-from-multiple-git-repos-to-a-monorepo-and-how-we-set-it-up-f4abb0cfe469

https://dl.acm.org/citation.cfm?id=2854146

https://developer.atlassian.com/blog/2015/10/monorepos-in-git/

https://medium.com/windmill-engineering/bazel-is-the-worst-build-system-except-for-all-the-others-b369396a9e26

https://docs.bazel.build/versions/master/remote-caching.html

https://code.fb.com/core-data/scaling-mercurial-at-facebook/

https://medium.com/zukame/why-monorepo-in-2018-89221acd4bfb

https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9

https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository/52269934#52269934

https://stackoverflow.com/questions/51462090/does-github-support-git-wire-protocol-v2/51462372#51462372

https://stackoverflow.com/questions/49917616/what-is-the-git-clone-filter-options-syntax

https://www.git-scm.com/docs/partial-clone

https://github.blog/changelog/2018-11-08-git-protocol-v2-support/

https://gitlab.com/gitlab-org/gitlab-ce/issues/46555#note_106736004

Monorepo

One ring repo to rule them all

MASMOVIL TECHNOLOGY

Q&A

@anthanh | #frontend-general | #tech-chat

Thanks!