Anthanh PRO
I ♥ the web, technologies and....beers! Co-founder of https://etereo.io
Monorepo
One ring repo to rule them all
MASMOVIL TECHNOLOGY
What?
monorepo !== monolith
Who?
TB
- 4 million commits
- ~1.3 million files
- + 17M LOCs
- +1000 commits per week
- Linear history
- .git ~15GB
- index file ~191MB
Facebook monorepo in numbers
- Ultra-large-scale system
- +17 years
- 35 Millions of commits
- One billion files
- Nine million unique source files
- Two billion lines of code
- 86TB of data
- 25.000 engineers around the globe
- 16,000 changes plus 24,000 by automated systems (since 2012)
Google monorepo in numbers
Why?
MYRIAD OF REPOS
DEPENDENCY MANAGEMENT
Artifactory, nexus, private npm, ...
Communicate, update and test dependency releases
RELEASE MANAGEMENT
Release order
DEPRECATION PLANS
Backward support effort
Deprecation and cleanup plan
Advantages
Shared code
Models
Services
Automations
feat/branch-2
x.y.z-sta.1
x.y.z
master
feat/branch-1
hotfix/branch-1
release/1.y.z
PULL REQUEST!!!
x.y.z-sta.2
squash+merge
Every commit at master could become a release!
ATOMIC CHANGES
Front
BFF
BSS
Front
Cross refactors
Product Value defined by just 1 PR
Continuous integration
Risks
Permissions
Which permissions for in-house developers
Which permissions for external developers
Code Owners
CI | CD
Feature branches 2.0?
Value-focused testing
How to version
How to release
BUILDS
Incremental
Fast
TESTS
Incremental
Fast
Full regression?
TOOLING
Large scale builder
Code repository performance
Distributed Builder+cache?
Offline support?
FEATURE FLAGS
Culture
Tooling
Two-steps roll-outs
Cross teams
Maintain a healthy repo
Cross refactors/tasks
Tooling
Bazelfarian
- Makefile as a contract between projects-
devops - Bazel over makefiles
-
Less advantages - More feasible
Bazelful
- Build+test process fully defined in bazel
- Build performance
- Boil the ocean
https://book.restafarians.com/
Partial Clone
git clone --depth 1 --no-checkout --filter=blob:none \
"file://$(pwd)/server_repo" local_repo
cd local_repo
git checkout master -- mydir/'--filter=blob:none' # omits all blobs.
--filter=blob:limit=<n>[kmg]' # omits blobs larger than n bytes
or units. n may be zero. The suffixes k, m, and g can be used to name
units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same
as 'blob:limit=1024'.
'--filter=sparse:oid=<blob-ish>' # uses a sparse-checkout
specification contained in the blob (or blob-expression) '<blob-ish>'
to omit blobs that would not be not required for a sparse checkout on
the requested refs.
'--filter=sparse:path=<path>' # similarly uses a sparse-checkout
specification contained in <path>.
'--filter=tree:<depth>' # omits all blobs and trees whose depth
from the root tree is >= <depth>Roadmap
Whe are here
SERIALIZED
Makefile everything
Deterministic builds
Code Review culture
Trunk based development
Stack convergence
Performant builds
Performant tests
Incremental tests
New CI/CD paradigm
Merge repos
Feature branches 2.0
Monorepo party hard!!!
Bazelify everything
Take aways
open your mind
challenge status quo
References
https://www.perforce.com/products/helix-teamhub-enterprise
https://www.perforce.com/products/helix4git
https://github.com/korfuri/awesome-monorepo
https://github.com/jin/awesome-bazel
https://github.com/hectim/create-react-app-bazel
https://medium.com/@Jakeherringbone/you-too-can-love-the-monorepo-d95d1d6fcebe
http://www.syntaxsuccess.com/viewarticle/scalable-react-build-with-bazel
https://medium.com/@jorgeucano/entendiendo-bazel-377ef4063fa4
https://itsvit.com/blog/monorepo-google-way-ci-cd/
https://codeburst.io/develop-and-deploy-your-own-react-monorepo-app-in-under-2-hours-using-lerna-travis-and-now-2b140d647238
https://blog.box.com/blog/how-we-improved-webpack-build-performance-95
https://slack.engineering/keep-webpack-fast-a-field-guide-for-better-build-performance-f56a5995e8f1
https://webpack.js.org/configuration/other-options/#recordspath
https://webpack.js.org/guides/caching/
https://webpack.js.org/plugins/commons-chunk-plugin/#manifest-file
https://medium.com/onfido-tech/speed-up-webpack-ff53c494b89c
https://hackernoon.com/one-vs-many-why-we-moved-from-multiple-git-repos-to-a-monorepo-and-how-we-set-it-up-f4abb0cfe469
https://dl.acm.org/citation.cfm?id=2854146
https://developer.atlassian.com/blog/2015/10/monorepos-in-git/
https://medium.com/windmill-engineering/bazel-is-the-worst-build-system-except-for-all-the-others-b369396a9e26
https://docs.bazel.build/versions/master/remote-caching.html
https://code.fb.com/core-data/scaling-mercurial-at-facebook/
https://medium.com/zukame/why-monorepo-in-2018-89221acd4bfb
https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9
https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository/52269934#52269934
https://stackoverflow.com/questions/51462090/does-github-support-git-wire-protocol-v2/51462372#51462372
https://stackoverflow.com/questions/49917616/what-is-the-git-clone-filter-options-syntax
https://www.git-scm.com/docs/partial-clone
https://github.blog/changelog/2018-11-08-git-protocol-v2-support/
https://gitlab.com/gitlab-org/gitlab-ce/issues/46555#note_106736004
Monorepo
One ring repo to rule them all
MASMOVIL TECHNOLOGY
Q&A
@anthanh | #frontend-general | #tech-chat
Thanks!
MásMóvil Monorepo, One repo to rule them all
By Anthanh
