Zen and the Art of Package Management
Zen and the Art of Package Management




Build Higher
#TheDetailsMatter
First, Some Terms
Global Package Manager
Global Package Managers
- apt
- homebrew
- yum
- emerge
Global Package Managers
- apt
- homebrew
- yum
- emerge
but also
- rubygems
- pip
Local Package Manager
Local Package Managers
- bundler
- npm
- cargo
- yarn
- composer
- pub
- cocoapods
- swift








Isolation
- virtualenv
- gemsets
- flatpak
- snap
- docker
- chroot
Isolation
- virtualenv
- gemsets
- flatpak
- snap
- docker
- chroot
makes sense to use with global package managers
Local Package Managers
Today, we're talking about



I've worked on these:
Non-Obvious Goals
- enable an ecosystem with rich dependency graphs
- allow the ecosystem to innovate in areas historically in the domain of giant standard libraries
- make it possible for common abstractions to be used as if they were stdlib, but versioned separately
Don't overrotate on left-pad

Cargo
Servo's cssparser

encoding reverse dependencies
copperline
cssparser
email
encoding_literals
escposify
fetch
formdata
gettext
hiirc
http-fetch
id3
irc
java-properties
loirc
mailparse
maman
maybe_utf8
mime_multipart
spread
strason
substudy
tendril
tiberius
timeline
tiny_http
tokei
tripcode
uncbv
unidiff
url
Rule of 2
By breaking up your code, you massively increase the total number of people who can work on the entirety of your project's code
Union of all use cases
>
Intersection of all use cases
Non-Obvious Goals
- enable an ecosystem with rich dependency graphs
- allow the ecosystem to innovate in areas historically in the domain of giant standard libraries
- make it possible for common abstractions to be used as if they were stdlib, but versioned separately
- to make this work, we need a great deal of predictability, which is largely what I'm talking about
A good local package manager...
A good local package manager...
is primarily a tool for humans to share and manage workflows and secondarily a tool that can be integrated into continuous delivery systems
Shared Workflow is Deceptive
Cargo's built-in "profiles"
- release
- dev
- test
- bench
- doc
cargo build --release cargo build cargo test cargo bench cargo doc
pub struct Profile {
pub opt_level: String,
pub lto: bool,
pub codegen_units: Option<u32>,
pub rustc_args: Option<Vec<String>>,
pub rustdoc_args: Option<Vec<String>>,
pub debuginfo: bool,
pub debug_assertions: bool,
pub rpath: bool,
pub test: bool,
pub doc: bool,
pub run_custom_build: bool,
pub panic: Option<String>,
}
What's a "profile"?
You never have to think about that!
Cargo's built-in "profiles"
- release
- dev
- test
- bench
- doc
cargo build --release cargo build cargo test cargo bench cargo doc
[profile.dev]
opt-level = 0
debug = true
rpath = false
lto = false
debug-assertions = true
codegen-units = 1
panic = 'unwind'
configuring the profile
[profile.release]
opt-level = 3
debug = false
rpath = false
lto = false
debug-assertions = false
codegen-units = 1
panic = 'unwind'
dev default
release default
Developers still think in terms of the core workflows
A great local package manager...
coordinates updates through ecosystem-wide norms requiring the use of semver
encourages the use of semver ranges (^), and discourages other kinds of ranges (~, >)
Why "hardcode" to Semver?
The truth is, anything will do.
We need a universal standard for describing compatibility in a way that can be understood by a dependency resolver
A stable ecosystem with rich dependency graphs
A good local package manager...
isolates projects on the same machine from each other automatically



manual namespacing
A good local package manager...
has a solution for optional dependencies
One Option
"Weak Linking"
try {
var foo = require('foo')
var fooVersion = require('foo/package.json').version
} catch (er) {
foo = null
}
if ( notGoodFooVersion(fooVersion) ) {
foo = null
}
// .. then later in your program ..
if (foo) {
foo.doFooThings()
}
"Weak Linking" from the npm docs
foo is optional
Other solutions?
A great local package manager...
has a better solution for optional dependencies than weak linking and DIY
Cargo "Features"
[package]
name = "awesome"
[features]
# The default set of optional packages. Most people will want to use these
# packages, but they are strictly optional. Note that `session` is not a package
# but rather another feature listed in this manifest.
default = ["jquery", "uglifier", "session"]
# A feature with no dependencies is used mainly for conditional compilation,
# like `#[cfg(feature = "go-faster")]`.
go-faster = []
# The `secure-password` feature depends on the bcrypt package. This aliasing
# will allow people to talk about the feature in a higher-level way and allow
# this package to add more requirements to the feature in the future.
secure-password = ["bcrypt"]
# Features can be used to reexport features of other packages. The `session`
# feature of package `awesome` will ensure that the `session` feature of the
# package `cookie` is also enabled.
session = ["cookie/session"]
[dependencies]
# These packages are mandatory and form the core of this package’s distribution.
cookie = "1.2.0"
oauth = "1.1.0"
route-recognizer = "=2.1.0"
# A list of all of the optional dependencies, some of which are included in the
# above `features`. They can be opted into by apps.
jquery = { version = "1.0.2", optional = true }
uglifier = { version = "1.5.3", optional = true }
bcrypt = { version = "*", optional = true }
civet = { version = "*", optional = true }
example: declaring features
[dependencies.awesome]
version = "1.3.5"
default-features = false # do not include the default features, and optionally
# cherry-pick individual features
features = ["secure-password", "civet"]
using features
#[cfg(feature_name)]
Rationalized optional dependencies!
A good local package manager...
establishes conventions for units of code at the package level even when the underlying language only understands code at the module level
"Gem" Conventions
- meaning of dependencies
- compiling native extensions
- location of the library code
- location of any binaries
- cryptographical signatures
"Crate" Conventions
- meaning of dependencies
- compiling native code
- compiling and applying plugins
- location of the library code
- location of any binaries
- configuration of built-in profiles
- "optional features"
- expose #[cfg(feature_name)]
- define a group of optional dependencies
A good local package manager...
reliably includes the same source code for each build
across machines and environments
Consider all potential dependencies from other profiles and platforms
A great local package manager...
minimizes changes to unrelated packages when updating a single package
A great local package manager...
caches as much work as possible between projects
without sacrificing isolation
A good local package manager...
supports temporary override of an application's dependencies for emergency fixes, upgrades or co-development
escape valves are critical for maintaining faith in the overall system and especially its reliance on semver compliance
apps need to ship
emergency fixes
upgrades sometimes require tricky coordination across an entire dependency graph and use of not-yet-released code
many developers co-develop secondary packages locally while working on dependent packages
all of these scenarios call for an override feature that can apply to indirect dependencies
A great local package manager...
distinguishes between applications temporarily overriding a dependency for:
- emergency fixes
- upgrades
- co-development
A good local package manager...
attempts to unify dependency ranges into single packages when possible
moment.js
time.js ^1.1
pretty-time.js
time.js ^1.0
app.js
should unify into...
time.js 1.latest
moment.js
time.js ^1.1
pretty-time.js
time.js ^1.0
app.js
should unify into...
time.js 1.3.latest
time.js ~1.3.0
TLDR You will need to backtrack and as far as I know, any real version of the algorithm is NP complete
It's even harder with possible duplicates because deciding to duplicate vs. backtrack is a judgment call
But don't skimp! Correctness really matters here.
A great local package manager...
ensures that shared, public types are always unified between the packages that share them
function time(hour, minute, second) {
return {
hour: hour,
minute: minute,
second: second
};
}
function format(time) {
let { hour, minute, second } = time;
return `${hour}:${minute}:${second}`;
}
case study: a simple time library
function time(hour, minute, second) {
return {
hour: hour,
minute: minute,
second: second
};
}
function format(time) {
return `${time.hour}:${time.minute}:${time.second}`;
}
time.js 1.0
moment.js
time.js 1.0
pretty-time.js
time.js 1.0
app.js
const moment = require('moment');
const pretty = require('pretty-date');
// returns a time.js 1.0
let now = moment.now();
// expects a time.js 1.0
let formatted = pretty.format(now);
app.js
a time 1.0 is passed from moment.js to pretty-time.js through app.js
function time(hour, minute, second) {
return {
hour: hour,
minute: minute,
second: second
};
}
function format(time) {
let { hour, minute, second } = time;
return `${hour}:${minute}:${second}`;
}
time.js 1.0
moment.js
time.js 1.0
pretty-time.js
time.js 1.1
app.js
const moment = require('moment');
const pretty = require('pretty-date');
// returns a time.js 1.0
let now = moment.now();
// expects a time.js 1.0
let formatted = pretty.format(now);
app.js
a time 1.0 is still passed from moment.js to pretty-time.js through app.js
function time(hour, minute, second, tz) {
return {
hour: hour,
minute: minute,
second: second,
tz: tz
};
}
function format(time) {
let { hour, minute, second, tz } = time;
return `${hour}:${minute}:${second} ${tz}Z`;
}
time.js 1.1
which will get stringified incorrectly due to missing tz
pretty-time updates its time.js dependency
a new non-breaking version of time.js is released
moment.js
time.js 1.0
pretty-time.js
time.js 1.1
app.js
the structure of this POJO
is shared with this package
through this common parent
This problem is very general and affects almost every library that exposes data structures and functions that operate on them as part of their public API
YAGNI is tempting but wrong as this problem crops up in a huge number of diffuse ways

Build Higher
deck
By Yehuda Katz