Zen and the Art of Package Management

Zen and the Art of Package Management

First, Some Terms

Global Package Manager

Global Package Managers

• apt
• homebrew
• yum
• emerge

Global Package Managers

• apt
• homebrew
• yum
• emerge

but also

• rubygems
• pip

Local Package Manager

Local Package Managers

• bundler
• npm
• cargo
• yarn
• composer
• pub
• cocoapods
• swift

Isolation

• virtualenv
• gemsets
• flatpak
• snap
• docker
• chroot

Isolation

• virtualenv
• gemsets
• flatpak
• snap
• docker
• chroot

makes sense to use with global package managers

Non-Obvious Goals

• enable an ecosystem with rich dependency graphs
• allow the ecosystem to innovate in areas historically in the domain of giant standard libraries
• make it possible for common abstractions to be used as if they were stdlib, but versioned separately

Don't overrotate on left-pad

Cargo

Servo's cssparser

encoding reverse dependencies

copperline
cssparser
email
encoding_literals
escposify
fetch
formdata
gettext
hiirc
http-fetch

id3
irc
java-properties
loirc
mailparse
maman
maybe_utf8
mime_multipart
spread
strason

substudy
tendril
tiberius
timeline
tiny_http
tokei
tripcode
uncbv
unidiff
url

Rule of 2

By breaking up your code, you massively increase the total number of people who can work on the entirety of your project's code

Union of all use cases

>

Intersection of all use cases

Non-Obvious Goals

• enable an ecosystem with rich dependency graphs
• allow the ecosystem to innovate in areas historically in the domain of giant standard libraries
• make it possible for common abstractions to be used as if they were stdlib, but versioned separately
• to make this work, we need a great deal of predictability, which is largely what I'm talking about

A good local package manager...

is primarily a tool for humans to share and manage  workflows and secondarily a tool that can be integrated into continuous delivery systems

Shared Workflow is Deceptive

Cargo's built-in "profiles"

• release
• dev
• test
• bench
• doc

cargo build --release
cargo build
cargo test
cargo bench
cargo doc

pub struct Profile {
    pub opt_level: String,
    pub lto: bool,
    pub codegen_units: Option<u32>,
    pub rustc_args: Option<Vec<String>>,
    pub rustdoc_args: Option<Vec<String>>,
    pub debuginfo: bool,
    pub debug_assertions: bool,
    pub rpath: bool,
    pub test: bool,
    pub doc: bool,
    pub run_custom_build: bool,
    pub panic: Option<String>,
}

What's a "profile"?

You never have to think about that!

Cargo's built-in "profiles"

• release
• dev
• test
• bench
• doc

cargo build --release
cargo build
cargo test
cargo bench
cargo doc

[profile.dev]
opt-level = 0
debug = true
rpath = false
lto = false
debug-assertions = true
codegen-units = 1
panic = 'unwind'

configuring the profile

[profile.release]
opt-level = 3
debug = false
rpath = false
lto = false
debug-assertions = false
codegen-units = 1
panic = 'unwind'

dev default

release default

Developers still think in terms of the core workflows

A great local package manager...

coordinates updates through ecosystem-wide norms  requiring the use of semver
encourages the use of semver ranges (^), and discourages other kinds of ranges (~, >)

Why "hardcode" to Semver?

The truth is, anything will do.

We need a universal standard for describing compatibility in a way that can be understood by a dependency resolver

A stable ecosystem with rich dependency graphs

A good local package manager...

isolates projects on the same machine from each other automatically

manual namespacing

A good local package manager...

has a solution for optional dependencies

One Option

"Weak Linking"

try {
  var foo = require('foo')
  var fooVersion = require('foo/package.json').version
} catch (er) {
  foo = null
}
if ( notGoodFooVersion(fooVersion) ) {
  foo = null
}

// .. then later in your program ..

if (foo) {
  foo.doFooThings()
}

"Weak Linking" from the npm docs

Other solutions?

A great local package manager...

has a better solution for optional dependencies than weak linking and DIY

Cargo "Features"

[package]
name = "awesome"

[features]
# The default set of optional packages. Most people will want to use these
# packages, but they are strictly optional. Note that `session` is not a package
# but rather another feature listed in this manifest.
default = ["jquery", "uglifier", "session"]

# A feature with no dependencies is used mainly for conditional compilation,
# like `#[cfg(feature = "go-faster")]`.
go-faster = []

# The `secure-password` feature depends on the bcrypt package. This aliasing
# will allow people to talk about the feature in a higher-level way and allow
# this package to add more requirements to the feature in the future.
secure-password = ["bcrypt"]

# Features can be used to reexport features of other packages. The `session`
# feature of package `awesome` will ensure that the `session` feature of the
# package `cookie` is also enabled.
session = ["cookie/session"]

[dependencies]
# These packages are mandatory and form the core of this package’s distribution.
cookie = "1.2.0"
oauth = "1.1.0"
route-recognizer = "=2.1.0"

# A list of all of the optional dependencies, some of which are included in the
# above `features`. They can be opted into by apps.
jquery = { version = "1.0.2", optional = true }
uglifier = { version = "1.5.3", optional = true }
bcrypt = { version = "*", optional = true }
civet = { version = "*", optional = true }

example: declaring features

[dependencies.awesome]
version = "1.3.5"
default-features = false # do not include the default features, and optionally
                         # cherry-pick individual features
features = ["secure-password", "civet"]

using features

#[cfg(feature_name)]

Rationalized optional dependencies!

A good local package manager...

establishes conventions for units of code at the package level even when the underlying language only understands code at the module level

"Gem" Conventions

• meaning of dependencies
• compiling native extensions
• location of the library code
• location of any binaries
• cryptographical signatures

"Crate" Conventions

• meaning of dependencies
• compiling native code
• compiling and applying plugins
• location of the library code
• location of any binaries
• configuration of built-in profiles
• "optional features"
  • expose #[cfg(feature_name)]
  • define a group of optional dependencies 

A good local package manager...

reliably includes the same source code for each build
across machines and environments

Consider all potential dependencies from other profiles and platforms

A good local package manager...

supports temporary override of an application's dependencies for emergency fixes, upgrades or co-development

escape valves are critical for maintaining faith in the overall system and especially its reliance on semver compliance

apps need to ship
emergency fixes

upgrades sometimes require tricky coordination across an entire dependency graph and use of not-yet-released code

many developers co-develop secondary packages locally while working on dependent packages

all of these scenarios call for an override feature that can apply to indirect dependencies

A good local package manager...

attempts to unify dependency ranges into single packages when possible

moment.js

time.js ^1.1

pretty-time.js

time.js ^1.0

app.js

moment.js

time.js ^1.1

pretty-time.js

time.js ^1.0

app.js

time.js ~1.3.0

TLDR You will need to backtrack and as far as I know, any real version of the algorithm is NP complete

It's even harder with possible duplicates because deciding to duplicate vs. backtrack is a judgment call

But don't skimp! Correctness really matters here.

A great local package manager...

ensures that shared, public types are always unified between the packages that share them

function time(hour, minute, second) {
  return {
    hour: hour,
    minute: minute,
    second: second
  };
}

function format(time) {
  let { hour, minute, second } = time;
  return `${hour}:${minute}:${second}`;
}

case study: a simple time library

function time(hour, minute, second) {
  return {
    hour: hour,
    minute: minute,
    second: second
  };
}

function format(time) {
  return `${time.hour}:${time.minute}:${time.second}`;
}

time.js 1.0

moment.js

time.js 1.0

pretty-time.js

time.js 1.0

app.js

const moment = require('moment');
const pretty = require('pretty-date');

// returns a time.js 1.0
let now = moment.now();
// expects a time.js 1.0
let formatted = pretty.format(now);

app.js

function time(hour, minute, second) {
  return {
    hour: hour,
    minute: minute,
    second: second
  };
}

function format(time) {
  let { hour, minute, second } = time;
  return `${hour}:${minute}:${second}`;
}

time.js 1.0

moment.js

time.js 1.0

pretty-time.js

time.js 1.1

app.js

const moment = require('moment');
const pretty = require('pretty-date');

// returns a time.js 1.0
let now = moment.now();
// expects a time.js 1.0
let formatted = pretty.format(now);

app.js

moment.js

time.js 1.0

pretty-time.js

time.js 1.1

app.js

This problem is very general and affects almost every library that exposes data structures and functions that operate on them as part of their public API

YAGNI is tempting but wrong as this problem crops up in a huge number of diffuse ways