PkgMgmt & CfgMgmt have failed

Long live Docker?

Franz Pletz // @fpletz

Mayflower GmbH

The speaker's bias

  • Free Software fundamentalist
  • Functional Programming aficionado
  • Unix user for ~20 years 
  • "Full Stack Engineer"
    • Webdev for 4 years
    • Sysadmin for 10 years
    • Puppet for 6 years
  • Debian Maintainer
  • NixOS Security Team

Chapter 1

The sorry state of package management

Features of a Pkg Manager

  • codifies software build process
  • creates, distributes, fetches packages
  • ensures dependencies are met
  • manages package lifecycle & versioning 
  • ensure package integrity & authenticity
  • helps users install software
  • helps distributions manage metadata

OS-level pkg managers

  • BSDs: pkg / ports / pkgsrc
  • Debian: dpkg / apt
  • Red Hat: rpm / yum / DNF
  • Arch: pacman
  • Gentoo: emerge / portage

Common Challenges

  • different, difficult automations
  • lots of boilerplate code
  • managing state is hard
  • rollbacks are hard
  • dependency hells
  • reproducible builds
  • setting up repos is painful
  • distribution packaging policies

Solution Attempts

  • jordansissel/fpm
    • easy package creation
    • many package managers
  • omnibus packages
    • with all dependencies included
  • third-party package managers
    • checkinstall

PkgMgmt all the way down

  • programming language have lots of modules
  • most are not packaged by distributions
  • so languages started rolling their own…
    • Ruby: gem, bundler
    • Python: pip, wheel, virtualenv
    • Haskell: cabal
    • PHP: pear, pecl, composer
    • Node: npm, yarn
    • C/C++: conan
  • … and made it even worse!

Down the rabbit hole

  • language pkg managers generally not very mature
    • most don't even implement full SAT solvers :(
    • ​missing integrations in other package managers
  • lots of developer-provided packages
    • broken dependencies
    • bad versioning
    • no CI, no automatic testing
  • external library deps are not manageable

Trick Question

Say you have a really cool $hiplang app that uses an awesome web framework.

 

It depends on hundreds of $hiplang packages and some of those on C libraries.

 

How would you package that for your favorite OS to deploy it on your machines?

You
don't!
Docker plz!

Nope, not yet! ;)

Chapter 2

Config Management - aka "we'll get there eventually"

Once upon a time

  • machines were physical
  • multiple unrelated services per machine
  • sysadmins edited config files by hand
  • machines were caressed like babies
  • hostnames were nicknames
  • upgrades were done manually
  • sysadmins had feelings for their systems

Along comes the "Cloud"

  • lots of machines per physical system
  • machines easy to create & tear down
  • one logical service per machine
  • distribution of base or golden images
  • automated provisioning & configuration
  • thousands of machines

Increased operational workload meant new tools for configuration management and orchestration were needed!

Benefits of CfgMgmt

  • infrastructure as code
  • abstractions (roles, profiles)
  • testable & reviewable infrastructure
  • consistency
  • reproducibility
  • service dependencies

Traditional CfgMgmt Tools

  • CFEngine
  • Puppet
  • Chef
  • Ansible
  • Salt
  • Rudder

The issue with CfgMgmt

Traditional CfgMgmt tools mutate OS state.

Undeclared aspects of the target systems are undefined!

 

Implications

  • convergence to target state
  • eventual consistency

What would we need?

  • declarative configuration
  • OS-independent configuration
  • fully reproducible
  • idempotency
  • atomic upgrades

"You should use Docker!"

  • image concept provides necessary features
  • simple: one service per container
  • configuration in Dockerfiles not reproducible
  • needs external container scheduler
    • like kubernetes, docker swarm, etc.
  • can help a lot, but no silver bullet

Chapter 3

What we might use a few years from now

Application Containers

  • upstreams will provide container images
  • format needs standardization first
  • Pioneers:
    • appimage
    • flatpak
    • snap

More Tooling around Docker

  • standards for linking containers
  • "plug and play containers"
  • Pioneers:
    • habitat

s/orchestration/choreography/g

  • centralized deployments becomes distributed
  • basic idea: use raft/gossip from etcd & consul
  • infrasture configures itself by what's available
  • Pioneers:
    • kubernetes
    • mgmt

NixOS

  • declarative Linux Distribution
  • uses purely functional package manger Nix
  • premise: pkgmgmt == cfgmgmt
  • consistency through immutability
  • fully reproducible, pinned to one git commit
  • atomic upgrades & rollbacks
  • testing framework builtin
  • cloud deployments with nixops

Thanks!

Questions?

Package & Configuration Managed has failed

By fpletz

Package & Configuration Managed has failed

  • 689

More from fpletz