Rethinking Packaging, Development and Deployment 



Domen Kožar / @iElectric

Nix / Python / JavaScript consultant


PyCon Ukraine 2014

Issues with packaging in Python



  • dynamic packaging metadata: setup.py
    • solution: upcoming PEP-0426

  • legacy infrastructure tools: distutils
    • solution: PyPA

  • non-Python dependencies
    • solution: ?

Best tool for the job?

JavaScript stack is inevitable in 2014

Stateful email verification

  1. Generate a random key
  2. Store the key in our database
  3. Send email to the user with a link to our website containing the key
  4. Once user has clicked the link in the email, check that the random key matches the one in database and hence we are sure our user has access to that mailbox
  5. Remove the key from the database

Stateless email verification

  1. Take email of the user and sign it cryptographically (using HMAC) 
    from itsdangerous import TimestampSigner

s = TimestampSigner('secret-key')
hash = s.sign('me@example.com')
  2. Send email to the user with the link to our website containing the signature 
    send_mail('http://example.com/email/verify/' + hash)
  3. Once user has clicked the link in the email, verify digital signature with our private key and if it's valid, we confirm his/hers email 
    s.unsign(hash, max_age=24 * 3600)

Stateful packaging



Stateless packaging


Nix project

  • minimalistic purely functional language (configuration syntax + lambda functions)
  • lazy evaluated, dynamically typed
  • runs on POSIX (Linux / OS X / FreeBSD)
  • (could work on Windows, if there was enough interest)
  • solves dependency hell set of problems
  • developed by Eelco Dolstra as part of his PhD thesis in 2006, Utrecht, The Netherlands
  • 11 years old project
    •

Purely functional language

Software package is an output of a function that is deterministic (it depends only on the function inputs, without any side effects)


/nix/store/r8vvq9kq18pz08v249h8my6r9vs7s0n3-nginx-1.5.13/


  • /nix/store is immutable (mounted read-only)
  • r8vvq9kq18pz08v249h8my6r9vs7s0n3 is a SHA1 hash of the package function inputs (packaging metadata)


nginx package

{ stdenv, fetchurl, openssl, zlib, pcre, libxml2, libxslt, expat }:

stdenv.mkDerivation rec {
  name = "nginx-${version}";
  version = "1.5.13";

  src = fetchurl {
    url = "http://nginx.org/download/nginx-${version}.tar.gz";
    sha256 = "1f82845mpgmhvm151fhn2cnqjggw9w7cvsqbva9rb320wmc9m63w";
  };

  buildInputs = [ openssl zlib pcre libxml2 libxslt ];
  configureFlags = [ "--with-http_spdy_module" ];

  meta = with stdenv.lib; {
    description = "A reverse proxy and lightweight webserver";
    maintainers = [ maintainers.iElectric ];
    platforms = platforms.all;
    license = licenses.bsd2;
  };
}

Lazy language (hackability)


      Override lambda function parameters for the file:
myNginx = pkgs.nginx.override { openssl = openssl101h; }

      Override derivation attributes:
myNginx = pkgs.lib.overrideDerivation pkgs.nginx (args:
  src = fetchgit { ... }; 
})


Installing Nix

$ bash <(curl https://nixos.org/nix/install)
$ source ~/.nix-profile/etc/profile.d/nix.sh


Uninstalling Nix

$ rm -rf /nix
$ rm -rf ~/nix-profile/

 

Profiles / User Environment

 

Demo

Packaging Python software with Nix

{ fetchurl, buildPythonPackage, freetype, libjpeg, zlib, libtiff, libwebp }:

buildPythonPackage rec {
  name = "Pillow-2.3.0";

  src = fetchurl {
    url = "http://pypi.python.org/packages/source/P/Pillow/${name}.zip";
    sha256 = "0pzm0qk5ilqhwz74pydg1jwrds27vm47185dakdrxidb5bv3b5ia";
  };

  buildInputs = [ freetype libjpeg zlib libtiff libwebp ];
 
  meta = with stdenv.lib; {
    homepage = http://python-imaging.github.com/Pillow;
    description = "Fork of The Python Imaging Library (PIL)";
    license = "http://www.pythonware.com/products/pil/license.htm";
    maintainers = [ maintainers.iElectric ];
    platforms = platforms.linux;
  };
};

Ways to package Python packages for Nix


    1. manually (cumbersome, if you have lots of packages)
    2. python2nix (spits out simple templates)
    3. pypi2nix (tries to handle everything automatically)

    nix-shell

    (virtualenv, but for all software packages)



    Using nix-shell to provide tools

    $ cat default.nix
    with import <nixpkgs> {};
with pkgs.python27Packages; 

buildPythonPackage {
  name = "my-env";

  buildInputs = [ virtualenv pkgs.zlib pkgs.libjpeg pkgs.libxml2 pkgs.git ];

  src = null;
}

    Activate nix-shell:

    $ nix-shell

    Don't preserve current bash environment:

    $ nix-shell --pure

     

    Demo: nix-shell and development of
    a Python package

    NixOS: declarative linux distro

    {
  boot.loader.grub.device = "/dev/sda";
  fileSystems."/".device = "/dev/sda1";

  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 80 ];
  };

  environment.systemPackages = with pkgs; [
    wget
    git
    gnupg
    tmux
  ];
  
  services = {
    sshd.enable = true;

    munin-node.enable = true;
    munin-cron = {
      enable = true;
      hosts = ''
        [${config.networking.hostName}]
        address localhost
      '';
    };
  };
}

    Deploying Pyramid with NixOS

    let 
  myproject = import ../default.nix {};
  productionini = pkgs.writeText "myproject-production.ini" ''
    ...
  '';
in {
  systemd.services.myproject = {
    after = [ "network.target" ];
    description = "My Project";
    wantedBy = [ "multi-user.target" ];
    path = [ myproject ];
    environment.PYTHONPATH = "${myproject}/lib/python2.7/site-packages";
    serviceConfig = {
      ExecStart = "${pkgs.pythonPackages.pyramid}/bin/pserve ${productionini}";
      User = "myuser";
      Group = "myuser";
    };
  };
}

    Getting started with NixOps

    $ nix-env -i nixops

    trivial.nix: 

    {
  webserver = { config, pkgs, ... }:
    { services.httpd.enable = true;
      services.httpd.adminAddr = "alice@example.org";
      services.httpd.documentRoot = "${pkgs.valgrind}/share/doc/valgrind/html";
    };
}

    trivial-virtualbox.nix: 

    {
  webserver = { config, pkgs, ... }:
    { deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 1024; # megabytes
    };
}

    Provision and deploy 

    $ nixops create -d trivial ./trivial.nix ./trivial-virtualbox.nix 33bced96-5f26-11e1-b9d7-9630d48abec1

$ nixops deploy -d trivial
creating VirtualBox VM ‘webserver’...

     

     Nixops Demo

    2nd NixOS Sprint @ Ljubljana, Slovenia




    Enjoy the purity! Questions?


    • #nixos on Freenode IRC
    • http://nixos.org

    (Shameless plugs)

    Made with Slides.com