Automating your development and production environments with Puppet, Vagrant and Packer

Who I am

A brief history of why/Case study

A brief what is
[puppet|vagrant|packer]

Puppet

Vagrant

Packer

Who I am

Head of caffeine consumption and

automating-all-the-crap-that-pisses-me-off-when-noones-looking

@fusions

Real title:
Systems Developer

@gabriel403

http://autonomicpilot.co.uk

A Brief History of Why

monolithic application
emails, admin, sales and
main site in one
just developers
just ubuntu
monolithic set-up guide

rewrite decided to split
multiple applications
use more ready made tech
use more diverse tech
less DIY code
queued and centralised log
queueing for email

started off small, just the main site
I came up with the centralised log,
convinced others and started work
initially just on my machine
email and queue, just on mine & 1 dev
even larger set up
what to do when it needed spreading
UX on windows
Sys Admins on Mac
Me on arch linux
Another dev on pure debian

Brief What Is

puppet

provisioning software

run a set of commands in an automated way

vagrant

VM config/control abstraction

packer

virtual image builder

What is Puppet

simply a way of executing commands

sudo apt-get install apache2
sudo nano /etc/apache2/sites-enable/somesite
sudo service apache2 restart

apache2 is ubuntu specific

httpd on debian/arch/bsd

apt-get is ubuntu/debian specific

pacman/yum/emerge on arch/redhat/gentoo

service is ubuntu specific

systemd/init.d/rc.d on arch/redhat/bsd

Puppet magic

package {'apache2':
  ensure => installed
}

service {'apache2':
  ensure => running
}

file {'/etc/apache2/sites-enable/somesite':
  content => 'somecontent',
  notify  => Service['apache2']
}

file {'/etc/apache2/sites-enable/someothersite':
  source => template('module/templatename'),
  notify  => Service['apache2']
}

(templates are good mkay)

There are still some problems

Facter

Ruby library for system info

architecture => amd64
hostname => gabriel-dev
lsbdistcodename => saucy
lsbdistid => Ubuntu
lsbdistrelease => 13.10
osfamily => Debian
is_virtual => true
virtual => virtualbox

More magic!

if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
  $package = 'httpd'
} elsif $::osfamily == 'Debian' {
  $package = 'apache2'
} elsif $::osfamily == 'Archlinux' {
  $package = 'apache'
} else {
  fail("Unsupported osfamily: ${::osfamily}")
}

if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
  $package = 'httpd'
} elsif $::osfamily == 'Debian' {
  $package = 'apache2'
} elsif $::osfamily == 'Archlinux' {
  $package = 'apache'
} else {
  fail("Unsupported osfamily: ${::osfamily}")
}
package {$package:
  ensure => installed
}service {$package:
  ensure => running,
  require => Package[$package]
}
file {'/etc/apache2/sites-enable/somesite':
  content => 'somecontent',
  notify  => Service[$package]
}
file {'/etc/apache2/sites-enable/someothersite':
  source => template('module/templatename'),
  notify  => Service[$package]
}

parameters class

class apache::params {
  if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
    $package      = 'httpd'
    $httpd_dir    = '/etc/httpd'
    $vhost_dir    = "${httpd_dir}/conf.d"
  } elsif $::osfamily == 'Debian' {
    $package      = 'apache2'
    $httpd_dir    = '/etc/apache2'
    $vhost_dir    = "${httpd_dir}/sites-available"
  } elsif $::osfamily == 'Archlinux' {
    $package      = 'apache'
    $httpd_dir    = '/etc/apache'
    $vhost_dir    = "${httpd_dir}/conf.d"
  } else {
    fail("Unsupported osfamily: ${::osfamily}")
  }
}

The Final Puppet magic

include apache::params
package {$apache::params::package:
  ensure => installed
}service {$apache::params::package:  ensure => running,
  require => Package[$apache::params::package]}
file {"${apache::params::vhost_dir}/somesite":
  content => 'somecontent',
  notify  => Service[$apache::params::package]}
file {"${apache::params::vhost_dir}/someothersite":  source => template('module/templatename'),
  notify  => Service[$apache::params::package]}

Puppet Forge

https://forge.puppetlabs.com

Case Study

Puppet allows us to automate
our installation and setup
No setup guide documentation, just code!
When updating our setup guide
we're really just writing code

What is Vagrant

Interface to personal Virtual Machines

Abstraction layer for VM configuration

Virtual Box for example

Imports a base box and runs provisioners

Easy plugin system

Huge range of existing plugins

Can be extended to work with cloud providers

AWS

Digital Ocean

Proxmox

Simple Config

Vagrant.configure("2") do |config|
  config.vm.box = "lamp base 1"
  config.vm.box_url = "https://some.host.com/lampbase.box"
  config.vm.network :private_network, ip: "5.5.5.5"
  config.vm.hostname = "lampy.dev"
  config.vm.synced_folder "~/Dropbox/code", "/code", type: "nfs"
  config.vm.provision :puppet, :module_path => "~/puppet/modules" do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file  = "init.pp"
  end
end

Multi Box Configuration

Vagrant.configure("2") do |config|
    # bring them up in these order
    # db cdn services web pound

  config.vm.define :db do |db|
    db.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    db.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    db.vm.network :private_network, ip: "10.11.12.43"
    db.vm.hostname = "stagingmysql.icheev.com"
    db.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    db.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "db.pp"
    end
  end

  config.vm.define :cdn do |cdn|
    cdn.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    cdn.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    cdn.vm.network :private_network, ip: "10.11.12.42"
    cdn.vm.hostname = "stagingcdn.icheev.com"
    cdn.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    cdn.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "cdn.pp"
    end
  end

  config.vm.define :services do |services|
    services.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    services.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    services.vm.network :private_network, ip: "10.11.12.44"
    services.vm.hostname = "stagingservices.icheev.com"
    services.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    services.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "services.pp"
    end
  end

  config.vm.define :web do |web|
    web.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    web.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    web.vm.network :private_network, ip: "10.11.12.41"
    web.vm.hostname = "staging.icheev.com"
    web.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    web.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "web.pp"
    end
  end

  config.vm.define :pound do |pound|
    pound.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    pound.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    pound.vm.network :private_network, ip: "10.11.12.40"
    pound.vm.hostname = "stagingpound.icheev.com"
    pound.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    pound.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "pound.pp"
    end
  end
end

Case study

despite being on different host OS
we now have a unified dev env
Works the same on windows/mac/linux
Any changes to provisioners is reflected
in everyone's dev env

What is Packer

AWS,Digital Ocean,Proxmox,
Rackspace all use base boxes
Some are as simple as base OS installs
Some are slightly complex, user set ups
Some are very complex, full LAMP + deployment
Can use the same provisioning as vagrant
Packer build base boxes
From ISOs
From other base boxes
Immutable servers

{
  "builders": [{
    "type"          : "digitalocean",
    "client_id"     : "clientid",
    "api_key"       : "apikey",
    "image_id"      : 1505699,
    "region_id"     : 5,
    "size_id"       : 64,
    "snapshot_name" : "test-snap",
    "droplet_name"  : "test.jeepers.com"
  }],
  "provisioners": [{
    "type": "shell",
    "execute_command": "echo 'vagrant' | sudo -S sh '{{ .Path }}'",
    "inline": [
      "apt-get update -y",
      "apt-get install -y linux-headers-$(uname -r) build-essential dkms puppet-common subversion ruby-hiera",
      "apt-get clean"
    ]
  },{
    "type"          : "puppet-masterless",
    "manifest_file" : "manifests/site.pp",
    "module_paths"  : ["../../puppet/modules"],
    "facter"        : { "fqdn" : "test.jeepers.com" }
  }]
}

{
  "builders": [{
    "type"                   : "virtualbox-iso",
    "iso_url"                : "http://releases.ubuntu.com/13.10/ubuntu-13.10-server-amd64.iso",
    "iso_checksum"           : "4d1a8b720cdd14b76ed9410c63a00d0e",
    "iso_checksum_type"      : "md5",
    "disk_size"              : 80000,
    "guest_os_type"          : "Ubuntu_64",
    "http_directory"         : "preseed",
    "ssh_username"           : "vagrant",
    "ssh_password"           : "vagrant",
    "output_directory"       : "output/13.10_5.5_64_virtualbox",
    "boot_command"           : [
      "<esc><esc><enter><wait>",
      "/install/vmlinuz noapic ",
      "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg ",
      "debian-installer=en_GB auto locale=en_GB kbd-chooser/method=uk ",
      "hostname={{ .Name }} ",
      "fb=false debconf/frontend=noninteractive ",
      "leopard-configuration/modelcode=SKIP leopard-configuration/layout=GB ",
      "leopard-configuration/variant=GB console-setup/ask_detect=false ",
      "initrd=/install/initrd.gz -- <enter>"
    ],
    "vboxmanage"             : [
      ["modifyvm", "{{.Name}}", "--memory", "1024"]
    ],
    "shutdown_command"       : "echo 'shutdown -P now' > shutdown.sh; echo 'vagrant'|sudo -S sh 'shutdown.sh'"
  }],
  "provisioners"             : [{
    "type": "shell",
    "execute_command": "echo 'vagrant' | sudo -S sh '{{ .Path }}'",
    "inline": [
      "apt-get update -y",
      "apt-get install -y linux-headers-$(uname -r) build-essential dkms puppet-common nfs-kernel-server nfs-common ruby-hiera",
      "apt-get clean",
      "mount -o loop VBoxGuestAdditions.iso /media/cdrom",
      "sh /media/cdrom/VBoxLinuxAdditions.run",
      "umount /media/cdrom",
      "mkdir ~/.ssh",
      "wget -qO- https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub >> ~/.ssh/authorized_keys",
      "echo 'vagrant ALL=NOPASSWD:ALL' > /tmp/vagrant",
      "chmod 0440 /tmp/vagrant",
      "mv /tmp/vagrant /etc/sudoers.d/"
    ]
  },{
    "type"          : "puppet-masterless",
    "manifest_file" : "manifests/site.pp",
    "module_paths"  : ["../../puppet/modules","../../../puppet"],
    "facter"        : {
      "fqdn"        : "serverbase.dev"
    }
  }],
  "post-processors"          : [
    {
      "type"               : "vagrant",
      "only"               : ["virtualbox-iso"],
      "output"             : "output/ubuntu_13.10_5.5_base_v2.box"
    }
  ]
}

Case Study

Using the same provisioning on
production and dev leads to a nicer life
Updating OS images (dev/prod) is simpler
by just editing a config file

In Conclusion

Puppet is used for installing and configuring packages/services/files

Vagrant abstracts configuration and control for multiple VM types into a simple config file and command line.
It can also use puppet and other provisioning s/w.

Packer builds and provisions base boxes and servers, using other base boxes or isos as a starting point and can use puppet or other provisiong s/w.