Automating your development and production environments with Puppet, Vagrant and Packer



Who I am
A brief history of why/Case study
A brief what is
[puppet|vagrant|packer]
Puppet
Vagrant
Packer

Who I am

Head of caffeine consumption and 
automating-all-the-crap-that-pisses-me-off-when-noones-looking

@fusions

Real title:
Systems Developer

@gabriel403
http://autonomicpilot.co.uk

A Brief History of Why




  • monolithic application
  • emails, admin, sales and
    main site in one
  • just developers
  • just ubuntu
  • monolithic set-up guide




  • rewrite decided to split
  • multiple applications
  • use more ready made tech
  • use more diverse tech
  • less DIY code
  • queued and centralised log
  • queueing for email



  • started off small, just the main site
  • I came up with the centralised log,
    convinced others and started work
  • initially just on my machine
  • email and queue, just on mine & 1 dev
  • even larger set up
  • what to do when it needed spreading
  • UX on windows
  • Sys Admins on Mac
  • Me on arch linux
  • Another dev on pure debian

Brief What Is



puppet
provisioning software
run a set of commands in an automated way

vagrant
VM config/control abstraction

packer
virtual image builder

What is Puppet

simply a way of executing commands
sudo apt-get install apache2
sudo nano /etc/apache2/sites-enable/somesite
sudo service apache2 restart

apache2 is ubuntu specific
httpd on debian/arch/bsd

apt-get is ubuntu/debian specific
pacman/yum/emerge on arch/redhat/gentoo

service is ubuntu specific
systemd/init.d/rc.d on arch/redhat/bsd

Puppet magic


package {'apache2':
  ensure => installed
}
service {'apache2':
  ensure => running
}
file {'/etc/apache2/sites-enable/somesite':
  content => 'somecontent',
  notify  => Service['apache2']
}
file {'/etc/apache2/sites-enable/someothersite':
  source => template('module/templatename'),
  notify  => Service['apache2']
}
(templates are good mkay)
There are still some problems

Facter



Ruby library for system info

architecture => amd64
hostname => gabriel-dev
lsbdistcodename => saucy
lsbdistid => Ubuntu
lsbdistrelease => 13.10
osfamily => Debian
is_virtual => true
virtual => virtualbox

More magic!



if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
  $package = 'httpd'
} elsif $::osfamily == 'Debian' {
  $package = 'apache2'
} elsif $::osfamily == 'Archlinux' {
  $package = 'apache'
} else {
  fail("Unsupported osfamily: ${::osfamily}")
}

if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
  $package = 'httpd'
} elsif $::osfamily == 'Debian' {
  $package = 'apache2'
} elsif $::osfamily == 'Archlinux' {
  $package = 'apache'
} else {
  fail("Unsupported osfamily: ${::osfamily}")
}
package {$package:
  ensure => installed
}service {$package:
  ensure => running,
  require => Package[$package]
}
file {'/etc/apache2/sites-enable/somesite':
  content => 'somecontent',
  notify  => Service[$package]
}
file {'/etc/apache2/sites-enable/someothersite':
  source => template('module/templatename'),
  notify  => Service[$package]
}

parameters class 

class apache::params {
  if $::osfamily == 'RedHat' or $::operatingsystem == 'amazon' {
    $package      = 'httpd'
    $httpd_dir    = '/etc/httpd'
    $vhost_dir    = "${httpd_dir}/conf.d"
  } elsif $::osfamily == 'Debian' {
    $package      = 'apache2'
    $httpd_dir    = '/etc/apache2'
    $vhost_dir    = "${httpd_dir}/sites-available"
  } elsif $::osfamily == 'Archlinux' {
    $package      = 'apache'
    $httpd_dir    = '/etc/apache'
    $vhost_dir    = "${httpd_dir}/conf.d"
  } else {
    fail("Unsupported osfamily: ${::osfamily}")
  }
}

The Final Puppet magic

include apache::params
package {$apache::params::package:
  ensure => installed
}service {$apache::params::package:  ensure => running,
  require => Package[$apache::params::package]}
file {"${apache::params::vhost_dir}/somesite":
  content => 'somecontent',
  notify  => Service[$apache::params::package]}
file {"${apache::params::vhost_dir}/someothersite":  source => template('module/templatename'),
  notify  => Service[$apache::params::package]}

Puppet Forge


https://forge.puppetlabs.com

Case Study


  • Puppet allows us to automate
    our installation and setup
  • No setup guide documentation, just code!
  • When updating our setup guide
    we're really just writing code

What is Vagrant


Interface to personal Virtual Machines
Abstraction layer for VM configuration
Virtual Box for example
Imports a base box and runs provisioners
Easy plugin system
Huge range of existing plugins

Can be extended to work with cloud providers
AWS
Digital Ocean
Proxmox

Simple Config



Vagrant.configure("2") do |config|
  config.vm.box = "lamp base 1"
  config.vm.box_url = "https://some.host.com/lampbase.box"
  config.vm.network :private_network, ip: "5.5.5.5"
  config.vm.hostname = "lampy.dev"
  config.vm.synced_folder "~/Dropbox/code", "/code", type: "nfs"
  config.vm.provision :puppet, :module_path => "~/puppet/modules" do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file  = "init.pp"
  end
end

Multi Box Configuration


Vagrant.configure("2") do |config|
    # bring them up in these order
    # db cdn services web pound

  config.vm.define :db do |db|
    db.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    db.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    db.vm.network :private_network, ip: "10.11.12.43"
    db.vm.hostname = "stagingmysql.icheev.com"
    db.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    db.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "db.pp"
    end
  end

  config.vm.define :cdn do |cdn|
    cdn.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    cdn.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    cdn.vm.network :private_network, ip: "10.11.12.42"
    cdn.vm.hostname = "stagingcdn.icheev.com"
    cdn.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    cdn.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "cdn.pp"
    end
  end

  config.vm.define :services do |services|
    services.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    services.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    services.vm.network :private_network, ip: "10.11.12.44"
    services.vm.hostname = "stagingservices.icheev.com"
    services.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    services.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "services.pp"
    end
  end

  config.vm.define :web do |web|
    web.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    web.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    web.vm.network :private_network, ip: "10.11.12.41"
    web.vm.hostname = "staging.icheev.com"
    web.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    web.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "web.pp"
    end
  end

  config.vm.define :pound do |pound|
    pound.vm.box = "Ubuntu 12.10 Quantal x86_64 (Guest Additions 4.2.2)"
    pound.vm.box_url = "https://github.com/downloads/roderik/VagrantQuantal64Box/quantal64.box"
    pound.vm.network :private_network, ip: "10.11.12.40"
    pound.vm.hostname = "stagingpound.icheev.com"
    pound.vm.synced_folder "~/Dropbox/GoMADTech/code", "/code", :nfs => true
    pound.vm.provision :puppet, :module_path => "~/Dropbox/GoMADTech/boxen/puppet/modules" do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "pound.pp"
    end
  end
end

Case study



  • despite being on different host OS
    we now have a unified dev env
  • Works the same on windows/mac/linux
  • Any changes to provisioners is reflected
    in everyone's dev env

What is Packer


  • AWS,Digital Ocean,Proxmox,
    Rackspace all use base boxes
  • Some are as simple as base OS installs
  • Some are slightly complex, user set ups
  • Some are very complex, full LAMP + deployment
  • Can use the same provisioning as vagrant
  • Packer build base boxes
  • From ISOs
  • From other base boxes
  • Immutable servers
{
  "builders": [{
    "type"          : "digitalocean",
    "client_id"     : "clientid",
    "api_key"       : "apikey",
    "image_id"      : 1505699,
    "region_id"     : 5,
    "size_id"       : 64,
    "snapshot_name" : "test-snap",
    "droplet_name"  : "test.jeepers.com"
  }],
  "provisioners": [{
    "type": "shell",
    "execute_command": "echo 'vagrant' | sudo -S sh '{{ .Path }}'",
    "inline": [
      "apt-get update -y",
      "apt-get install -y linux-headers-$(uname -r) build-essential dkms puppet-common subversion ruby-hiera",
      "apt-get clean"
    ]
  },{
    "type"          : "puppet-masterless",
    "manifest_file" : "manifests/site.pp",
    "module_paths"  : ["../../puppet/modules"],
    "facter"        : { "fqdn" : "test.jeepers.com" }
  }]
}

{
  "builders": [{
    "type"                   : "virtualbox-iso",
    "iso_url"                : "http://releases.ubuntu.com/13.10/ubuntu-13.10-server-amd64.iso",
    "iso_checksum"           : "4d1a8b720cdd14b76ed9410c63a00d0e",
    "iso_checksum_type"      : "md5",
    "disk_size"              : 80000,
    "guest_os_type"          : "Ubuntu_64",
    "http_directory"         : "preseed",
    "ssh_username"           : "vagrant",
    "ssh_password"           : "vagrant",
    "output_directory"       : "output/13.10_5.5_64_virtualbox",
    "boot_command"           : [
      "<esc><esc><enter><wait>",
      "/install/vmlinuz noapic ",
      "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg ",
      "debian-installer=en_GB auto locale=en_GB kbd-chooser/method=uk ",
      "hostname={{ .Name }} ",
      "fb=false debconf/frontend=noninteractive ",
      "leopard-configuration/modelcode=SKIP leopard-configuration/layout=GB ",
      "leopard-configuration/variant=GB console-setup/ask_detect=false ",
      "initrd=/install/initrd.gz -- <enter>"
    ],
    "vboxmanage"             : [
      ["modifyvm", "{{.Name}}", "--memory", "1024"]
    ],
    "shutdown_command"       : "echo 'shutdown -P now' > shutdown.sh; echo 'vagrant'|sudo -S sh 'shutdown.sh'"
  }],
  "provisioners"             : [{
    "type": "shell",
    "execute_command": "echo 'vagrant' | sudo -S sh '{{ .Path }}'",
    "inline": [
      "apt-get update -y",
      "apt-get install -y linux-headers-$(uname -r) build-essential dkms puppet-common nfs-kernel-server nfs-common ruby-hiera",
      "apt-get clean",
      "mount -o loop VBoxGuestAdditions.iso /media/cdrom",
      "sh /media/cdrom/VBoxLinuxAdditions.run",
      "umount /media/cdrom",
      "mkdir ~/.ssh",
      "wget -qO- https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub >> ~/.ssh/authorized_keys",
      "echo 'vagrant ALL=NOPASSWD:ALL' > /tmp/vagrant",
      "chmod 0440 /tmp/vagrant",
      "mv /tmp/vagrant /etc/sudoers.d/"
    ]
  },{
    "type"          : "puppet-masterless",
    "manifest_file" : "manifests/site.pp",
    "module_paths"  : ["../../puppet/modules","../../../puppet"],
    "facter"        : {
      "fqdn"        : "serverbase.dev"
    }
  }],
  "post-processors"          : [
    {
      "type"               : "vagrant",
      "only"               : ["virtualbox-iso"],
      "output"             : "output/ubuntu_13.10_5.5_base_v2.box"
    }
  ]
}

Case Study



  • Using the same provisioning on
    production and dev leads to a nicer life
  • Updating OS images (dev/prod) is simpler 
    by just editing a config file

In Conclusion


Puppet is used for installing and configuring packages/services/files

Vagrant abstracts configuration and control for multiple VM types into a simple config file and command line.
It can also use puppet and other provisioning s/w.

Packer builds and provisions base boxes and servers, using other base boxes or isos as a starting point and can use puppet or other provisiong s/w.

Automating your development and production environments with Packer, puppet and vagrant

By Gabriel Baker

Made with Slides.com

Automating your development and production environments with Packer, puppet and vagrant

  • 4,832

More from Gabriel Baker