Puppet

training



Simon Josi 
YOKTO Engineering

About me

(in respect to configuration management)


clusterssh in 2005
cfengine in 2006 until 2007
puppet since 2007
never used chef, bcfg2, cfengine3,ansible,salt or your mom's favourite cfg mgmt tool... (I've heard and read about them though...)

clusterssh


i.e. execute the same command on multiple hosts and hope that all host behave the same way

easy to begin with
very manual
scalability issues ;)

MY CLUSTERSSH EXPERIENCE


managed up to ~10 at one time @ Puzzle ITC

CFENGINE (2.x)


i.e. lots of boilerplate manifest code of which knowbody knows what it does => "WTF is THIS for?!?"

idempotent
declarative ?!?
lots non-declarative functions
e.g. AppendIfNoSuchLine, really?
no progress in development
12 years old in 2007
code looks bulky and old

MY CFENGINE EXPIERENCE


managed ~60 hosts @ Puzzle ITC

puppet


i.e. Model Driven Datacenter

declarative
abstract
idempotent
a model of "resources" and "relationships"
compiled

MY PUPPET EXPERIENCE


managed ~120 nodes with ~170 Modules @ Puzzle ITC

bootstrapped puppet setup @ Die Mobiliar
15 (!) environments, now ~3000 nodes

managed webstack & deployment @ Atizo
~10 nodes, ~110 Modules

PUPPET @ SWISSTXT


platform-services for private cloud offering
~144 nodes, ~50 modules

boostrapped puppet environment @ SWISS TXT
 currently ~20 nodes, ~30 modules

the big picture


Puppet COMPONENTS

internal (puppet executable)

Language parser and compiler (for internal DSL)
Resource Abstraction Layer (RAL, Types and Providers)
Agent/Master (Client/Server)
PKI (PuppetCA)

external (other executables)

PuppetDB (formerly storedconfigs)
Facter for normalized node information
Hiera for external parameterization
Marionette Collective for centralized orchestration

OVERVIEW


Compiler output is a catalog

catalog is a DAG


http://somethingsinistral.net/blog/puppet-the-catalog-and-you/
http://www.masterzen.fr/2011/12/27/puppet-internals-the-parser/
http://www.masterzen.fr/2012/03/17/puppet-internals-the-compiler/

types and providers


TYPES AND PROVIDERS

TYPES AND PROVIDERS


MASTER/AGENT


Master is a RESTful webservice
Agents consume this webservice
Secured with SSL client authentication

PuppetCA

Every host got a SSL certificate
Certificates are signed with the PuppetCA (Puppet's own Certificate Authority)

FACTER

or how do your nodes look like?
[root@ellen ~]# facter
...
architecture => x86_64
domain => yokto.net
fqdn => ellen.yokto.net
hardwaremodel => x86_64
hostname => ellen
interfaces => eth0,lo
ipaddress => 195.141.111.126
ipaddress_eth0 => 195.141.111.126
ipaddress_lo => 127.0.0.1
is_virtual => true
kernel => Linux
kernelmajversion => 2.6
kernelversion => 2.6.32
lsbdistid => CentOS
lsbdistrelease => 6.4
lsbmajdistrelease => 6
...
available as variables anywhere in your puppet code

puppetdb


used for exported resources
declare a  resource on one host, apply on another
central inventory of all facts from all nodes

Hiera

hierarchical parameter storage

MCollective

framework to build server orchestration or parallel job execution systems

architecture


anatomy of a puppet run

detailed agent/master communication


http://docs.puppetlabs.com/puppet/3/reference/subsystem_agent_master_comm.html

PUPPET'S DSL


restricted by design
specific to the domain
managing resources, declaring relationships
declarative (not imperative/no iterators)
uses functional programming paradigms to work with collections of items (e.g. map)
https://puppetlabs.com/blog/why-puppet-has-its-own-configuration-language/

Give me code please!

class openssh::server {
  package{'openssh-server':
    ensure => present,
  }
  file{'/etc/ssh/sshd_config':
    ensure  => present,
    source  => 'puppet:///modules/openssh/sshd_config',
    group   => 'root',
    mode    => '0444',
    require => Package['openssh-server'],
    notify  => Service['sshd'],
  }
  service{'sshd':
    ensure  => running,
    enable  => true,
    require => [
      Package['openssh-server'],
      File ['/etc/ssh/sshd_config'],
    ],
  } 
}

The same without redundanT RELATIONSSHIPS OR DEFAULT VALUES

and the use of relationship operators
class openssh::server {
  package{'openssh-server':
    ensure => present,
  } ->
  file{'/etc/ssh/sshd_config':
    source  => 'puppet:///modules/openssh/sshd_config',
  } ~>
  service{'sshd':
    ensure  => running,
    enable  => true,
  } 
}

language constructs


MANIFESTS
IMPORT
NODES
MODULES
CLASSES
NAMESPACING AND AUTOLOADING
DEFINED TYPES
NATIVE TYPES
FUNCTIONS

manifests


every file ending in .pp and containing puppet DSL code is called a manifests

IMPORT


included a file in place and evaluate it
just like php's include

never use it it in modules
only use it to bootstrap code

nodes 


defined in site.pp (common practice)
puppet's entry point
only include/declare classes (best practice)

modules


a convention to structure/group


manifests
FILES
TEMPLATES
FUNCTIONS
FACTS
TYPES and providers

http://docs.puppetlabs.com/module_cheat_sheet.pdf

classes


singletons
parameterizable
can be declared multiple times (without parameters)
but will get applied only once
can de declared only once (with parameters)

namespacing and autoloading


modulepaths

Default: $confdir/modules:/usr/share/puppet/modules
MPC:
$confdir/$environment/forge
$confdir/$environment/site
$confdir/$environment/role
$confdir/$environment/platform-services

defined types


not singletons ;)
parameterizable
can de declared multiple times (with different identifiers)
look and behave like native resources

native types


file  package  service 
user  group  host
mount  exec  augeas
ssh_authorized_key  yumrepo

functions


are executed on the server during catalog compilation
are written in ruby
e.g. template compilation

NAMESPACING AND AUTOLOADING


$modulepath/modulename/init.pp =>
class modulename {

$modulepath/modulename/server.pp =>
class modulename::server {

$modulepath/modulename/server/debian.pp =>
class modulename::server::debian {

SCOPES



more LANGUAGE CONSTRUCTS


INHERITANCE
RESOURCE PARAMETERS AND METAPARAMETERS
VARIABLES AND SCOPES
RESOURCE ORDERING
NOTIFY/SUBSCRIBE
CONTROL FLOW STRUCTURES
EXPORTED RESOURCES
RESOURCE COLLECTORS

INHERITANCE



DO NOT USE IT

only in edge cases really useful or the way to go

parameters and metaparameters


each resource has its own set of supported parameters
metaparameters are the ones valid for all resources

variables and scopes



resource ordering


before/after



relationship/CHaining operators


notify/subscribe

control flow structures


exported resources

resource collectors

definitions and declarations



define vs declare

DEFINTIONS

the origin of things

  class modulename::classname {
    package{'package_name':} ->
    file{'/etc/filename':} ~>
    service{'service_name':}
  }
  define modulename::definename(
    $parameter_one = 'default_value',
  ) {
    file{"/etc/$parameter_one"}
  }

declarations

the use of things

include modulename::classname
class{'modulename::classname':}
class{'modulename::classname':
  param_one => 'value',
}
modulename::definename{'instance_one':
  param_one => 'value',
}
modulename::definename{'instance_two':
  param_one => 'value',
}

TYPES OF PUPPET MODULES


PUBLIC/FORGE

SITE

ROLE
...


many more depending of how you want to model your infrastructure
can be seperated and integrated via multiple puppet modulepaths and the use of git submodules

COMPOSITION


http://sysadvent.blogspot.ch/2012/12/day-13-configuration-management-as-legos.html

PUBLIC/FORGE MODULES


found in the intertubes
http://forge.puppetlabs.com
http://puppet-modules.git.puzzle.ch
http://git.puppet.immerda.ch
https://labs.riseup.net/code/projects/puppetmodules
https://github.com/camptocamp

site modules


site-specific use of a public module

reusable site-wide defaults/helpers for a public module

ROLE MODULES


composition of multiple forge and site modules into a "role"

COMPOSITION ORDER OF PUPPET MODULE TYPES


FORGE -> SITE -> ROLE -> NODE


sometimes you don't need an intermediate module, e.g.

FORGE -> NODE
ROLE -> NODE
FORGE -> ROLE -> NODE

you get the idea...

REAL WORLD EXAMPLE

manage openstack with puppet


NODE CLASSIFICATION PATTERNs

or wtf is an ENC




node defintions


node 'www1.example.com' {
  include common
  include apache
  include squid
}
node /w+.example.com/ {
  include common
  include apache
  include squid
}

externaL node classifier


[master]
  node_terminus = exec

  external_nodes = /usr/local/bin/puppet_node_classifier

ENCs must return either a YAML hash or nothing. This hash may contain classes, parameters, and environment keys, and must contain at least either classes or parameters.

nodeless configuration


fact driven

node default {
  include $role
}

event propagation


class sshd:service {
  service{'sshd':}
}

File['somefile']{
  notify => Class['sshd::service'],
}


EVENT PROPAGATION


file{'/somefile':
  source => 'puppet:///modules/module/file',
} ~>
mount{'/target':
  device => '/dev/sdx1',
} ~>
service{'someservice':
}
Made with Slides.com