Recap

• Basics check - puppet 101
• What are they?
• Why bother to develop them?
• How to do it?
• How to test and maintain them?

REUSABLE Puppet MODULES

Puppet 101

Quick repetition

Teory of operation

Puppet 101

Module Example

mymodule          # This outermost directory’s name matches the
│                 # name of the module.
├── manifests     # Contains all of the manifests in the module.
│   └── init.pp   # Contains a class definition. This class’s 
│                 # name must match the module’s name.
├── metadata.json # Contains META information about module
├── spec          # Contains spec tests for any plugins in the
│                 # lib directory.
├── templates     # Contains templates, which the module’s 
│                 # manifests can use.
├── tests         # Contains examples showing how to declare 
│   │             # the module's classes and defined types.
│   └── init.pp
└── lib           # Contains plugins, like custom facts and 
    │             # custom resource types.
    └── puppet
        ├── provider
        └── type

Puppet 101

Modules

• Modules are stored on PM
• Modules contain puppet code
  • puppet
  • ruby
• Modules have layout
• Modules can be easily installed

More on modules

What are does reusable Puppet modules?

Unity
Ease
Mastery

Unity

One single and well defined responsibility for given puppet module

Should support installation and management of given software

EasE

It should be possible to use module as simply as it possible. Best will be with one single line

include xtreemfs::role::directory

Module should work with it's default settings as it brings easy start for new comers

Mastery

Interfaces should have as deep input as possible.

• Most used setting as parameters
• Less often as property maps

define xtreemfs::volume (
  $ensure       = 'present',
  $volume       = $name,
  $dir_host     = undef,
  $dir_port     = undef,
  $dir_protocol = undef,
  $options      = {},
) {
  # [..]
}

Why to develop?

as a reusable

modules

Plan
Your
Interface

Public interface

.
├── internal
│   ├── configure
│   │   ├── directory.pp
│   │   ├── metadata.pp
│   │   └── storage.pp
│   ├── packages
│   │   ├── client.pp
│   │   └── server.pp
│   ├── repo.pp
│   ├── settings.pp
│   └── workflow.pp
├── mount.pp
├── policy.pp
├── replicate.pp
├── role
│   ├── directory.pp
│   ├── metadata.pp
│   └── storage.pp
├── settings.pp
└── volume.pp

4 directories, 16 files

• Simplify public interface
• Put internals in internal package
• Focus on usability for users
• Use private function from stdlib > 4.4

Make
Covering
Defines

Covering Defines

define xtreemfs::volume (
  $volume       = $name,
  $ensure       = 'present',
  $dir_host     = undef,
  $dir_port     = undef,
  $dir_protocol = undef,
  $options      = {},
) {
  include xtreemfs::internal::packages::client
  include xtreemfs::internal::workflow
  include xtreemfs::settings

  validate_hash($options)

  $host = directory_address($dir_host, $dir_port,
    $dir_protocol, $xtreemfs::settings::dir_service)

  xtreemfs_volume { $volume:
    ensure  => $ensure,
    host    => $host,
    options => $options,
    require => Anchor[$xtreemfs::internal::workflow::service],
  }
}

Design Module

workflow

Module Workflow

# INTERNAL PRIVATE CLASS: do not use directly!
class xtreemfs::internal::workflow {
  
  $repo      = 'xtreemfs::repo'
  $packages  = 'xtreemfs::packages'
  $configure = 'xtreemfs::configure'
  $service   = 'xtreemfs::service'
  $end       = 'xtreemfs::end'
  
  anchor { $repo: } ->
  anchor { $packages: } ->
  anchor { $configure: } ~>
  anchor { $service: } ->
  anchor { $end: }
  
}

Puppet module Tool

Generate a module

puppet module generate company-service

Puppet Module tool - Example

$ puppet module generate wavesoftware-acpid
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 

# [...] MORE QUESTIONS

----------------------------------------
{
  "name": "wavesoftware-acpid",
  "version": "0.1.0",
  "author": "wavesoftware",
  "summary": "A module that manages and install a Acpid service",
  "license": "Apache-2.0",
  "source": "github.com/wavesoftware/puppet-acpid",
  "project_page": "https://github.com/wavesoftware/puppet-acpid",
  "issues_url": "https://github.com/wavesoftware/puppet-acpid/issues",
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
--> 
Notice: Generating module at /home/ksuszynski/git/wavesoftware/puppet-acpid...

Puppet Module tool - 2

wavesoftware-acpid/
├── CHANGELOG
├── CONTRIBUTING.md
├── CONTRIBUTORS
├── files
├── Gemfile
├── Guardfile
├── lib
│   └── puppet
│       ├── provider
│       └── type
├── LICENSE
├── manifests
│   ├── config.pp
│   ├── init.pp
│   ├── install.pp
│   ├── params.pp
│   └── service.pp
├── metadata.json
├── Modulefile
├── Rakefile
├── README.markdown
├── spec
│   ├── acceptance
│   │   ├── class_spec.rb
│   │   └── nodesets
│   │       ├── centos-64-x64.yml
│   │       ├── default.yml
│   │       └── ubuntu-server-12042-x64.yml
│   ├── classes
│   │   ├── coverage_spec.rb
│   │   └── example_spec.rb
│   ├── spec_helper_acceptance.rb
│   └── spec_helper.rb
├── templates
└── tests
    └── init.pp

Delivers a working structure of new module with:

• manifests
• lib directories
• Rakefile & Gemfile
• smoke tests
• spec test
• beaker tests

RVM & Bundler

Gems & Ruby libraries Install

RVM & Bundler

 $ rvm use 2.1
Using /home/ksuszynski/.rvm/gems/ruby-2.1.4
 $ export PUPPET_VERSION="~> 3.7.0"
 $ bundle install --path .vendor
Fetching https://github.com/rodjek/rspec-puppet.git
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Installing rake 10.4.2

# [..] more lines

Installing facter 1.7.6
Installing hiera 1.3.4
Installing beaker 2.4.1
Installing rspec-core 2.99.2
Installing rspec-expectations 2.99.2
Installing rspec-mocks 2.99.3
Installing rspec 2.99.0
Installing specinfra 1.27.5
Installing serverspec 1.16.0
Installing beaker-rspec 4.0.0
Installing puppet 3.7.4
Using rspec-puppet 2.0.0 from https://github.com/rodjek/rspec-puppet.git (at master)
Installing puppetlabs_spec_helper 0.8.2
Using bundler 1.7.9
Your bundle is complete!
It was installed into ./.vendor

Gems & Ruby libraries Install

RVM & Bundler - 2

bundle exec command

bundle exec

Runs a command in bundle context

RVM & Bundler - 2

$ bundle exec rake -T
rake acceptance                              # Run acceptance tests
rake beaker                                  # Run beaker acceptance tests
rake beaker_nodes                            # List available beaker nodesets
rake build                                   # Build puppet module package
rake clean                                   # Clean a built module package
rake coverage                                # Generate code coverage information
rake help                                    # Display the list of available rake tasks
rake lint                                    # Check puppet manifests with puppet-lint / Run puppet-lint
rake module:bump                             # Bump module version to the next minor
rake module:bump_commit                      # Bump version and git commit
rake module:clean                            # Runs clean again
rake module:dependency[module_name,version]  # Set specific module dependency version
rake module:push                             # Push module to the Puppet Forge
rake module:release                          # Release the Puppet module, doing a clean, build, tag, push, bump_commit
rake module:tag                              # Git tag with the current module version
rake spec                                    # Run spec tests in a clean fixtures directory
rake spec_clean                              # Clean up the fixtures directory
rake spec_prep                               # Create the fixtures directory
rake spec_standalone                         # Run spec tests on an existing fixtures directory
rake syntax                                  # Syntax check Puppet manifests and templates
rake syntax:hiera                            # Syntax check Hiera config files
rake syntax:manifests                        # Syntax check Puppet manifests
rake syntax:templates                        # Syntax check Puppet templates
rake test                                    # Run syntax, lint, and spec tests
rake validate                                # Check syntax of Ruby files and call :syntax

bundle exec - runs in context

Puppet Ide's

Geppetto

Puppet Ide

others

• Sublime Text

• InteliJ IDEA

• VIM, Emacs

Install puppet plugins to:

Use GIT

if possible

Puppet

code

• puppetlabs standards: /guides/style_guide.html
• keep manifest's code on 1, max 2 screens
• document your manifests: puppet rdoc
• use puppet lint: rake lint
• use metadata lint: rake metadata

Ruby

code

• style guide: github.com/styleguide/ruby
• document your code: rdoc
• test your code: rspec
• use rubocop to check code style
• use puppet_x convention for shared code

Puppet_X

convention - example

require File.join(File.dirname(__FILE__),
  '../../puppet_x/wavesoftware/xtreemfs/type/replicable')

Puppet::Type.newtype :xtreemfs_replicate do
  desc "The xtreemfs_replicate type"
  newparam :file do
    isnamevar
  end
  newproperty :factor do
    Puppet_X::Wavesoftware::Xtreemfs::Type::Replicable
      .configure_factor(self)
  end
  newproperty :policy do
    Puppet_X::Wavesoftware::Xtreemfs::Type::Replicable
      .configure_policy(self)
  end
  Puppet_X::Wavesoftware::Xtreemfs::Type::Replicable
    .configure_global_validation(self)
end

Puppet_X

convention

lib/
├── puppet
│   ├── provider
│   │   ├── xtreemfs_policy
│   │   │   └── xtfsutil.rb
│   │   └── xtreemfs_replicate
│   │       └── xtfsutil.rb
│   └── type
│       ├── xtreemfs_policy.rb
│       └── xtreemfs_replicate.rb
└── puppet_x
    └── wavesoftware
        └── xtreemfs
            ├── provider
            │   └── xtfsutil.rb
            └── type
                └── replicable.rb

• Safely separate your code from other's code
• Can be easily indexed by tools operating on standard ruby code

Puppet Testing

Smoke tests

• Easy
• Just use your code
• Inside test folder

# contents of tests/role/directory.pp
include xtreemfs::role::directory

Puppet Testing

Smoke tests Output

Notice: Compiled catalog for ksuszynski-gs70.suszynski.org 
in environment production in 1.24 seconds
Notice: /Stage[main]/Xtreemfs::Internal::Repo/Apt::Source[xtreemfs]
/Apt::Key[Add key: 07D6EA4F2FA7E736 from Apt::Source xtreemfs]
/Apt_key[Add key: 07D6EA4F2FA7E736 from Apt::Source xtreemfs]/ensure:
 current_value absent, should be present (noop)
Notice: Apt::Key[Add key: 07D6EA4F2FA7E736 from 
Apt::Source xtreemfs]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Apt/File[/etc/apt/apt.conf.d/15update-stamp]
/content: current_value {md5}b9de0ac9e2c9854b1bb213e362dc4e41,
 should be {md5}4355b3e7824866a503fc221621fc65ba (noop)
Notice: /Stage[main]/Xtreemfs::Internal::Repo/Apt::Source[xtreemfs]
/File[xtreemfs.list]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Would have 
triggered 'refresh' from 1 events
Notice: Class[Apt::Update]: Would have triggered 'refresh' from
1 events
Notice: Apt::Source[xtreemfs]: Would have triggered 'refresh' 
from 2 events
Notice: Class[Apt]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Xtreemfs::Internal::Packages::Server/
Package[xtreemfs-server]/ensure: current_value purged, should be present (noop)
Notice: Finished catalog run in 0.35 seconds

Puppet Testing

RSpec Unit example

# contents of spec/unit/defines/mount_spec.rb
require 'spec_helper'
describe 'xtreemfs::mount', :type => :define do
  let :facts do
    {
      :osfamily               => 'RedHat',
      :operatingsystem        => 'CentOS',
      :operatingsystemrelease => '6.6',
      :fqdn                   => 'slave1.vm'
    }
  end
  let :title do
    '/mnt/my-xtreemfs-mount'
  end
  describe 'shouldn\'t work with only default parameters, must pass a volume' do
    it { expect { should compile }.to raise_error(/Must pass volume to Xtreemfs::Mount/) }
  end
  describe 'should work with only volume passed' do
    let :params do
      { :volume      => 'myVolume' }
    end
    it { should compile }
    it { should contain_class('xtreemfs::internal::packages::client') }
    it do
      should contain_xtreemfs__mount('/mnt/my-xtreemfs-mount').with(
        'ensure'      => 'mounted',
        'volume'      => 'myVolume',
        'dir_host'    => nil,
        'atboot'      => false,
        'options'     => 'defaults,allow_other'
      )
    end
  end
end

Puppet Testing

RSpec Unit output

xtreemfs::mount
  shouldn't work with only default parameters, must pass a volume
    should compile into a catalogue without dependency cycles
  should work with only volume passed
    should compile into a catalogue without dependency cycles
    should contain Class[xtreemfs::internal::packages::client]
    should contain Xtreemfs::Mount[/mnt/my-xtreemfs-mount] with ensure => "mounted", volume => "myVolume", dir_host defined, atboot => false and options => "defaults,allow_other"
    should contain File[/mnt/my-xtreemfs-mount] with ensure => "directory"
    should contain Mount[/mnt/my-xtreemfs-mount] that requires Anchor[xtreemfs::service]

Finished in 1.85 seconds
6 examples, 0 failures
Coverage report generated for RSpec to xtreemfs/coverage. 0.0 / 0.0 LOC (100.0%) covered.

Beaker

rspec acceptance

testing

Beaker

HOSTS:
  ubuntu-server-1404-x64-docker:
    roles:
      - master
    platform: ubuntu-14.04-amd64
    image: ubuntu:14.04
    hypervisor: docker
    docker_cmd: '/usr/sbin/sshd -D -o "PermitRootLogin yes"
       -o "PasswordAuthentication yes"'
CONFIG:
  type: foss

Nodeset configuration

Beaker

require 'spec_helper_acceptance'
describe 'xtreemfs::role::directory class' do

  describe 'executing without installing packages' do
    pp = <<-eos
      class { 'xtreemfs::role::directory':
        install_packages => false,
      }
    eos
    it 'shouldn\'t work' do
      apply_manifest(pp, :expect_failures => true)
    end
  end

  describe 'executing simple include with puppet code' do
    pp = <<-eos
    include xtreemfs::role::directory
    eos    
    it 'should work without errors' do
      apply_manifest(pp, :catch_failures => true)
    end
    it 'should not make any changes when executed twice' do
      apply_manifest(pp, :expect_changes => false)
    end
    describe service('xtreemfs-dir') do 
      it { should be_running }
    end    
  end
end

RSpec + Serverspec

Good to do

• Calculate manifest coverage               
  
   
• Calculate rspec coverage
  
   
• Keep them above 95%

Puppet & Ruby Testing

at_exit { RSpec::Puppet::Coverage.report! }

gem 'simplecov'

All at Once 

bundle exec rake test acceptance

Online services

Better visibility of code

Open Source

Closed Source

• GitHub
• Travis
• Coveralls
• Code Climate
• Inch
• Gemnasium
• Puppet Forge
   

many more!!

• Stash
• Jenkins
• Sonar
• "puppet forge mirrors"

not so much more

Stash

Jenkins

Github

Sheilds

Gemnasium

Travis CI

Coveralls

Code Climate

Inch

Puppetlabs

documentation

docs.puppetlabs.com

Type & Providers

Book

Write
Quality
CODE