Puppet 4 migration and Invenio modules

What?

• How to Puppet3 -> Puppet4
• New features
• Testing puppet code
• Automating  the tests (CI)
• Future of the Invenio modules

How to Puppet3 -> Puppet4

First of all

Run the tests!!

Wait for the next section for the how to, but that will pop up MOST issues

Hiera lookups on templates

Use the new lookup function

Load into local variable before calling the template

# Instead of
some data <%= scope.hiera_lookup('::myclass::myvalue') %>

# Use
some data <%= lookup('::myclass::myvalue') %>

# Or better
some data <%= @myvalue %>
# And then on the .pp file that uses this template:
#
# $myvar = hiera('::myclass::myvalue')
# $content = template('mymodule/mytemplate.erb')
# 

Variable namespace on templates

# Given this class:
----- mymodule/myclass.pp
class mymodule::myclass {
    $anothervar = 'something'
    notify { template('anothermodule/anothertemplate.erb')
}
-----

---- anothermodule/templates/anothertemplate.erb 
This is the template from anothermodule, <%= @anothervar %>
----


# with puppet3
----
This is the template from anothermodule, something
----

# BUT!!! in puppet4
----
This is the template from anothermodule,
----

No dynamic namespaces anymore!

Variable namespace on templates

CERN ONLY: Careful with teigi secrets, they might end up being empty

# Instead of passing the template name

  teigi::secret::sub_file{"/file_with_secrets":
    template   => 'mymod/template.erb',
    teigi_keys => $tbags,
    owner      => $user,
    group      => $group,
    mode       => '0640',
  }

# use the new 'content' parameter

  teigi::secret::sub_file{"/file_with_secrets":
    content    => template('mymod/template.erb'),
    teigi_keys => $tbags,
    owner      => $user,
    group      => $group,
    mode       => '0640',
  }

Empty string == True!!

# This same snippet

$port = ''
if $port {
  notify {"Got port ${port}": }
} else {
  notify {'No port passed!': }
}

# On puppet 3
[test@0fc2514c4e5d code]$ bundle exec puppet --version
3.8.7

[test@0fc2514c4e5d code]$ bundle exec puppet apply test.pp 
Notice: Compiled catalog for 0fc2514c4e5d.home in environment production in 0.01 seconds
Notice: No port passed!
Notice: /Stage[main]/Main/Notify[No port passed!]/message: defined 'message' as 'No port passed!'
Notice: Finished catalog run in 0.01 seconds


# But on puppet 4
[test@0815d666e8ff code]$ bundle exec puppet --version
4.8.2

[test@0815d666e8ff code]$ bundle exec puppet apply test.pp 
Notice: Compiled catalog for 0815d666e8ff.home in environment production in 0.10 seconds
Notice: Got port 
Notice: /Stage[main]/Main/Notify[Got port ]/message: defined 'message' as 'Got port '
Notice: Applied catalog in 0.02 seconds

New featues!

Data types

Why?

Data types

Basic (core) data types:

• String

• Integer

• Float

• Numeric

• Boolean

• Array

• Hash

• Regexp

• Undef

• Default

# Use them everywhere!

class dummy(
  String name,
  Integer age,
  Array[Integer] grades,
  Hash extra_params,
) {
...
}

# A combo with Default and bool-string handling:

$use_feature = $feature ? {
  Boolean => $feature,
  String  => str2bool($feature),
  Default => true,
}

Data types

Flexible data types:

• Variant

• Pattern

• Optional

• NotUndef

• Enum

• Tuple

• Struct

class myclass(
  # Might be string or integer
  Variant[String, Integer] port = 12,

  # Match any sting starting with lowercase letter
  Pattern[/\A[a-z].*/] lowercased = 'aOEU',

  # Might be undef or string
  Optional[String] might_be_undef = undef,

  # Might be anything but undef
  NotUndef not_undef = 'something',

  # Matches either string 'one' or 'two'
  Enum['one', 'two'] how_many = 'one',

  # Specify one by one the types of each element
  Tuple[String, String, Integer] triplet = ['one', 'two', 3],

  # Definition of a hash structure
  Struct[{
    name => String,
    home => String,
    uid  => Integer,
  }] user,
)

Data types

Parent data types:

• Scalar

• Collection

• Data

• Catalogentry

• Type

• Any

Unusual data types:

• Callable

Data types: restrictions

# Integer ranges
Integer[1,1024] $listen_port = 22

# Enums
Enum['*','::1','127.0.0.1'] $listen_ip = '*',

# Patterns
Pattern[/^\/.*/] $home

# Complex hashmaps
Hash[String, Struct[{ uid => Integer, home => Pattern[/^\/.*/]}]] $hash

# That matches for example:
# {
#   uid  =>  0,
#   home => '/root',
# }
#
# but not
# {
#   uid  => 'root',
#   home => 'my/relative/path',
# }

EPP templates

EPP == Embedded puppet templates

Some diffs with erb:

• Puppet style vars
• Typed parameters
• Can use any func

------ example.epp file --------
<% |  String $text, $Array $array, Boolean $bool |>
<% if $string -%>
Text from variable: <%= $string %>
<% end -%>
<% if $array -%>
<% $array.each |$element| { -%>
Array item: <%= $element %>
<% } -%>
<% end -%>
<% if $bool -%>
Bool value is true
<% end -%>
---------------------------------

------- .pp code using it -------
  file { '/dummy':
    ensure  => file,
    content => epp(
      'test/example.epp',
      {
        text  => 'foo',
        array => ['one', 'two'],
        bool  => false
      }),
  }
---------------------------------

Heredocs

Nicer strings!

class my_example {
  $content = @("EOF")
    This is a very long string
    with some variables on it, for example:
      -> here is the fqdn ${::fqdn)
    You will also notice that the indentation
    in front of the string will be removed, that
    can be avoided (if wanted), by not using the
    pipe right before the end-of-file tag string.
    | EOF
  
  file { '/example':
    ensure  => file,
    content => $content,
  }
}

Lambdas and new functions

Finally loops!! (kinda)

• each
• map
• filter
• reduce
• slice

$data = ["routers", "servers", "workstations"]
$data.each |$item| {
 notify { $item:
   message => $item
 }
}

# For the hash $data, return a hash containing all
# values of keys that end with "berry"
$data = {
  "orange"    => 0,
  "blueberry" => 1,
  "raspberry" => 2
}
$filtered_data = $data.filter |$items| {
  $items[0] =~ /berry$/
}
# $filtered_data = {blueberry => 1, raspberry => 2}


$data = [1,2,3]
$transformed_data = $data.map |$items| { $items * 10 }

$sum = $data.reduce |$memo, $value| { $memo + $value }

slice([1,2,3,4,5], 2) # produces [[1,2], [3,4], [5]]

Testing puppet code

Puppetlint

Checking that the code looks nice

Puppet parse

Checking that puppet understands the code

Rspec

Unit test the s#!$ out of it

Acceptance

To be defined...

possibly with beaker

Acceptance - kind of*

*CERN specific

• Create new environment
  • Add file to it-puppet-environments
  • Duplicate any hiera data on your repo for the new env
• Override module and hostgroup branches on the environment file
• Move one node to the new environment (foreman)
• Run puppet on it
• Rinse and repeat

Run them all locally

With docker

Dockerfile:

# Puppet 3 build:
#     docker build --build-arg puppet_version=3.0 Dockerfile
#
# Puppet 4 build:
#     docker build --build-arg puppet_version=4.8.0 Dockerfile
#
# Latest Puppet 4 build:
#     docker build --build-arg puppet_version=4.0 Dockerfile
#
FROM gitlab-registry.cern.ch/ai/it-puppet-module-ci_images:cc7-puppet-ci-master

ARG puppet_version=3.0

WORKDIR /code/code

ENV BUNDLE_GEMFILE=../ci/Gemfile
ENV PUPPET_VERSION="~> ${puppet_version}"
ENV LANG=en_US.UTF-8

# Run tests
RUN useradd test
USER test
ENTRYPOINT rm -f ../ci/Gemfile.lock \
&& bundle install --local --without system_tests development \
&& bundle exec rake --rakefile ../ci/Rakefile test

Run them all locally

With docker

docker-compose.yml:

version: '2'
# To run the tests run the service for the puppet version you want:
#     docker-compose run --rm puppet3
#
# If not already built, it will build the images for you.

services:
  puppet3:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        puppet_version: 3.0
    container_name: puppet3_ci
    volumes:
        - "$PWD:/code"

Run them all locally

With docker

docker-compose.yml (continued):

  puppet4:
    extends: puppet3
    build:
      args:
        puppet_version: 4.8.0
    container_name: puppet4_ci

  puppet4_latest:
    extends: puppet3
    build:
      args:
        puppet_version: 4.0
    container_name: puppet4_latest_ci

Run them all locally

With docker

$ docker-compose run puppet3

$ docker-compose run puppet4

$ docker-compose run puppet4_latest

Automating the tests (CI) *

* CERN specific

The modulesync branch

Maintained by CERN config management team

Automatic updates (as merge requests)

https://gitlab.cern.ch/ai/it-puppet-modulesync-configs

The modulesync branch

The modulesync branch