hiera 101

A basic introduction

Francisco Martínez
Puppet Camp Barcelona 2013

CODE AND DATA SEPARATION

When writing software,

it is useful to keep data and code separated

This allows for reusability

Code (Behaviour)

Data (Configuration)

Usually, in the different environments of a project, you have copies of you machines with very similar behavior

BUT

different data, as machines will refer to each other

environments

example: environments

In the example, the backend in development will have a connection string refering to db-dev

mongodb://db-dev:2500/?replicaSet=test

while the other backends point to their respective db servers

mongodb://db-prod:2500/?replicaSet=real

How to configure the backend is code.

The reference to the db server is data.

(To be more specific, environment-specific data)

there's more than one way to do it

Worthwhile read IMHO:

https://puppetlabs.com/blog/the-problem-with-separating-data-from-puppet-code/

Proposes different solutions to the problem

Node inheritance
Parametrized Classes
External Node Classifier
Extlookup
Hiera

hiera to the rescue!

Hiera is a key/value lookup tool for configuration data

$dnsserver = hiera('dnsserver')

Allows to get data from different data sources and establish a hierarchy among them

:hierarchy:
    - location
    - global

Look into "location" and if I don't find my key, go search for it in "global"

Dynamic data sources

Let's dive in the configuration file (hiera.yaml):

---
:backends: 
   - yaml
:hierarchy: 
   - %{::clientcert}
   - %{::environment}
   - virtual_%{::is_virtual}
   - common
:yaml:
   :datadir: /etc/puppet/hieradata

In :hierarchy: we put data sources

static (common)
dynamic (reference facts)

Order matters!

Data sources

development.yaml

db_connect_string: 'mongodb://db-dev:2500/?replicaSet=devel'
backend_base_url: 'http://backend-devel.example.com/apirest'
puppet_interval: '5'

production.yaml

db_connect_string: 'mongodb://db-prod:3500/?replicaSet=prod'
backend_base_url: 'http://backend-prod.example.com/apirest'
puppet_interval: '720'

nameserver.example.com.yaml

puppet_interval: '120'

virtual_true.yaml

puppet_interval: '60'

common.yaml

puppet_interval: '15'

dynamically look up parameters based on facts

Dynamic data sources are based on facts, so we

can make fact-dependent hiera files.

Specify different tcp tunning values depending on the kernelversion fact
Use different dns servers depending on the is_virtual fact
Connecting to one server or another depending on the machine having selinux on
And all these without using if or case clauses :)

Backends

A backend is how hiera implements the actual storage of the data

Ships with support for:

JSON
YAML

But there's around other backends

redis
mysql
gpg
puppet

implications on puppet code

Using hiera means that configuration (data that changes in the different instantiations of your class) can be managed either

by parameters
by hiera

So, instead of:

class web_server ($port = 80) {...

You might want to use

class web_server ( ) {
   hiera('web_server::port')
}

Tricks

non scalar data

The value of a query can be another structure (hash of hashes, for example):

openstack::all: 
   admin_email: devnull@example.com
   keystone_admin_token: G6943LMReKj_kqdAVrAiPbpRloAfE1fqp0eVAJ
   rabbitmq: 
      user: rabbitmq
      password: secret

Can then be queried via

$data = hiera_hash('openstack::all') 
$rabbitmq_user = $data['rabbitmq']['user']

merging data

hiera(): performs a simple lookup, stops searching after it finds a value
hiera_array(): returns a flattened array with all the matching values
hiera_hash(): returns one merged hash with all the values returned

include

hiera_include(): looks up as in hiera_array() and then includes each classname returned

web01.example.com.yaml

---
classes:
   - apache
   - redis
   - wordpress

common.yaml

---
classes:
   - base::linux

site.pp

hiera_include(classes)

puppet 3

Include

Starting in Puppet 3, the include function will rely on external data for parameters

When a class is declared, Puppet will try the following:

Request a value from hiera, using the key <class name::parameter name>
Example: apache::version
Use the default value
Fail compilation with an error if no value can be found

ENC-like behavior

You can use this feature in Puppet 3 and the hiera_include() function to set up an ENC-like behavior with hiera

site.pp

node default {
   hiera_include("classes")
}

datacenter-mad.yaml

---
classes:
   - ntp
ntp::server: ntp-mad.example.com

datacenter-bcn.yaml

---
classes:
   - ntp
ntp::server: ntp-bcn.example.com

q & A

BACKUP SLIDES

EXTERNAL NODE CLASSIFIER

Binding of nodes based on the certname:

node "devel-*" {

The ENC is a script that puppet can use to decide, dynamically and based on external data sources, which configuration each node gets.

So you can decide which configuration the node receives the manifests to get configured as a webserver on an external ldap server, for example.

YAML

YAML is a language to serialize data.

Similar to XML, but less verbose and more readable.

Allows to modelize structured data:

--- # Favorite movies, Array/List
- Matrix
- Sucker Punch
- Scott Pilgrim vs. the World
- American Psycho

--- # domains and countries, Hash/Dictionary
de: 'Germany'
sk: 'Slovakia'
hu: 'Hungary'
us: 'United States'
no: 'Norway'

environments

Puppet supports the concept of "environment".

Environments are supplied by the host or the ENC and decide which "root" of configuration to use.

So you can keep development code completely separated from staging or production code, and play with your development environment day-to-day, and decide when to put that code into production.

caveats

Stablishing a hiearchy for hiera is terribly complicated, and you could easily end up making a mess

Hiera as a source of external data is flexible, and has many advantages. Nonetheless, it's easily an overkill in many setups where you don't need that degree of flexibility: Think KISS