Requirements

• A basic understanding of GNU/Linux

• A basic understanding of YAML and Jinja2

• A GNU/Linux distribution (you can run everything in your laptop/desktop even if we are going to use servers here)

• A passion for automation :-)

Python

YAML

Jinja2

SaltStack vs Puppet vs Chef vs Ansible

Additional notes

• (1): Salt uses a Salt Master on the master node and a Salt minion on the slave node. When the user setup a configuration, the Salt minion daemon, receives commands from a remote Salt master.

• (2): Salt uses Salt minion on the remote node to control it

• (3): The minion can act as a masterless node, if the user setup the right configurations.

• (4): Salt ssh is a Salt module that allows executing Salt commands and states over ssh without installing a salt-minion. The same module can help the user to install and configure a remote Salt minion.

• (5): Declarative programming is where the developer says what he/she want without having to say how to do it. However, in procedural programming, the developer has to specify the exact steps to get the result.

• (6): Immutability in infrastructure is an approach wherein components are replaced rather than changed.

Installing and Configuring Salt

HOL: Installing & Configuring Salt

Conclusion & Insights

Overview of Salt Features 

Remote execution

Event driven infrastructure

Configuration management

Salt Components

• Salt master

• Salt minion

• Executions modules

• Formulas (states)

• Grains

• Pillar

• Top file

• Runners

• Returners

• Reactor

• Salt Cloud / Salt Virt

• Salt ssh

Salt Master

Salt Minion

Salt Modules

Salt States/Formulas

Salt Grains

Salt Pillar

Top File

Salt Runners

Salt Returners

Salt Reactors & Beacons

Salt Virt

Salt SSH

Salt Proxy

HOL: Obtaining System Information with Salt Grains

Salt compound matchers

HOL: Remote Execution

Salt Command Anatomy

Conclusion & Insights

Understanding State and the State Ecosystem

Creating, Managing & Testing State with Salt (HoL)

Salt State

Making Salt States Reusable with Pillar

Pillar

• Like grains, pillars are tree-like structures of data, however, they are defined on the Salt Master and passed through to minions.

• They are mainly useful to store confidential data securely, they allow sending this data to only to the relevant minion.

• They also allow us to store any data that need to be assigned to specific minions or groups of minions and accessed inside SLS formulas and template files.

HOL: Pillar Manipulation

HOL: Using Pillar Data in Salt States

Orchestration and Mining - Salt Mine

Salt Mine

• Imagine a scenario where a minion needs to get data from another one. We need a peer publishing system to send data from one minion to another one. While this solution is feasible, it may encounter a problem of slowness in the case a minion should send data to multiple other minions: Imagine a data center with 1000 minions.

• Rather than having a minion reach all of the other minions, having a single large node that could synchronize and orchestrate all of this data is a good solution. This node can be the master.

• SaltStack solves this problem by implementing Salt mine, a system that is very similar to grains but faster and more up-to-date. Also, grains does not need to be refreshed very frequently because of their static nature.

HOL: Using Salt Mine

Managing Hosts with Salt SSH

Salt SSH

• Salt ssh is very easy to use.

• It allows executing salt commands and states over ssh without installing a salt-minion. Salt ssh is production ready in the version 2014.7.0.

• The only thing you need on the agentless (without salt-minion) host is Python.

• The master node should be able to access all of the managed hosts using ssh.

HOL: Configuration of Salt SSH and Remote Execution

HOL: Installing Nginx Using Nginx-Formula and Salt SSH

Conclusions and Insights

Strategies for Scaling the Infrastructure

Salt at Scale

According to Salt official documentation, the most common problems on the Salt Master are:

• Too many minions authing at once: acceptance_wait_time, acceptance_wait_time_max

• Too many minions re-authing at once: random_reauth_delay

• Too many minions re-connecting at once: recon_default, recon_max, recon_randomize (Tue|False)

• Too many minions returning at once: salt '*' disk.usage -b 50

• Too few resources:

  • keysize: 2048 (default: 4096) 

  • keep_jobs: 48

  • cachedir: /var/salt/cache/ (SSD)

  • ext_job_cahe: redis

  • master_job_cache: mysql

HOL: Masterless Salt

salt-call disk.usage

=

salt 'minion' disk.usage

Salt Syndic

The common way to use Salt is having a master and many minions, but you can also have as many masters as you want since a minion can be managed by multiple masters.

For greater flexibility and in order to scale more, you can use Salt syndic, which acts as a passthrough minion node.

Like any master node, a syndic node run the Salt master daemon, at the same time, a Salt syndic daemon will run in the same system.

You can create special Salt topologies using Salt syndic.

Salt Syndic

Salt syndic works as follow:

• It controls a group of lower-level minion nodes

• It (the syndic daemon) connects a higher level master node

Using this topology, a master can be called, in some cases, a master of masters.

Monitoring with Salt

• Salt has built-in modules that allows getting information about minion nodes.

• For instance, Salt ps is a salt interface to psutil which is a cross-platform lib for process and system monitoring in Python.

• Salt Service module can also help in watching services since it can check whether a service is running or not.

• We already have seen some examples in SLS files, but we can use it from a Python script on the CLI.

Example, we have a Redis server and we want to check its status.

• This is done using the Redis module which provides Redis functionality to Salt.
• This module requires the Redis python module and should be configured on the minion.
• Salt provides modules to different other platforms and software like RabbitMQ or mysql.
• You can also build your own module using a Python script that returns data.

You can ship data to different data stores

• Mysql
• Cassandra
• Redis
• Nagios
• Postgres
• SQLite3
• Syslog
• Telegram
• Xmpp
• Slack

The Salt status module returns metrics and information about a targeted minion and can be used in monitoring.

Using Monitoring States

Managing the Salt Schedule on a Minion

Monitoring with Beacons

Example: Posting to Slack When a File Change

HOL: Using Salt ps

Best Practices

Rules

1. Modularity and clarity should be emphasized whenever possible.

2. Create clear relations between pillars and states.

3. Use variables when it makes sense but don't overuse them.

4. Store sensitive data in pillar.

5. Don't use grains for matching in your pillar top file for any sensitive pillars.

Using Includes and Extends in Salt States

Conventions

Following the SLS Namespace Guidelines 

Organizing Grains

Using the Documentation

Follow the Official Conventions to Write Formulas

Start Your Cheat Sheet :)

The Usual Lecture From The Local System Administrator :)