SaltStack Essentials

Requirements

  • A basic understanding of GNU/Linux

  • A basic understanding of YAML and Jinja2

  • A GNU/Linux distribution (you can run everything in your laptop/desktop even if we are going to use servers here)

  • A passion for automation :-)

Python

nterms = 10

n1 = 0
n2 = 1
count = 0

print("Fibonacci sequence upto",nterms,":")
while count < nterms:
  print(n1,end=' , ')
  nth = n1 + n2
  # update values
  n1 = n2
  n2 = nth
  count += 1

YAML

invoice: 34843
date   : 2001-01-23
bill-to: &id001
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
ship-to: *id001
product:
    - sku         : BL394D
      quantity    : 4
      description : Basketball
      price       : 450.00
    - sku         : BL4438H
      quantity    : 1
      description : Super Hoop
      price       : 2392.00
tax  : 251.42
total: 4443.52
comments: >
    Late afternoon is best.
    Backup contact is Nancy
    Billsmer @ 338-4338.

Jinja2

{% if var == 'value' %}
key_one:
    key_two:
        - value_one
{% elif var < 'value' %}
key_three:
    key_four:
        - value_two
{% else %}
key_five:
    key_six:
        - value_three
{% endif %}

SaltStack vs Puppet vs Chef vs Ansible

 

SaltStack Ansible Puppet Chef
Why
(Motivation) Thomas S. Hatch, the author of Salt, had previously solved the problem of systems management at scale by creating tools for companies and teams, but found these and other open source solutions to be lacking Ansible were created to provide an agent-less configuration and remote execution tool Luke Kanies founded Puppet and Puppet Labs in 2005 out of fear and desperation, with the goal of producing better operations tools and changing how we manage systems. Chef was created by Adam Jacob as a tool for his company Opscode to simplify building end-to-end server/deployment tools. The tool turned out to have more potential
How
(Terminology) Salt uses States as directives scripted in SLS (Salt States) Formulas. It has a master that we call Master and children called Minions. Anislbe uses Tasks as directives scripted in Playbooks. It has a master that we call the Control Machine and children called Hosts. Puppet uses Resources as directives scripted in Manifests. It has a master that we call Master and children called Agents. Puppet uses Resources as directives scripted in Recipes and Cookbooks. It has a master that we call Server and children called Clients.
Who
(Companies Using this Technology) CloudFlare, Paypal Cisco, NASA Wallmart, Salesforce IMDB, Splunk
License Apache License v2 GNU Public License v3 Apache License v2 Apache License v2
Github Stars 10k 37.5k 5.5k 5.8k
Programming Language Python Python C++, Clojure Ruby, Erlang
Configuration Language YAML YAML, JSON Proprietary Ruby
Database N/A N/A PuppetDB PosgtreSQL
Deployments (1)
(Push/Pull) Push Push Pull Pull
Transport ZeroMQ SSH Mcollective RabbitMQ
Agent (2) Yes (Optional) No Yes Yes
Agentless (3) Yes Yes No No
Public Clouds
(AWS, Azure, GCP) Yes (Salt Cloud) Yes Yes Yes
Entreprise GUI SaltStack Entreprise Ansible Tower Puppet Entpreprise Opscode Manage
Open Source GUI SaltPad Semaphore Foreman Chef Manage
High Availability Yes Yes Yes Yes
Interoperability High High High High
Child Nodes Bootstrapping (4) Yes (Using salt-ssh) Yes No No
Remote Execution Built-in Built-in Mcollective Knife
Architecture Client Only Client Only Client/Server Client/Server
Language Type (5)
(Procedural/Declarative) Declarative Procedural Declarative Procedural
Mutability (6) Mutable Mutable Mutable Mutable

Additional notes

  • (1): Salt uses a Salt Master on the master node and a Salt minion on the slave node. When the user setup a configuration, the Salt minion daemon, receives commands from a remote Salt master.

  • (2): Salt uses Salt minion on the remote node to control it

  • (3): The minion can act as a masterless node, if the user setup the right configurations.

  • (4): Salt ssh is a Salt module that allows executing Salt commands and states over ssh without installing a salt-minion. The same module can help the user to install and configure a remote Salt minion.

  • (5): Declarative programming is where the developer says what he/she want without having to say how to do it. However, in procedural programming, the developer has to specify the exact steps to get the result.

  • (6): Immutability in infrastructure is an approach wherein components are replaced rather than changed.

Installing and Configuring Salt

 

Legend:

  • 1 Automated and manual testing-

  • 2 Manual testing only

  • 3 Package testing only

  • 4 Automated testing only

  • 5 No OS-specific testing or packages, use RedHat/CentOS packages

  • 6 Available to SaltStack Enterprise Customers only

  • 7 Only supported by Salt 2016.3.0 and later

  • 9 Only supported by Salt 2017.7.1 and later

  • 10 Only supported by Salt 2015.8.11 and later

  • 11 Only supported by Salt 2016.3.4 and later

  • 12 Only supported by Salt 2016.3.5 and later

  • 13 Only supported by Salt 2017.7.3 and later

  • 14 Only supported by Salt 2017.7.6/2018.3.1 and later

  • 15 Only supported by Salt 2018.3.4/2019.2 and later

 

 

Full Support means that:

  • Packages and all required dependencies created by SaltStack or official upstream packager

  • Packages hosted by SaltStack

  • Tested by SaltStack

  • SaltStack provides full technical support for SaltStack customers

  • SaltStack will fix bugs for SaltStack customers

Reasonable -Effort Support means:

  • Packages created and hosted by Salt community

  • Some testing done by SaltStack

  • SaltStack provides best-effort technical support for SaltStack customers

  • SaltStack may fix bugs for SaltStack customers

HOL: Installing & Configuring Salt

Conclusion & Insights

Overview of Salt Features 

Remote execution

 

Event driven infrastructure

 

Configuration management

Salt Components

  • Salt master

  • Salt minion

  • Executions modules

  • Formulas (states)

  • Grains

  • Pillar

  • Top file

  • Runners

  • Returners

  • Reactor

  • Salt Cloud / Salt Virt

  • Salt ssh

Salt Master

Salt Minion

Salt Modules

Salt States/Formulas

Salt Grains

Salt Pillar

Top File

base:
  'minion':
    - nginx

Salt Runners

Salt Returners

Salt Reactors & Beacons

Salt Virt

Salt SSH

Salt Proxy

HOL: Obtaining System Information with Salt Grains

Salt compound matchers

HOL: Remote Execution

Salt Command Anatomy

Conclusion & Insights

Understanding State and the State Ecosystem

# This state is usually saved under `/etc/salt/apache.sls` or `/etc/salt/apache/init.sls`.



apache:
  pkg.installed: []
  service.running:
    - require:
      - pkg: apache

# salt 'minion' state.highstate
# The executed states will be read from the top file.
base:
  'minion':
    - nginx
    - apache
    - my_custom_sls_1
    - my_custom_sls_2
    - my_custom_sls_3
    [...]
    - my_custom_sls_N

# Another example:
base:
  '*':
    - nginx
    - apache
    - my_custom_sls_1
    - my_custom_sls_2
    - my_custom_sls_3
    [...]
    - my_custom_sls_N

# Another example with targeting:
base:
  'G@os:Ubuntu and minion* or S@192.168.1.*':
    - nginx
    - apache
    - my_custom_sls_1
    - my_custom_sls_2
    - my_custom_sls_3
    [...]
    - my_custom_sls_N
# To create multiple environments, say (dev, test and prod), modify the `/etc/salt/master` and add the following lines:

file_roots:
  dev:
    - /srv/salt/dev
  test:
    - /srv/salt/test
  prod:
    - /srv/salt/prod


#Now the top file should reference these environments

dev:
  'webserver-dev-*':
    - webserver
  'db-dev-*':
    - db
test:
  'webserver-test-*':
    - webserver
  'db-test-*':
    - db
prod:
  'webserver-prod-*':
    - webserver
  'db-prod-*':
    - db

base:
  '*':
    - tools.sls
    
dev:
  'webserver-dev-*':
    - webserver
  'db-dev-*':
    - db
    
test:
  'webserver-test-*':
    - webserver
  'db-test-*':
    - db
    
prod:
  'webserver-prod-*':
    - webserver
  'db-prod-*':
    - db
# A good way to organize your environment + states and pillar is the following:

/srv/salt
  /dev
    /states
    /pillars
  /test
    /states
    /pillars
  /prod
    /states
    /pillars

Creating, Managing & Testing State with Salt (HoL)

Salt State

Making Salt States Reusable with Pillar

Pillar

  • Like grains, pillars are tree-like structures of data, however, they are defined on the Salt Master and passed through to minions.

  • They are mainly useful to store confidential data securely, they allow sending this data to only to the relevant minion.

  • They also allow us to store any data that need to be assigned to specific minions or groups of minions and accessed inside SLS formulas and template files.

HOL: Pillar Manipulation

HOL: Using Pillar Data in Salt States

Orchestration and Mining - Salt Mine

Salt Mine

  • Imagine a scenario where a minion needs to get data from another one. We need a peer publishing system to send data from one minion to another one. While this solution is feasible, it may encounter a problem of slowness in the case a minion should send data to multiple other minions: Imagine a data center with 1000 minions.

  • Rather than having a minion reach all of the other minions, having a single large node that could synchronize and orchestrate all of this data is a good solution. This node can be the master.

  • SaltStack solves this problem by implementing Salt mine, a system that is very similar to grains but faster and more up-to-date. Also, grains does not need to be refreshed very frequently because of their static nature.

HOL: Using Salt Mine

Managing Hosts with Salt SSH

Salt SSH

  • Salt ssh is very easy to use.

  • It allows executing salt commands and states over ssh without installing a salt-minion. Salt ssh is production ready in the version 2014.7.0.

  • The only thing you need on the agentless (without salt-minion) host is Python.

  • The master node should be able to access all of the managed hosts using ssh.

HOL: Configuration of Salt SSH and Remote Execution

 

HOL: Installing Nginx Using Nginx-Formula and Salt SSH

 

Conclusions and Insights

 

Strategies for Scaling the Infrastructure

 

Salt at Scale

According to Salt official documentation, the most common problems on the Salt Master are:

  • Too many minions authing at once: acceptance_wait_time, acceptance_wait_time_max

  • Too many minions re-authing at once: random_reauth_delay

  • Too many minions re-connecting at once: recon_default, recon_max, recon_randomize (Tue|False)

  • Too many minions returning at once: salt '*' disk.usage -b 50

  • Too few resources:

    • keysize: 2048 (default: 4096) 

    • keep_jobs: 48

    • cachedir: /var/salt/cache/ (SSD)

    • ext_job_cahe: redis

    • master_job_cache: mysql

HOL: Masterless Salt

salt-call disk.usage

=

salt 'minion' disk.usage

Salt Syndic

The common way to use Salt is having a master and many minions, but you can also have as many masters as you want since a minion can be managed by multiple masters.

 

For greater flexibility and in order to scale more, you can use Salt syndic, which acts as a passthrough minion node.

 

Like any master node, a syndic node run the Salt master daemon, at the same time, a Salt syndic daemon will run in the same system.

 

You can create special Salt topologies using Salt syndic.

 

Salt Syndic

 

Salt syndic works as follow:

  • It controls a group of lower-level minion nodes

  • It (the syndic daemon) connects a higher level master node

 

 

 

 

Using this topology, a master can be called, in some cases, a master of masters.

 

Monitoring with Salt

 

  • Salt has built-in modules that allows getting information about minion nodes.

 

 

  • For instance, Salt ps is a salt interface to psutil which is a cross-platform lib for process and system monitoring in Python.

 

 

  • Salt Service module can also help in watching services since it can check whether a service is running or not.

 

  • We already have seen some examples in SLS files, but we can use it from a Python script on the CLI.

Example, we have a Redis server and we want to check its status.

 

salt-call service.status redis-server
  • This is done using the Redis module which provides Redis functionality to Salt.
  • This module requires the Redis python module and should be configured on the minion.
  • Salt provides modules to different other platforms and software like RabbitMQ or mysql.
  • You can also build your own module using a Python script that returns data.

You can ship data to different data stores

  • Mysql
  • Cassandra
  • Redis
  • Nagios
  • Postgres
  • SQLite3
  • Syslog
  • Telegram
  • Xmpp
  • Slack

The Salt status module returns metrics and information about a targeted minion and can be used in monitoring.

salt '*' status.loadavg
salt '*' status.cpustats
salt '*' status.meminfo
salt '*' status.vmstats

Using Monitoring States

schedule:
  diskusage_monitoring:
    function: status.diskusage
    minutes: 10
    returner: mysql  
schedule:
  diskusage_monitoring:
    function: status.diskusage
    minutes: 10
    returner: mysql  
  schedule.present:
    - function: status.loadavg
    - minutes: 10
    - returner: mysql

Managing the Salt Schedule on a Minion

# Schedule is a Salt module that you can manage via Salt CLI:

salt '*' schedule.add my_job function='test.ping' seconds=3600


# Or

salt '*' schedule.add sched_monitoring function='cmd.run' \ 
job_args="['wget \"www.website.com\" --timeout 30 -O - 2>/dev/null \
| grep \"Salt\" || echo \"The site is down\"  | mail -s \"Your site is down\" \
admin@website.com']"seconds=10

# To enable a job

salt '*' schedule.enable_job sched_monitoring

# To disable it

salt '*' schedule.disable_job sched_monitoring

Monitoring with Beacons

beacons:
  inotify:
    /tmp/important_file_to_watch:
      mask:
        - modify
        - delete_self
salt/beacon/minion/inotify//tmp/services	{
    "_stamp": "2019-05-22T18:36:57.860459",
    "change": "IN_MODIFY",
    "id": "minion",
    "path": "/tmp/important_file_to_watch"
}
beacons:
  haproxy:
    - backends:
        www-backend:
            threshold: 45
            servers:
              - web1
              - web2
    - interval: 120
  memusage:
    beacon.present:
      - percent: 75%
      - interval: 30

Example: Posting to Slack When a File Change

# Open `/srv/reactor/reaction.sls`


post_to_slack:
  local.slack.post_message:
    - tgt: {{ data['id'] }}
    - kwarg:
        channel: "#general"
        api_key: "xxxxxxxxxxxx"
        message: "your message"
        from_name: "bot"
# Now open `/etc/salt/master.d/reactor.conf` and add:


reactor:
  - 'salt/beacon/*/memusage/':
    - '/srv/reactor/reaction.sls'

# You can setup the reactor to watch other events 
# such us the modification of the file `/tmp/services`:



reactor:
  - 'salt/beacon/minion/inotify//tmp/services':
    - '/srv/reactor/reaction.sls'

HOL: Using Salt ps

Best Practices

 

Rules

 
  1. Modularity and clarity should be emphasized whenever possible.

  2. Create clear relations between pillars and states.

  3. Use variables when it makes sense but don't overuse them.

  4. Store sensitive data in pillar.

  5. Don't use grains for matching in your pillar top file for any sensitive pillars.

Using Includes and Extends in Salt States

Conventions

Following the SLS Namespace Guidelines 

Organizing Grains

Using the Documentation

Follow the Official Conventions to Write Formulas

Start Your Cheat Sheet :)

The Usual Lecture From The Local System Administrator :)

HOL: Troubleshooting

Conclusion

SaltStack Essentials

By eon01

SaltStack Essentials

  • 1,265