SaltStack @ Aetna

Tehmasp Chaudhri | @tehmaspc

Team Background

• We are the Aetna (Consumer Digital) Operations team

• We are an Agile Operations team (currently) consisting of:

  • 3 Site Reliability Engineers (SRE)

  • 2 Cloud Systems Developers (CSD)

  • Product Owner

  • ScrumMaster

  • Director of Operations

• We build and maintain Utopia, an internal developer platform-as-a-service (PaaS)

• We employ SaltStack for managing all our infrastructure needs for Utopia (and have been using it for almost a year)

• We try not to take ourselves too seriously but we're serious about what we do

Why Salt?

• Speed of execution

• Flexibility (not as opinionated as other CM tools)

• Extensibility

• Environments support

• No steep learning curve (initial familiarity in team)

• Salt-Cloud & Salt-API

• Community & Support

Salt Architecture

• 2014.7.5 (Helium)

• Single Salt Master in a management AWS VPC​

  • ​could scale out to environment Syndics

• ~100 Minions currently

• Environments: {Base, Dev, Stg, Prd, Inf} 

• Well defined server naming scheme:

  • ​e.g. dev-use1a-pr-01-app-db-0001

Network Topology

Salt Formula Design

• Every formula in its own repository

• Use settings.sls to define good defaults as well as Pillar and Grain lookups

• Use Pillars and Grains consistently across all formulas:

  • ​Pillars for common infrastructure configuration

  • Grains for node (or cluster) specific configuration

• Documentation

  • Every repo should have a proper README

  • README should describe how to use the formula

  • README should provide admin info (if applicable)

• Testing

  • Isolate testing setup into .vagrant-salt/

  • Manage dependencies with vagrant-saltdeps

  • Manage integration tests with serverspec

Salt Formula Demo

Environments, Top.sls & Pillars

• We employ multiple environments in order to allow us to test (breaking) changes:

  • {Base, Dev, Stg, Prd, Inf}

• ​Our Base environment only includes a single formula, core, which we apply to all minions.

• Reduce top.sls file changes (for States and Pillars) by templating it out and backing it with a Minion's Grain data

• Use a separate repository for Pillar data

{%- raw %}
{% set roles = salt['grains.get']('roles', '') %}
{%- if roles %}
{%- endraw %}
{{ env }}:
  'env:{{ env }}':
{%- raw %}
    - match: grain
    {%- for role in roles %}
    - {{ role }}
    {%- endfor %}
{% endif %}
{%- endraw %}

Example Environment top.sls

Salt-Cloud Design

• Leverage Salt-Cloud for spinning up/down all servers

• Maintain a separate repository for Salt-Cloud configuration

• Use our own Python DSL for simplifying Salt Cloud configurations

• Use Grains extensively to manage Minions

Salt Cloud Demo

Shortcomings with Salt

• Highly active project which seems to introduce a lot of bugs

• Salt documentation can be out of sync with actual implementation

• Salt-Cloud improvements are desired

• Salt environments can be problematic for some

Future Goals with Salt

• Salt-API

• Reactor

• Enterprise GUI

Questions?