SaltStack @ Aetna

Tehmasp Chaudhri | @tehmaspc

Team Background

  • We are the Aetna (Consumer Digital) Operations team

  • We are an Agile Operations team (currently) consisting of:

    • 3 Site Reliability Engineers (SRE)

    • 2 Cloud Systems Developers (CSD)

    • Product Owner

    • ScrumMaster

    • Director of Operations

  • We build and maintain Utopia, an internal developer platform-as-a-service (PaaS)

  • We employ SaltStack for managing all our infrastructure needs for Utopia (and have been using it for almost a year)

  • We try not to take ourselves too seriously but we're serious about what we do

Why Salt?

  • Speed of execution

  • Flexibility (not as opinionated as other CM tools)

  • Extensibility

  • Environments support

  • No steep learning curve (initial familiarity in team)

  • Salt-Cloud & Salt-API

  • Community & Support

Salt Architecture

  • 2014.7.5 (Helium)

  • Single Salt Master in a management AWS VPC​

    • ​could scale out to environment Syndics

  • ~100 Minions currently

  • Environments: {Base, Dev, Stg, Prd, Inf} 

  • Well defined server naming scheme:

    • ​e.g. dev-use1a-pr-01-app-db-0001

Network Topology

Salt Formula Design

  • Every formula in its own repository

  • Use settings.sls to define good defaults as well as Pillar and Grain lookups

  • Use Pillars and Grains consistently across all formulas:

    • ​Pillars for common infrastructure configuration

    • Grains for node (or cluster) specific configuration

  • Documentation

    • Every repo should have a proper README

    • README should describe how to use the formula

    • README should provide admin info (if applicable)

  • Testing

    • Isolate testing setup into .vagrant-salt/

    • Manage dependencies with vagrant-saltdeps

    • Manage integration tests with serverspec

Salt Formula Demo

Environments, Top.sls & Pillars

  • We employ multiple environments in order to allow us to test (breaking) changes:

    • {Base, Dev, Stg, Prd, Inf}

  • ​Our Base environment only includes a single formula, core, which we apply to all minions.

  • Reduce top.sls file changes (for States and Pillars) by templating it out and backing it with a Minion's Grain data

  • Use a separate repository for Pillar data

{%- raw %}
{% set roles = salt['grains.get']('roles', '') %}
{%- if roles %}
{%- endraw %}
{{ env }}:
  'env:{{ env }}':
{%- raw %}
    - match: grain
    {%- for role in roles %}
    - {{ role }}
    {%- endfor %}
{% endif %}
{%- endraw %}

Example Environment top.sls

Salt-Cloud Design

  • Leverage Salt-Cloud for spinning up/down all servers

  • Maintain a separate repository for Salt-Cloud configuration

  • Use our own Python DSL for simplifying Salt Cloud configurations

  • Use Grains extensively to manage Minions

Salt Cloud Demo

Shortcomings with Salt

  • Highly active project which seems to introduce a lot of bugs

  • Salt documentation can be out of sync with actual implementation

  • Salt-Cloud improvements are desired

  • Salt environments can be problematic for some

Future Goals with Salt

  • Salt-API

  • Reactor

  • Enterprise GUI

Questions?

Made with Slides.com