Secrets 2.0 - The Vault

Why

Today, I'm going to show you...

 

Who am I

Infrastructure Team @ Under Armour Connected Fitness

Develop & suppport internal PaaS systems for all of you

Kyle Rockman

Agenda

Principles & Problems

Overview of the Solution

Live Demo

Recap / Conclusion / Questions

Principles & Problems

The Problem...

Principles & Problems

Single source of secrets storage

ACL's

Ability to dynamically provision secrets for things

UI & Programmatic access

Audit-ability

Latin: Volvere

to roll;

a large room or chamber used for storage, especially and underground one.

        Vault - Concepts

  • Storage (how the secrets are stored)
    • Google Cloud Storage for high availability
  • Auth Backends (how you get authorization)
    • UACF LDAP (username + password)
    • Kubernetes (service accounts)
    • Auth0/SAML (future?)
    • Lots more - https://www.vaultproject.io/docs/auth/index.html

        Vault - Concepts

  • Secret Engines (types of secrets you can get out of vault)
    • Key/Value (static)
    • AWS (dynamic)
    • Databases (dynamic)
      • MYSQL
      • Postgres
      • Mongo
    • RabbitMQ (dynamic)
    • Lots more - https://www.vaultproject.io/docs/secrets/index.html

      Vault - Concepts

- Everything in vault is a PATH

- Key/Value data exists at a PATH

- Policies apply capabilities (permissions) to a PATH

path "secret/fs/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

path "database/fs/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

...

vault write sys/policy/fs policy=@./policy.hcl

      Vault - Concepts

        Vault - Auth Workflows

LDAP (Username + Password)

Kubernetes (Service Account)

        Vault - Auth Workflows

        Vault - Auth Workflow (LDAP)

  1. Install vault cmdline - `brew install vault`
  2. Login - `vault login -method ldap username=krockman`
  3. Enter AWS VPN LDAP creds
  4. Vault token provisioned and stored on local machine (with TTL)
    1. Vault permissions assigned by policies associated with LDAP groups
  5. Happy Vaulting

        Vault - Auth Workflow (K8S)

        Vault - Auth Workflow (K8S)

  1. Configure K8S yaml
  2. Launch on K8S
  3. Startup script will trade K8S service account token for vault token
  4. Envconsul will use vault token to get secrets out of vault
  5. Envconsul runs downstream process handing vault secrets as environment variables - will not be exposed to container

        Vault - Auth Workflow (K8S)

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: challenges
  namespace: run-prod
spec:
  template:
    spec:
      ...
      containers:
        ...
        command:
          - "/vault/bin/shim.sh"
          - "/uacf/src/python/httpserver/httpserver.sh"
        volumeMounts:
          - name: secrets-config
            mountPath: /vault/secrets
          - name: vault-tools
            mountPath: /vault/bin
      volumes:
        - name: secrets-config
          configMap:
            name: challenges
        - name: vault-tools
          configMap:
            name: vault-shim
            defaultMode: 0755
apiVersion: v1
kind: ConfigMap
metadata:
  name: challenges
  namespace: run-prod
data:
  config.hcl: |
    secret {
        no_prefix = true
        path = "secret/infra/service/prod"
    }
    secret {
      no_prefix = true
      path = "aws/run/creds/route-53"
      format = "AWS_{{ key }}"
    }

        Vault - Auth Workflow (K8S)

function generate_post_data()
{
    cat <<EOF
{"jwt": "$KUBE_TOKEN", "role": "$KUBE_SERVICEACCOUNT_NAME"}
EOF
}

export VAULT_ADDR=https://vault.uacf.io:443
export KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)
export KUBE_SERVICEACCOUNT_NAME=$(decode_jwt 2 $KUBE_TOKEN | jq -r '."kubernetes.io/serviceaccount/service-account.name"')
export VAULT_RESPONSE=$(curl -s -X POST -d "$(generate_post_data)" ${VAULT_ADDR}/v1/auth/k8s/us/green/login)
export VAULT_TOKEN=$(echo $VAULT_RESPONSE | jq -r '.auth.client_token')


envconsul -config="/vault/secrets/config.hcl" -sanitize -upcase $@

Heres the Magic!!!

        Vault - Secrets (Static)

Key = Value - V1

Key = Value - V2 (versioned)

        Vault - Secrets (Static)

vault write secret/infra/my_service/prod/hello value=world foo=bar

vault read secret/infra/my_service/prod/hello
Key                 Value
---                 -----
refresh_interval    168h
value               world
foo                 bar

...

vault write secret/infra/my_service/prod/hello hello=world foo=bar

vault read secret/infra/my_service/prod/hello
Key                 Value
---                 -----
refresh_interval    168h
hello               world
foo                 bar

        Vault - Secrets (Dynamic)

AWS IAM/STS

Google Cloud IAM

Database User/Pass (mysql, postgres, mongo, etc)

RabbitMQ User/Pass

        Vault - Secrets (Dynamic)

  1. Configure vault (whole new engine or role for existing engine)
  2. Launch app without secret
  3. Get secret from vault (each request is a newly provisioned secret)
  4. Vault logs request to auditlog and starts a lease TTL on secret
  5. Profit using a secure dynamic secret!!!
  6. If app stops before TTL
    • Vault will delete secret from dynamic source (IE: delete IAM user)
  7. If app lives past TTL
    • Need to renew lease before TTL expires or secret will be deleted

        Vault - Secrets (Dynamic)

        Vault - Secrets (Dynamic)

We want vault configuration to be as turnkey as possible

 

So we created an automated process for it

 

https://code.uacf.io/projects/INFRA/repos/vault-configuration/browse

 

Merge to Master == Vault Configured

Thanks!

Thank you to everyone who beta tested this out!!!!

User Stories

Use Case: Privacy and Consent Services

  • Dynamic AWS credentials
  • Lightstep integration
  • Documentation

User Stories

Use Case: Training Plans Service

The Good:

  • Static secrets are easily stored.
    • Can be read by engineers for local testing too!
  • Dynamically provisioned secrets are amazing!
    • Each k8s pod will acquire a unique lease for a dynamically created user which is automatically cleaned up at the end of a lease.
  • When `exec`ing into a pod, you can re-run the shim manually
    • Any dynamically provisioned secrets receive __a separate, new lease__

The Bad/Ugly:

  • Two sources of truth (compared to escrow) for config and secrets
  • Adds at least two volume mounts to your container

Recap

  • 1 Vault to rule them all
    • Organization of PATH's in vault is IMPORTANT - but left up to you
  • Vault access policies are how we protect the secrets - practice safety!!!
  • Shorter lifecycles (leases) are always better - renewing is easy
  • Envconsul is your friend (very powerful)
    •  https://github.com/hashicorp/envconsul
  • UACF's vault configuration is Automated
    • https://code.uacf.io/projects/INFRA/repos/vault-configuration/browse
    • Jenkinsfile applies configuration on merge to master

Demo

https://slides.com/rocktavious/vault/

Thanks!

Secrets 2.0 - Vault

By Kyle Rockman

Made with Slides.com

Secrets 2.0 - Vault

Using Vault at Under Armour Connected Fitness.

  • 842

More from Kyle Rockman