Vault for Kubernetes

A better workflow

本次演講資源

  • 投影片
  • 逐字講稿
  • 一鍵部署 Vault
  • 30 天手把手教學文
  • 發問群

 



 

Me

Che-Chia Chang

SRE @ Maicoin

Microsoft MVP

 

Kubernetes, Public Cloud
 

@Golang Taipei

@CNCF Taiwan

@iThome Summit

@DevOps Taipei

Resources

  • 30 天手把手學 Terraform
    https://github.com/chechiachang/terraform-30-days
     
  • Some useful code
    https://github.com/chechiachang/vault-playground
     
  • Terraform (How I build this vault)
    https://github.com/chechiachang/terraform-azure

Key Management

API server > Password/Username > Database

Virtual Machine > API Key > Third Party Service

K8s Pod > Private Key > Block Chain Service

 

 

API server > Password/Username > Database

Virtual Machine > API Key > Third Party Service

K8s Pod > Private Key > Block Chain Service

 

 

Key Storage

  • Plain Text
  • Base64 Encoded
  • Encrypted
  • File on Disk / in-memory
  • ...

Secret in Env

# everyone use k8s, right?
apiVersion: v1
kind: Pod
metadata:
  name: my-pod-need-secret
spec:
  containers:
  - name: my-container-need-secret
    command:
    - /bin/do-some-thing
    env:
    - name: MYSQL_HOST
      value: mysql.stag.chechia.com
    - name: MYSQL_PORT
      value: "3306"
    - name: MYSQL_USERNAME
      value: admin
    - name: MYSQL_PASSWORD
      value: my_password_in_plain_text

in K8s Secret(1)

apiVersion: v1
kind: Pod
metadata:
  name: my-pod-need-secret
spec:
  containers:
  - name: my-container-need-secret
    command:
    - /bin/do-some-thing
    env:
    - name: MYSQL_USERNAME
      valueFrom:
        secretKeyRef:
          name: mysql-secret
          key: MYSQL_USERNAME
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-secret
          key: MYSQL_PASSWORD

in K8s Secret(2)

apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
type: Opaque
data:
  username: YWRtaW4=
  password: bXlfcGFzc3dvcmRfaW5fcGxhaW5fdGV4dA==
  # encoded. looks better?

in K8s Secret(3)

echo YWRtaW4= | base64 -d
admin%

echo bXlfcGFzc3dvcmRfaW5fcGxhaW5fdGV4dA== | base64 -d
my_password_in_plain_text%

# encoded is not encrypted !!!
# base64 encoded is basically plain text

what's wrong w/ K8s Secret

  • K8s cluster admin > Key admin
    As a k8s admin, I really don't want see those secrets
     
  • Access control > rbac
     
  • Plain Text
  • If u must use k8s secret, Encrypt them
    Sealed-secrets
    https://www.hwchiu.com/ithome-20202-cicd-25.html

Encrypted K8s Secret(1)

 apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysql-secret
spec:
  encryptedData:
    MYSQL_USERNAME: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
    MYSQL_PASSWORD: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
  • K8s secrets
    • Good for well managed cluster
       
  • Helm secret
    https://github.com/jkroepke/helm-secrets
     
  • Sealed-secrets
    https://www.hwchiu.com/ithome-20202-cicd-25.html
     
  • ...

Encrypted K8s Secret(2)

API server > Password/Username > Database

Virtual Machine > API Key > Third Party Service

K8s Pod > Private Key > Block Chain Service

 

K8s Pod > Some SAFE Magic > get secret

Other Issues

  • Revoke Key
  • Rotate (30 days)
  • Fine grand Access Control
    1 Pod can see, other shouldn't
     
  • 改架構
  • 改權限
     
  • 進去 prod 幫我看一下 Key 內容對不對 :pray:

SRE Goals

  • Minimal key exposure
    • access control
    • minimal time-to-live
  • if exposed
    • Minimal damage
  • Others
    • Easy to manage

Vault Tutorial

有人有用過嗎?

https://www.vaultproject.io/

Vault Core Concepts

 

https://www.youtube.com/watch?v=VYfl-DpZ5wM

  • Auth Method
  • Secrets Engine
  • Policy
  • Storage
  • ...

Vault 101

export VAULT_ADDR=https://vault.chechia.net

# Auth method: Token
VAULT_TOKEN=s.abcdefgh12345678
vault login ${VAULT_TOKEN}

Key                                Value
---                                -----
token                              s.1234567890abcdefghijk
token_duration                     5m
policies                           ["default" "user"]

# Read key from path
vault read /user/mysql

Key                                Value
---                                -----
username                           my-username
password                           my-secret-password

# Read key without permission
vault read /admin/mysql

Permission Denied.

Secret Engines

  • An interface to handle secret
    • Generate, store
    • encrypt, audit
  • Key / Value
  • Versioning
  • Dynamic Secret

Auth Methods

  • Auth user to vault and grant permission
  • Support many auth methods
    • Username, Password, Token, ...
    • AWS, GCP, Azure IAM
    • Kubernetes

AWS IAM Auth Method(1)

API Server > Vault: 我是這個 IAM 請問我可以進去嗎
 

Vault > AWS: 請問他真的是這個 IAM 嗎

AWS > API Server: 你這台 VM 的 IAM 是...

AWS > Vault: 這台 VM 的 IAM 是...
 

Vault > API Server: 請進，你的權限 policy 是...

API Server > Vault: 我要這些 key...

AWS  IAM Auth Method(2)

Policy

# policy for user
path "/service/postgresql" {
  capabilities = ["read", "list"]
}

# policy for admin
path "/service/postgresql" {
  capabilities = ["read", "list", "update"]
}

# auth with different policy

vault token create -policy=k8s-pod
vault token create -policy=aws-ec2-iam
vault token create -policy=sre-admin

Key                                Value
---                                -----
token                              s.1234567890abcdefghijk

Storage

Secrets + Auth

  • Secret engines
    • keep key secured
    • and easy managed
  • Auth methods provide
    • better way to auth user and applications
    • without exchange the important secret

其他功能：Dynamic Secret

Token Auth

# Admin
$ vault token create -policy=user

Key                                Value
---                                -----
token                              s.1234567890abcdefghijk
token_duration                     5m

# Another user
$ vault login s.1234567890abcdefghijk

Key                                Value
policies                           ["default" "user"]

$ vault read /team/user/my-secret

User > Vault Token > Vault > policy

 

API Server > Vault Token in K8s secret > Vault

?

API Server Pod > Some Magic > Vault

Workflow 1: Token Auth

叫 vault 去信任 k8s cluster

 

K8s namespace

K8s service account

 

各自配權限 policy

Magic 1: K8s Auth Method

Magic 1: Kubernetes Auth

  • [Vault] Trust k8s cluster
    • [Admin] Bind Policy with service account
      • namespace/default + service account/chechia
      • Read key from /chechia/mysql
  • [Pod]   Run with k8s service account
  • [Pod]   Use service account jwt
  • [Vault] Validate JWT with K8s API
  • [Vault] Send key

Magic 1: K8s Auth Detail

  • [Sidecar]  Vault login when init (Vault-Injector)
  • [Pod]        Save key (in memory volume mount)
  • [Admin]   Config Vault Policy
  • [Vault]      Create short-live key
  • [Pod]        Use short-live key
  • [Pod]        Delete Pod also delete key

Magic 1: K8s Auth Method

  • Less key exposure: Pod & vault
    • Key in app (memory)
  • If exposed
    • Access Token has limited policy &
      short time-to-live (ex. 1h)
    • Dynamic Key has TTL (ex. 1h)
  • Security Model
    • Won't protect against hacked k8s

Better?

  • API call(s) to vault
  • API call(s) to K8s for each pod init
     
  • Lots of configuration
    • Policy
    • Dynamic Secret
    • Auth methods
    • ...

Cost

Lots of Config

# Policy
$ vault policy read user

path "team/user/*" {
  capabilities = ["read", "sudo"]
}
...

# Auth methods
$ vault auth list
Path                Type          Accessor                    Description
----                ----          --------                    -----------
approle/            approle       auth_approle_12345678       n/a
aws-ec2/            aws           auth_aws_12345678           n/a
aws-iam/            aws           auth_aws_12345678           n/a
dev-chechia-k8s/    kubernetes    auth_kubernetes_12324567    n/a
stag-chechia-k8s/   kubernetes    auth_kubernetes_12324567    n/a

Config Vault

  • Vault
    • https://github.com/chechiachang/terraform-azure/tree/main/modules/vault/config
  • Kubernetes
    •  https://github.com/chechiachang/vault-playground/tree/master/usage/kubernetes 

Other Features

  • Auth Cloud: EC2, IAM, Pod, ...
  • Dynamic secret
  • Transit (encrypt & decrypt)
  • Shot-live Token
  • Terraform vault provider
    • Execute scripts without key
  • Standalone VM
  • IAM Role
  • Network Rule
  • Backend database
    • Policy
    • Rotation mechanism
  • Metrics & Vault exporter
  • Audit Log
  • Share vault infra
    (with other service) could be risky

Management: Vault Infra

  • Configure auth methods
  • Secret engines
  • Policies & permissions
  • K8s SA - Policy bindings
  • Token Lease & revoke
  • Audit & log
  • ...
  • :(

Management: Vault Configs

2021 鐵人賽 30 天帶你玩 Terraform

https://ithelp.ithome.com.tw/users/20120327/ironman/4057

Terraform Introduction
https://slides.com/chechiachang/terraform-introduction

Configure Vault: Terraform

30 天 Terraform workshop

兩鍵安裝+設定 Vault

git clone https://github.com/chechiachang/terraform-30-days.git

cd dev/southeastasia/chechia_net/vault/singleton

terragrunt init && terragrunt apply

# vault operator init

cd dev/southeastasia/chechia_net/vault/config

terragrunt init && terragrunt apply

# and there is a vault
export VAULT_ADDR=http://vault.chechia.net:8200
export VAULT_ADDR=http://52.139.214.189:8200
export VAULT_TOKEN=

vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_81a553a8    per-token private secret storage
identity/     identity     identity_60038fa2     identity store
namespace/    kv           kv_6b29e9e3           This is an example KV Version 2 secret engine mount
sys/          system       system_b332aa7e       system endpoints used for control, policy and debugging

vault auth list

Path               Type          Accessor                    Description
----               ----          --------                    -----------
dev-kubernetes/    kubernetes    auth_kubernetes_9dfe88fb    n/a
token/             token         auth_token_a0708f09         token based credentials

vault policy list

default
namespace
root

兩鍵安裝+設定 Vault(2)

  • Infrastructure as Code + Config as Code
     
  • Key is hard to debug
    • many keys and policies
    • incorrect content
    • invalid permission
       
  • Use terraform to duplicate envs

Why Terraform

More About Vault

Debug Secrets

  • ex. Incorrect secret content​
  • Actually, we don't debug secrets
    • [SRE]      Write secret, don't know secret
    • [Dev]      Write app login, can't read secret
    • [Admin] Know secret, can't access app
  • Debug could be painful
    • In exchange for more security
  • Just generate new key & write again

Debug Config

  • ex. Incorrect policy cause permission deny
     
  • Different vault built by same terraform
    • dev-vault, stag-vault, prod-vault
  • List secret paths & diff
  • Enable auth device
    • ex. log for each read / write

Operation: HA

  • Primary (read-write) & replica
    • Grab lock from backend: master
    • Others standby & forward request
  • Incident Failover (~20-25 sec)
  • Step down & handover (1~5 sec)

Performance

  • API request to Vault server
    • Loading usually not an issue
    • app only access vault when init
  • Scale with Pods
    • potential bottle-neck

Performance

  • Be careful about auto-retry
    • App should have backoff
  • Key local cache
    • Use cache when valid
  • Minimal access to vault

Security

  • Official doc is great
    • Implement best practices
    • Don't invent your own
  • Vault is very secured
    • People is not

Summary

  • We use vault for quite some years
  • It works like a charm
  • Use terraform !

要不要 Vault

  • 資安觀念是否已經夠完善
    • 先改善整體資安意識
    • 完善 vault 好安全
    • 一封釣魚信就炸鍋
  • 有沒有人力維護
  • 我個人是很推

Q&A

歡迎透過 fb 私敲

有經驗的求討論！

 

投影片

講稿

SOP

範例 Github

 

有問題來社群找我
別人會幫我回答XD

CNTUG

https://t.me/cntug

https://fb.cloudnative.tw


DevOpsTW

https://t.me/devopstw

 

 

https://github.com/MaiAmis/Careers

真心覺得自己公司不錯
Maicoin

Made with Slides.com