Vault for Kubernetes
A better workflow
本次演講資源
- 投影片
- 逐字講稿
- 一鍵部署 Vault
- 30 天手把手教學文
- 發問群
Me
Che-Chia Chang
SRE @ Maicoin
Microsoft MVP
Kubernetes, Public Cloud
@Golang Taipei
@CNCF Taiwan
@iThome Summit
@DevOps Taipei
Resources
-
30 天手把手學 Terraform
https://github.com/chechiachang/terraform-30-days
- Some useful code
https://github.com/chechiachang/vault-playground
- Terraform (How I build this vault)
https://github.com/chechiachang/terraform-azure
Key Management
API server > Password/Username > Database
Virtual Machine > API Key > Third Party Service
K8s Pod > Private Key > Block Chain Service
API server > Password/Username > Database
Virtual Machine > API Key > Third Party Service
K8s Pod > Private Key > Block Chain Service
Key Storage
- Plain Text
- Base64 Encoded
- Encrypted
- File on Disk / in-memory
- ...
Secret in Env
# everyone use k8s, right?
apiVersion: v1
kind: Pod
metadata:
name: my-pod-need-secret
spec:
containers:
- name: my-container-need-secret
command:
- /bin/do-some-thing
env:
- name: MYSQL_HOST
value: mysql.stag.chechia.com
- name: MYSQL_PORT
value: "3306"
- name: MYSQL_USERNAME
value: admin
- name: MYSQL_PASSWORD
value: my_password_in_plain_text
in K8s Secret(1)
apiVersion: v1
kind: Pod
metadata:
name: my-pod-need-secret
spec:
containers:
- name: my-container-need-secret
command:
- /bin/do-some-thing
env:
- name: MYSQL_USERNAME
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_USERNAME
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_PASSWORD
in K8s Secret(2)
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
type: Opaque
data:
username: YWRtaW4=
password: bXlfcGFzc3dvcmRfaW5fcGxhaW5fdGV4dA==
# encoded. looks better?
in K8s Secret(3)
echo YWRtaW4= | base64 -d
admin%
echo bXlfcGFzc3dvcmRfaW5fcGxhaW5fdGV4dA== | base64 -d
my_password_in_plain_text%
# encoded is not encrypted !!!
# base64 encoded is basically plain text
what's wrong w/ K8s Secret
- K8s cluster admin > Key admin
As a k8s admin, I really don't want see those secrets
- Access control > rbac
- Plain Text
- If u must use k8s secret, Encrypt them
Sealed-secrets
https://www.hwchiu.com/ithome-20202-cicd-25.html
Encrypted K8s Secret(1)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: mysql-secret
spec:
encryptedData:
MYSQL_USERNAME: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
MYSQL_PASSWORD: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
-
K8s secrets
-
Good for well managed cluster
-
Good for well managed cluster
- Helm secret
https://github.com/jkroepke/helm-secrets
- Sealed-secrets
https://www.hwchiu.com/ithome-20202-cicd-25.html
- ...
Encrypted K8s Secret(2)
API server > Password/Username > Database
Virtual Machine > API Key > Third Party Service
K8s Pod > Private Key > Block Chain Service
K8s Pod > Some SAFE Magic > get secret
Other Issues
- Revoke Key
- Rotate (30 days)
- Fine grand Access Control
1 Pod can see, other shouldn't
- 改架構
- 改權限
- 進去 prod 幫我看一下 Key 內容對不對 :pray:
SRE Goals
- Minimal key exposure
- access control
- minimal time-to-live
- if exposed
- Minimal damage
- Others
- Easy to manage
Vault Tutorial
有人有用過嗎?
Vault Core Concepts
https://www.youtube.com/watch?v=VYfl-DpZ5wM
- Auth Method
- Secrets Engine
- Policy
- Storage
- ...
Vault 101
export VAULT_ADDR=https://vault.chechia.net
# Auth method: Token
VAULT_TOKEN=s.abcdefgh12345678
vault login ${VAULT_TOKEN}
Key Value
--- -----
token s.1234567890abcdefghijk
token_duration 5m
policies ["default" "user"]
# Read key from path
vault read /user/mysql
Key Value
--- -----
username my-username
password my-secret-password
# Read key without permission
vault read /admin/mysql
Permission Denied.
Secret Engines
- An interface to handle secret
- Generate, store
- encrypt, audit
- Key / Value
- Versioning
- Dynamic Secret
Auth Methods
- Auth user to vault and grant permission
-
Support many auth methods
- Username, Password, Token, ...
- AWS, GCP, Azure IAM
- Kubernetes
API Server > Vault: 我是這個 IAM 請問我可以進去嗎
Vault > AWS: 請問他真的是這個 IAM 嗎
AWS > API Server: 你這台 VM 的 IAM 是...
AWS > Vault: 這台 VM 的 IAM 是...
Vault > API Server: 請進,你的權限 policy 是...
API Server > Vault: 我要這些 key...
Policy
# policy for user
path "/service/postgresql" {
capabilities = ["read", "list"]
}
# policy for admin
path "/service/postgresql" {
capabilities = ["read", "list", "update"]
}
# auth with different policy
vault token create -policy=k8s-pod
vault token create -policy=aws-ec2-iam
vault token create -policy=sre-admin
Key Value
--- -----
token s.1234567890abcdefghijk
Storage
- Vault is backed by solid Storage
- GCP, AWS, Azure
- SQL DBs
- ex. Store encrypted
auth/policy/key in MySQL table
- ex. Store encrypted
- Cassandra, Elasticsearch, MongoDB
Secrets + Auth
- Secret engines
- keep key secured
- and easy managed
- Auth methods provide
- better way to auth user and applications
- without exchange the important secret
其他功能:Dynamic Secret
Token Auth
# Admin
$ vault token create -policy=user
Key Value
--- -----
token s.1234567890abcdefghijk
token_duration 5m
# Another user
$ vault login s.1234567890abcdefghijk
Key Value
policies ["default" "user"]
$ vault read /team/user/my-secret
User > Vault Token > Vault > policy
API Server > Vault Token in K8s secret > Vault
?
API Server Pod > Some Magic > Vault
Workflow 1: Token Auth
叫 vault 去信任 k8s cluster
K8s namespace
K8s service account
各自配權限 policy
Magic 1: K8s Auth Method
Magic 1: Kubernetes Auth
- [Vault] Trust k8s cluster
- [Admin] Bind Policy with service account
- namespace/default + service account/chechia
- Read key from /chechia/mysql
- [Admin] Bind Policy with service account
- [Pod] Run with k8s service account
- [Pod] Use service account jwt
- [Vault] Validate JWT with K8s API
- [Vault] Send key
Magic 1: K8s Auth Detail
- [Sidecar] Vault login when init (Vault-Injector)
- [Pod] Save key (in memory volume mount)
- [Admin] Config Vault Policy
- [Vault] Create short-live key
- [Pod] Use short-live key
- [Pod] Delete Pod also delete key
Magic 1: K8s Auth Method
- Less key exposure: Pod & vault
- Key in app (memory)
- If exposed
- Access Token has limited policy &
short time-to-live (ex. 1h) - Dynamic Key has TTL (ex. 1h)
- Access Token has limited policy &
- Security Model
- Won't protect against hacked k8s
Better?
- API call(s) to vault
- API call(s) to K8s for each pod init
-
Lots of configuration
- Policy
- Dynamic Secret
- Auth methods
- ...
Cost
Lots of Config
# Policy
$ vault policy read user
path "team/user/*" {
capabilities = ["read", "sudo"]
}
...
# Auth methods
$ vault auth list
Path Type Accessor Description
---- ---- -------- -----------
approle/ approle auth_approle_12345678 n/a
aws-ec2/ aws auth_aws_12345678 n/a
aws-iam/ aws auth_aws_12345678 n/a
dev-chechia-k8s/ kubernetes auth_kubernetes_12324567 n/a
stag-chechia-k8s/ kubernetes auth_kubernetes_12324567 n/a
Config Vault
- Vault
- https://github.com/chechiachang/terraform-azure/tree/main/modules/vault/config
- Kubernetes
- https://github.com/chechiachang/vault-playground/tree/master/usage/kubernetes
Other Features
- Auth Cloud: EC2, IAM, Pod, ...
- Dynamic secret
- Transit (encrypt & decrypt)
- Shot-live Token
- Terraform vault provider
- Execute scripts without key
- Standalone VM
- IAM Role
- Network Rule
- Backend database
- Policy
- Rotation mechanism
- Metrics & Vault exporter
- Audit Log
- Share vault infra
(with other service) could be risky
Management: Vault Infra
- Configure auth methods
- Secret engines
- Policies & permissions
- K8s SA - Policy bindings
- Token Lease & revoke
- Audit & log
- ...
- :(
Management: Vault Configs
2021 鐵人賽 30 天帶你玩 Terraform
https://ithelp.ithome.com.tw/users/20120327/ironman/4057
Configure Vault: Terraform
30 天 Terraform workshop
兩鍵安裝+設定 Vault
git clone https://github.com/chechiachang/terraform-30-days.git
cd dev/southeastasia/chechia_net/vault/singleton
terragrunt init && terragrunt apply
# vault operator init
cd dev/southeastasia/chechia_net/vault/config
terragrunt init && terragrunt apply
# and there is a vault
export VAULT_ADDR=http://vault.chechia.net:8200
export VAULT_ADDR=http://52.139.214.189:8200
export VAULT_TOKEN=
vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_81a553a8 per-token private secret storage
identity/ identity identity_60038fa2 identity store
namespace/ kv kv_6b29e9e3 This is an example KV Version 2 secret engine mount
sys/ system system_b332aa7e system endpoints used for control, policy and debugging
vault auth list
Path Type Accessor Description
---- ---- -------- -----------
dev-kubernetes/ kubernetes auth_kubernetes_9dfe88fb n/a
token/ token auth_token_a0708f09 token based credentials
vault policy list
default
namespace
root
兩鍵安裝+設定 Vault(2)
- Infrastructure as Code + Config as Code
- Key is hard to debug
- many keys and policies
- incorrect content
- invalid permission
- Use terraform to duplicate envs
Why Terraform
More About Vault
Debug Secrets
- ex. Incorrect secret content
- Actually, we don't debug secrets
- [SRE] Write secret, don't know secret
- [Dev] Write app login, can't read secret
- [Admin] Know secret, can't access app
- Debug could be painful
- In exchange for more security
- Just generate new key & write again
Debug Config
- ex. Incorrect policy cause permission deny
- Different vault built by same terraform
- dev-vault, stag-vault, prod-vault
- List secret paths & diff
- Enable auth device
- ex. log for each read / write
Operation: HA
- Primary (read-write) & replica
- Grab lock from backend: master
- Others standby & forward request
- Incident Failover (~20-25 sec)
- Step down & handover (1~5 sec)
Performance
- API request to Vault server
- Loading usually not an issue
- app only access vault when init
- Scale with Pods
- potential bottle-neck
Performance
- Be careful about auto-retry
- App should have backoff
- Key local cache
- Use cache when valid
- Minimal access to vault
Security
- Official doc is great
- Implement best practices
- Don't invent your own
- Vault is very secured
- People is not
Summary
- We use vault for quite some years
- It works like a charm
- Use terraform !
要不要 Vault
- 資安觀念是否已經夠完善
- 先改善整體資安意識
- 完善 vault 好安全
- 一封釣魚信就炸鍋
- 有沒有人力維護
- 我個人是很推
Q&A
歡迎透過 fb 私敲
有經驗的求討論!
投影片
講稿
SOP
範例 Github
有問題來社群找我
別人會幫我回答XD
CNTUG
https://t.me/cntug
https://fb.cloudnative.tw
DevOpsTW
https://t.me/devopstw
真心覺得自己公司不錯
Maicoin
Hashicorp Vault for Kubernetes
By chechiachang
Hashicorp Vault for Kubernetes
- 1,586