Опыт production-эксплуатации Kubernetes
Олег Овсиенко
Backend Team Lead, DevOps специалист
Astral Space
Astral Space
Зам. руководителя отдела программирования АО "Калуга Астрал"
Backend Team Lead
DevOps специалист
telegram: @oleg_ovsienko
email: ovsienko@astral.ru
О себе
Astral Space
-
Почему Kubernetes?
-
Модули Kubernetes.
-
Компоненты Kubernetes.
-
Архитектура проекта АО5.
-
Мониторинг кластера.
-
Проблемы эксплуатации.
-
С чего начать?
План
Astral Space
Введение
Kubernetes (K8s)
Astral Space
История Kubernetes
Astral Space
Kubernetes сегодя
Astral Space
Компании использующие Kubernetes
Astral Space
Особенности Kubernetes
- Run Anywhere
- Service discovery and load balancing
- Automated rollouts and rollbacks
- Automatic bin packing
- Self-healing
- Secret and configuration management
- Horizontal scaling
- Windows nodes support (since k8s 1.15)
Astral Space
Архитектура Kubernetes
master
node 1
node 2
...
Docker
kubelet
kube-proxu
Docker
kubelet
kube-proxy
etcd
API
Server
Controller
-manager
Scheduler
kubectl
YAML (Declarative DSL)
Astral Space
Примитивы Kubernetes: Pod
apiVersion: v1
kind: Pod
metadata:
name: abonents-app
labels:
tier: backend
spec:
containers:
- name: abonents-app
image: dockerhub/abonents.app:24018
ports:
- containerPort: 80
Astral Space
Примитивы Kubernetes: ReplicaSet
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: abonents-app
labels:
app: web-report
tier: backend
spec:
replicas: 3
selector:
matchLabels:
tier: backend
template:
metadata:
labels:
tier: backend
spec:
containers:
- name: abonents-app
image: dockerhub/abonents.app:24018
Astral Space
Примитивы Kubernetes: Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: abonents-app
labels:
app: web-report
tier: backend
spec:
replicas: 3
selector:
matchLabels:
tier: backend
template:
metadata:
labels:
tier: backend
spec:
containers:
- name: abonents-app
image: dockerhub/abonents.app:24018
Astral Space
Примитивы Kubernetes: Service
apiVersion: v1
kind: Service
metadata:
name: abonents-app
spec:
selector:
tier: back
v: 1.2
ports:
- protocol: TCP
port: 80
targetPort: 8080
Astral Space
Примитивы Kubernetes: Job и CronJob
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes
restartPolicy: OnFailure
Job = Pod +
Successful Completion
Astral Space
Примитивы Kubernetes: Volume
Types of Volumes
- awsElasticBlockStore
- azureDisk
- azureFile
- cephfs
- cinder
- configMap
- csi
- downwardAPI
- emptyDir
- fc (fibre channel)
- flexVolume
- flocker
- gcePersistentDisk
- gitRepo (deprecated)
- glusterfs
- hostPath
- iscsi
- local
- nfs
- persistentVolumeClaim
- projected
- portworxVolume
- quobyte
- rbd
- scaleIO
- secret
- storageos
- vsphereVolume
Astral Space
Примитивы Kubernetes: Secrets
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4= #base64 'admin'
password: MWYyZDFlMmU2N2Rm #base64 '1f2d1e2e67df'apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
restartPolicy: Never
Astral Space
Примитивы Kubernetes: StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
labels:
app: nginx
spec:
serviceName: "nginx"
selector:
matchLabels:
app: nginx
replicas: 14
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/htmlAstral Space
Примитивы Kubernetes: DaemonSet
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: nginx
name: nginx-daemon-set
spec:
template:
labels:
name: nginx-daemon-set
scheduler: poseidon
spec:
containers:
containers:
- name: nginx
image: "nginx:1.11.1-alpine"
ports:
- containerPort: 80
Astral Space
Примитивы Kubernetes: Ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: back-ingress
spec:
rules:
- host: my.app.com
- http:
paths:
- path: /backend
backend:
serviceName: back
servicePort: 80
Astral Space
Примитивы Kubernetes: CRD
Серверная архитектура АО5
Astral Space
Kubernetes dashboard
Astral Space
Мониторинг Kubernetes
Kubernetes
nodes
Prometheus
Graphana
Telegram
Мониторинг Kubernetes (nodes)
Мониторинг Kubernetes (pods)
Проблемы эксплуатации Kubernetes
1. Интеграция с системами CI/CD.
GitLab + K8s; Azure DevOps (TFS) + K8s
2. Смена IP адресов для nodes кластера k8s.
192.168.1.* -> 10.0.2.*
3. Использование тега latest для обозначения версий контейнеров.
image: postgres:latest
4. Чувствительность к версиям ядра Linux.
4.15.0-54-generic -> 5.0.2-050002-generic
5. Балансировка подов по worker nodes кластера.
6. Внезапное отключение nodes кластера.
7. Kubernetes certificates
Astral Space
Балансировка подов по worker nodes кластера
resources:
limits:
memory: 512Mi
requests:
memory: 256MiAstral Space
Внезапное зависание nodes кластера
Внезапное отключение nodes кластера
KUBELET_EXTRA_ARGS=" --event-qps=30 --event-burst=40 --kube-api-qps=30 --kube-api-burst=40 --registry-qps=20 --registry-burst=30 --system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi "
Astral Space
Certificate Rotation
Astral Space
С чего начать?
+
Kubernetes Interactive Tutorial
Astral Space
Astral Space
Полезные ссылки
- https://kubernetes.io
- https://kubernetes.io/docs/tutorials/kubernetes-basics
- https://github.com/kubernetes/kubernetes
- https://www.cncf.io
- https://habr.com/ru/post/258443/
- https://www.youtube.com/watch?v=XZQ7-7vej6w
- https://www.youtube.com/watch?v=aSrqRSk43lY
- https://www.youtube.com/playlist?list=PL7bmigfV0EqQw4WnD0wF-SRBYttCFeBbF
- https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/videos
Вопросы
Олег Овсиенко
Backend Team Lead, DevOps специалист
Astral Space
telegram: @oleg_ovsienko
email: ovsienko@astral.ru