K8S Bad Practices

Currently we don't have another ones :)

AR, 5.5.2017

Let's dance

Mad Skillz

Cold start

apt-get install szn-sklik-kubectl-conf
source /etc/profile

$ env | grep KUBE
KUBECONFIG=/etc/kube/szn-sklik-kubectl-conf

Wheezy too!

Auto complete

source <(kubectl completion bash)

$ kubectl TAB
annotate        cordon          get             run
api-versions    cp              label           scale
apply           create          logs            set
...

Requirements

Docker image somewhere and somehow
Access to dashboard.skube.dev
Straight arms and patience

Namespaces

DO NOT forget about 'em

Docker "namespaces"

cid.dev/sklik/something

docker.dev/sklik-ci/something

doc.ker/sklik/something

Kubernetes namespaces
kubectl command --namespace=playground

kubectl command --namespace=stable-dev

YAMLs everywhere

Kubernetes configuration

Deployment
Service
Ingress

"Simple one" example

One container in POD

Prerequisites

Dockerfile

...
EXPOSE 3395
ENTRYPOINT ["goenvtemplator", ..., "-exec"]
CMD [..., "runserver"]

docker build ... -t cid.dev/sklik/something:git-hash
docker push cid.dev/sklik/something:git-hash

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: partnerserver
spec:
  strategy:
    type: RollingUpdate
  replicas: 4
  template:
    metadata:
      annotations:
        cz.sklik.log.format: "dbglog"
        team: "sklik.prostor@firma.seznam.cz"
      labels:
        app: partnerserver
        component: restserver

Additional annotations:

cz.sklik.metrics.scrape: "true"
cz.sklik.metrics.port: "3395"
cz.sklik.metrics.path: "/metrics"

Deployment

  template:
    ...
    spec:
      containers:
      - name: partnerserver
        image: cid.dev/sklik/partnerserver:git-hash
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "1Gi"

Deployment

    spec:
      containers:
      - name: partnerserver
        ...
        livenessProbe:
          httpGet:
            path: "/liveness"
            port: 3395
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: "/readiness"
            port: 3395
          initialDelaySeconds: 30
          timeoutSeconds: 5

Deployment

    spec:
      containers:
      - name: partnerserver
        ...
        env:
        - name: "PS_FOO_BAR"
          value: "oh, my foo!"
        - name: "PS_SUPER_SECRET"
          valueFrom:
            name: partnerserver
            key: secret_from_secret_place

Deployment

kubectl apply -f k8s/partnerserver-deployment.yml --namespace=playground

kubectl delete -f k8s/partnerserver-deployment.yml --namespace=playground

Deployment result

At this point you can only access your pod using CLI:

kubectl logs $(kubectl get pods --namespace=playground -o name -l app=partnerserver) --namespace=playground -f --tail=100

kubectl exec -it $(kubectl get pods --namespace=plagroygroud -o name -l app=partnerserver | cut -d/ -f2) --namespace=playground bash

Service

is what we need

Service

apiVersion: v1
kind: Service
metadata:
  name: partnerserver
  labels:
    app: partnerserver
    component: restserver
spec:
  type: NodePort
  ports:
  - port: 3395
    nodePort: 10395
    protocol: TCP
  selector:
    app: partnerserver
    component: restserver

External ports

Dev k8s:

8000-12000
80, 443 for ingresses

Production:

11000-12000 (I think so)
no ingress ports

Service commands

kubctl apply --namespace=playground -f k8s/partnerserver-service.yml


kubctl delete --namespace=playground -f k8s/partnerserver-service.yml

Service result

Поиск, cykablyat

Accessing

Outside K8S: http://skube.dev:11395/v1/help

Inside K8S (same namespace): http://partnerserver:3395/v1

Inside K8S (another namespace): http://partnerserver.stable-dev:3395/v1

More than one container in POD

Splitting nginx from uwsgi be like

FE specified problems

nginx should serve static files

uwsgi/apache/python must not

Splitting monolith

copy static files to nginx container in Dockerfile
produce two deb packages with python and statics
use same image, but overwrite CMD/ENTRYPOINT in k8s deployment

Deployment one more time

    spec:
      containers:
      - name: web
        image: cid.dev/sklik/partnerweb2:2.7.3
        imagePullPolicy: IfNotPresent
        args: ["/usr/bin/uwsgi","--plugin","python","-x","/www/sklik/partnerweb2/conf/partnerweb-uwsgi.xml"]
      - name: nginx
        image: cid.dev/sklik/partnerweb2:2.7.3
        args: ["/www/nginx-extras/sbin/nginx","-c","/www/sklik/partnerweb2/conf/partnerweb-nginx.conf"]
        imagePullPolicy: IfNotPresent

Dockerfile

CMD

ENTRYPOINT

K8S YAML

spec.template.spec.containers.N.args

spec.template.spec.containers.N.command

Ingress

or one more "cool" name

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: partnerweb2
spec:
  rules:
  - host: partnerweb2.skube.dev
    http:
      paths:
      - backend:
          serviceName: partnerweb2
          servicePort: 8335
  tls:
  - hosts:
    - partnerweb2.skube.dev
    secretName: wildcard.skube.dev

Ingress commands

kubectl apply

kubectl delete

Ingress result

http://partnerweb2.skube.dev

or https with right certificate

Production doesn't have ingress

GitLab CI vs K8S

Hell circles steps

build Docker container for commit
make deploy phase in YAML
create task for deploy phase
configure k8s client in that task
enable Kubernetes for your project
add deployment environment in GitLab
deploy in yours task
test it
and ...

TEST IT!!!

Actual gitlab-ci testing experience

34 (thirty four) commits
each one longs ~2-10 minutes

But it works

step by step flow

Docker container

build-server-docker:
  image: docker.dev/debian:jessie-stable
  stage: build-docker
  dependencies:
    - build-server-package
  script: |
    apt-get -y install docker-engine dpkg-dev
    VERSION=$(dpkg -I szn-sklik-partnerserver_*.deb | grep Version: | awk '{print $2}' | sed 's/~/-/g')
    DCKR_IMAGE=$(make -n docker-release NAMESPACE=${DOCKER_IMG_NAMESPACE} version=${VERSION} | awk '{print $NF}')
    make docker-build version=${VERSION} NAMESPACE=${DOCKER_IMG_NAMESPACE} DOCKER_BUILD="--build-arg L_BLD_TYPE=automated --build-arg L_BLD_CI_JOB_URL=${CI_PROJECT_URL}"
    make _k8s-docker-push NAMESPACE=${DOCKER_IMG_NAMESPACE}

Deploy stage

stages:
  - build-package
  - build-docker
  - check
  - deploy

Deploy task

deploy-server:
  stage: deploy
  dependencies:
    - build-server-docker

K8S configuration

deploy-server:

script: |
  apt-get -y install szn-sklik-kubectl-conf
  echo "Generating kubeconfig..."
  KUBECONFIG=kubeconfig
  echo "$KUBE_CA_PEM" > kube.ca.pem
  kubectl config set-cluster gitlab-deploy --server="$KUBE_URL" --certificate-authority=kube.ca.pem
  kubectl config set-credentials gitlab-deploy --token="$KUBE_TOKEN" --certificate-authority=kube.ca.pem
  kubectl config set-context gitlab-deploy --cluster=gitlab-deploy --user=gitlab-deploy --namespace="$KUBE_NAMESPACE"
  kubectl config use-context gitlab-deploy
  kubectl config view
  K8S_NAMESPACE=${KUBE_NAMESPACE:-playground}

Enable K8S for repo

Ask DevOps for token and CA bundle

Create environment

deploy-server:
  environment:
    name: staging
    url: http://skube.dev:10395/

.gitlab-ci.yaml

Deploy to K8S

deploy-server:
  script: |
    ...
    make _k8s-deploy NAMESPACE=${DOCKER_IMG_NAMESPACE} DEVK8S=${K8S_NAMESPACE}
  only:
    - master

Small thing that saves your time

Add . (dot) before task name in .gitlab-ci.yaml, to skip this task.

Reference projects

Text

ar-k8s branch

How to deploy to production

Clone gitlab/Sklik-DevOps/kube-configs
Copy your YAML files to sklik/component
Control that deployment uses doc.ker registry
Make MR with your changes
Write RT with MR URL
...
PROFIT!

K8S Bad Practices

Let's dance

Cold start

Auto complete

Requirements

Namespaces

DO NOT forget about 'em

YAMLs everywhere

Kubernetes configuration

"Simple one" example

One container in POD

Prerequisites

Dockerfile

Deployment

Deployment

Deployment

Deployment

Deployment

Deployment result

Deployment result

Service

is what we need

Service

External ports

Service commands

Service result

Accessing

More than one container in POD

FE specified problems

Splitting monolith

Deployment one more time

Dockerfile

K8S YAML

Ingress

or one more "cool" name

Ingress

Ingress commands

Ingress result

GitLab CI vs K8S

Hell circles steps

TEST IT!!!

Actual gitlab-ci testing experience

34 (thirty four) commits each one longs ~2-10 minutes

But it works

Docker container

Deploy stage

Deploy task

K8S configuration

Enable K8S for repo

Create environment

Deploy to K8S

Small thing that saves your time

Reference projects

How to deploy to production

FIN

34 (thirty four) commits
each one longs ~2-10 minutes