K8S Bad Practices

Currently we don't have another ones :)

AR, 5.5.2017

Let's dance

Mad Skillz

Cold start

apt-get install szn-sklik-kubectl-conf
source /etc/profile

$ env | grep KUBE
KUBECONFIG=/etc/kube/szn-sklik-kubectl-conf

Wheezy too!

Auto complete

source <(kubectl completion bash)

$ kubectl TAB
annotate        cordon          get             run
api-versions    cp              label           scale
apply           create          logs            set
...

Requirements

1. Docker image somewhere and somehow
2. Access to dashboard.skube.dev
3. Straight arms and patience

Namespaces

DO NOT forget about 'em

Docker "namespaces"

cid.dev/sklik/something

docker.dev/sklik-ci/something

doc.ker/sklik/something

Kubernetes namespaces
kubectl command --namespace=playground

kubectl command --namespace=stable-dev
 

YAMLs everywhere

Kubernetes configuration

"Simple one" example

One container in POD

Prerequisites

Dockerfile

...
EXPOSE 3395
ENTRYPOINT ["goenvtemplator", ..., "-exec"]
CMD [..., "runserver"]

docker build ... -t cid.dev/sklik/something:git-hash
docker push cid.dev/sklik/something:git-hash

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: partnerserver
spec:
  strategy:
    type: RollingUpdate
  replicas: 4
  template:
    metadata:
      annotations:
        cz.sklik.log.format: "dbglog"
        team: "sklik.prostor@firma.seznam.cz"
      labels:
        app: partnerserver
        component: restserver

Additional annotations:

cz.sklik.metrics.scrape: "true"
cz.sklik.metrics.port: "3395"
cz.sklik.metrics.path: "/metrics"

Deployment

  template:
    ...
    spec:
      containers:
      - name: partnerserver
        image: cid.dev/sklik/partnerserver:git-hash
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "1Gi"

Deployment

    spec:
      containers:
      - name: partnerserver
        ...
        livenessProbe:
          httpGet:
            path: "/liveness"
            port: 3395
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: "/readiness"
            port: 3395
          initialDelaySeconds: 30
          timeoutSeconds: 5

Deployment

    spec:
      containers:
      - name: partnerserver
        ...
        env:
        - name: "PS_FOO_BAR"
          value: "oh, my foo!"
        - name: "PS_SUPER_SECRET"
          valueFrom:
            name: partnerserver
            key: secret_from_secret_place

Deployment

kubectl apply -f k8s/partnerserver-deployment.yml --namespace=playground

kubectl delete -f k8s/partnerserver-deployment.yml --namespace=playground

Deployment result

Deployment result

At this point you can only access your pod using CLI:

kubectl logs $(kubectl get pods --namespace=playground -o name -l app=partnerserver) --namespace=playground -f --tail=100

kubectl exec -it $(kubectl get pods --namespace=plagroygroud -o name -l app=partnerserver | cut -d/ -f2) --namespace=playground bash

Service

is what we need

Service

apiVersion: v1
kind: Service
metadata:
  name: partnerserver
  labels:
    app: partnerserver
    component: restserver
spec:
  type: NodePort
  ports:
  - port: 3395
    nodePort: 10395
    protocol: TCP
  selector:
    app: partnerserver
    component: restserver

External ports

Dev k8s:

• 8000-12000
• 80, 443 for ingresses

Production:

• 11000-12000 (I think so)
• no ingress ports

Service commands

kubctl apply --namespace=playground -f k8s/partnerserver-service.yml


kubctl delete --namespace=playground -f k8s/partnerserver-service.yml

Service result

Поиск, cykablyat

Accessing

Outside K8S: http://skube.dev:11395/v1/help

Inside K8S (same namespace): http://partnerserver:3395/v1

Inside K8S (another namespace): http://partnerserver.stable-dev:3395/v1

More than one container in POD

Splitting nginx from uwsgi be like

FE specified problems

nginx should serve static files

uwsgi/apache/python must not

Splitting monolith

1. copy static files to nginx container in Dockerfile
2. produce two deb packages with python and statics
3. use same image, but overwrite CMD/ENTRYPOINT in k8s deployment

Deployment one more time

    spec:
      containers:
      - name: web
        image: cid.dev/sklik/partnerweb2:2.7.3
        imagePullPolicy: IfNotPresent
        args: ["/usr/bin/uwsgi","--plugin","python","-x","/www/sklik/partnerweb2/conf/partnerweb-uwsgi.xml"]
      - name: nginx
        image: cid.dev/sklik/partnerweb2:2.7.3
        args: ["/www/nginx-extras/sbin/nginx","-c","/www/sklik/partnerweb2/conf/partnerweb-nginx.conf"]
        imagePullPolicy: IfNotPresent

Dockerfile

CMD

ENTRYPOINT

K8S YAML

spec.template.spec.containers.N.args

spec.template.spec.containers.N.command

Ingress

or one more "cool" name

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: partnerweb2
spec:
  rules:
  - host: partnerweb2.skube.dev
    http:
      paths:
      - backend:
          serviceName: partnerweb2
          servicePort: 8335
  tls:
  - hosts:
    - partnerweb2.skube.dev
    secretName: wildcard.skube.dev

Ingress commands

kubectl apply

kubectl delete

Ingress result

http://partnerweb2.skube.dev

or https with right certificate

Production doesn't have ingress

GitLab CI vs K8S

Hell circles steps

1. build Docker container for commit
2. make deploy phase in YAML
3. create task for deploy phase
4. configure k8s client in that task
5. enable Kubernetes for your project
6. add deployment environment in GitLab
7. deploy in yours task
8. test it
9. and ...

TEST IT!!!

Actual gitlab-ci testing experience

34 (thirty four) commits
each one longs ~2-10 minutes

But it works

step by step flow

Docker container

build-server-docker:
  image: docker.dev/debian:jessie-stable
  stage: build-docker
  dependencies:
    - build-server-package
  script: |
    apt-get -y install docker-engine dpkg-dev
    VERSION=$(dpkg -I szn-sklik-partnerserver_*.deb | grep Version: | awk '{print $2}' | sed 's/~/-/g')
    DCKR_IMAGE=$(make -n docker-release NAMESPACE=${DOCKER_IMG_NAMESPACE} version=${VERSION} | awk '{print $NF}')
    make docker-build version=${VERSION} NAMESPACE=${DOCKER_IMG_NAMESPACE} DOCKER_BUILD="--build-arg L_BLD_TYPE=automated --build-arg L_BLD_CI_JOB_URL=${CI_PROJECT_URL}"
    make _k8s-docker-push NAMESPACE=${DOCKER_IMG_NAMESPACE}

Deploy stage

stages:
  - build-package
  - build-docker
  - check
  - deploy

Deploy task

deploy-server:
  stage: deploy
  dependencies:
    - build-server-docker

K8S configuration

deploy-server:

script: |
  apt-get -y install szn-sklik-kubectl-conf
  echo "Generating kubeconfig..."
  KUBECONFIG=kubeconfig
  echo "$KUBE_CA_PEM" > kube.ca.pem
  kubectl config set-cluster gitlab-deploy --server="$KUBE_URL" --certificate-authority=kube.ca.pem
  kubectl config set-credentials gitlab-deploy --token="$KUBE_TOKEN" --certificate-authority=kube.ca.pem
  kubectl config set-context gitlab-deploy --cluster=gitlab-deploy --user=gitlab-deploy --namespace="$KUBE_NAMESPACE"
  kubectl config use-context gitlab-deploy
  kubectl config view
  K8S_NAMESPACE=${KUBE_NAMESPACE:-playground}

Enable K8S for repo

Ask DevOps for token and CA bundle

Create environment

deploy-server:
  environment:
    name: staging
    url: http://skube.dev:10395/

.gitlab-ci.yaml

Deploy to K8S

deploy-server:
  script: |
    ...
    make _k8s-deploy NAMESPACE=${DOCKER_IMG_NAMESPACE} DEVK8S=${K8S_NAMESPACE}
  only:
    - master

Small thing that saves your time

Add . (dot) before task name in .gitlab-ci.yaml, to skip this task.

Reference projects

1. gitlab/sklik-backend/partnerserver
   

2. gitlab/sklik-frontend/Partnerweb
   

Text

ar-k8s branch

How to deploy to production

1. Clone gitlab/Sklik-DevOps/kube-configs
2. Copy your YAML files to sklik/component
3. Control that deployment uses doc.ker registry
4. Make MR with your changes
5. Write RT with MR URL
6. ...
7. PROFIT!

FIN