Workflow on K8s

Argo

What's KintoHub ?

Connect Repo

Build

Combine

Deploy

Call

KintoHub Flow

Integration
/
Build

The Problems

Using Jenkins

Learning Curve

Scalability

Too many plugins

Slowness

Kubernetes fit

Jenkins Slave - Many languages

Docker in Docker

Mount docker socket

Install language packages

- https://github.com/argoproj/argo -> Open Source

- Kubernetes Custom Resource Definition
- Cloud agnostic

- Designed for container
- Graph or Step definition of workflows

- Workflow is a YAML file

- Easy to deploy / Easy to use

- Artifact / Timeout / TTL / Retry / Exit Handler / ...

- Get logs by workflow/step

- Labels / Volumes / Env vars / Secrets / Scheduling / ...

Argo

Spec body

- Entrypoint invocation

- List of template definitions

- Name of the template

- Optionally a list of inputs /outputs

- List of steps

- Container/Pod invocation

Argo Workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: coinflip-recursive-
spec:
  entrypoint: coinflip
  templates:
  - name: coinflip
    steps:
    - - name: flip-coin
        template: flip-coin
    - - name: heads
        template: heads
        when: "{{steps.flip-coin.outputs.result}} == heads"
      - name: tails
        template: coinflip
        when: "{{steps.flip-coin.outputs.result}} == tails"

  - name: flip-coin
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import random
        result = "heads" if random.randint(0,1) == 0 else "tails"
        print(result)
  - name: heads
    container:
      image: alpine:3.6
      command: [sh, -c]
      args: ["echo \"it was heads\""]

UseCase

Build NodeJS Application

    - `git clone`
    - `npm install`
    - `npm build`
    - `npm run test`
    - `push [artifact]`

pipeline {
  agent any
  stages {
    stage('Cloning Git') {
      steps { sh 'git clone ...' }
    }
    stage('Install dependencies') {
      steps { sh 'npm install' }
    }
    stage('Testing') {
      steps { sh 'npm run test' }
    }
    stage('Push') {
      steps { sh '[push artifacts]' }
    }
  }
}

Jenkins

UseCase

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: k8s-meetup-
spec:
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 300
  entrypoint: main
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  imagePullSecrets:
  - name: regsecret
  templates:
  - name: main
    steps:
    - - name: git-init
        template: git-init
    - - name: build-app
        template: build-app
    - - name: upload-to-minio
        template: upload-to-minio
  - name: git-init
    container:
      image: alpine/git:latest
      command:
      - "sh"
      - "-c"
      args:
      - "rm -rf * && git clone https://${GIT_REPO} . && git checkout ${GIT_COMMIT} || exit 1"
      workingDir: /build
      env:
      - name: "GIT_COMMIT"
        value: "b8167974a9efb7a2d0e1e4b87d7cfcdf009eff2e"
      - name: "GIT_REPO"
        value: "github.com/kintohub/node-examples.git"
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: git-init
  - name: build-app
    container:
      image: kintocloud.azurecr.io/node894-alpine:latest
      command:
      - "sh"
      - "-c"
      args:
      - "npm install || exit 1"
      workingDir: /build
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: build-app
  - name: upload-to-minio
    container:
      image: minio/mc:RELEASE.2018-06-22T23-32-12Z
      command:
      - "sh"
      - "-c"
      args:
      - "mc config host add k8s http://${MINIO_HOST} ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY} S3v4 && mc mb --ignore-existing k8s/artifact && mc cp -r * k8s/artifact || exit 1"
      workingDir: /build
      env:
      - name: "MINIO_ACCESS_KEY"
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: accessKey
      - name: "MINIO_SECRET_KEY"
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: secretKey
      - name: "MINIO_HOST"
        value: "minio-dev:9000"
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: upload-to-minio

Argo

Result

Name:                k8s-meetup-cvgvh
Namespace:           dev
ServiceAccount:      default
Status:              Succeeded
Created:             Tue Nov 06 20:50:01 +0800 (41 seconds ago)
Started:             Tue Nov 06 20:50:01 +0800 (41 seconds ago)
Finished:            Tue Nov 06 20:50:42 +0800 (now)
Duration:            41 seconds

STEP                    PODNAME                      DURATION  MESSAGE
 ✔ k8s-meetup-cvgvh
 ├---✔ git-init         k8s-meetup-cvgvh-678981496   27s
 ├---✔ build-app        k8s-meetup-cvgvh-2911520064  7s
 └---✔ upload-to-minio  k8s-meetup-cvgvh-3551202965  4s

Infra
Automation

The Problems

How to deploy easily our K8s resources?

- Terraform

- No support for beta resources.
- No Deployment, StatefulSet, DeamonSet, ...
- Helm

- Tiller

- Both ?
- Spinnaker
- ...

KintoHub Stack

Gateway Middleware

Cache Processor

Loader

...

Infra Deployment

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: create-infra-
spec:
  ttlSecondsAfterFinished: 600
  activeDeadlineSeconds: 600
  entrypoint: main
  serviceAccountName: init-infra
  imagePullSecrets:
  - name: regsecret
  templates:
  - name: main
    steps:
    - - name: infra-preparation
        template: infra-preparation
    - - name: infra-install-minio
        template: infra-install-minio
      - name: infra-install-redis
        template: infra-install-redis
      - name: infra-install-rabbitmq
        template: infra-install-rabbitmq
      - name: infra-install-mongodb-replicaset
        template: infra-install-mongodb-replicaset
      - name: infra-install-nginx
        template: infra-install-nginx
      - name: infra-install-external-dns
        template: infra-install-external-dns
      - name: infra-install-openresty
        template: infra-install-openresty
      - name: infra-install-elasticsearch
        template: infra-install-elasticsearch
        when: "false"
      - name: infra-install-elasticsearch-curator
        template: infra-install-elasticsearch-curator
        when: "false"
      - name: infra-install-fluentd
        template: infra-install-fluentd
        when: "false"
      - name: infra-install-kibana
        template: infra-install-kibana
        when: "false"
      - name: infra-install-prometheus-operator
        template: infra-install-prometheus-operator
        when: "false"
    - - name: infra-install-cache-processor
        template: infra-install-cache-processor
      - name: infra-install-gateway-middleware
        template: infra-install-gateway-middleware
      - name: infra-install-loader
        template: infra-install-loader
      - name: infra-install-jenkins
        template: infra-install-jenkins
      - name: infra-install-argo-rbmq-build
        template: infra-install-argo-rbmq-build
      - name: infra-install-argo-rbmq-consumer
        template: infra-install-argo-rbmq-consumer
      - name: infra-install-argo-workflow-generator
        template: infra-install-argo-workflow-generator
      - name: infra-install-services-creation
        template: infra-install-services-creation
        when: "false"
      - name: infra-install-services-manager
        template: infra-install-services-manager
        when: "false"
      - name: infra-install-shutdown-consumer
        template: infra-install-shutdown-consumer
        when: "false"
  - name: infra-preparation
    outputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "argocd login argocd-server.argocd.svc.cluster.local --config /argocd/.argocd --username admin --password $ARGOCD_PWD --insecure || exit 1"
      env:
        - name: ARGOCD_PWD
          value: admin123
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-preparation
  - name: infra-install-minio
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=minio-$ENV && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace $ENV --path helm/charts/minio --values values-$ENV.yaml --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
        - name: "GIT_TOKEN"
          valueFrom:
            secretKeyRef:
              name: argo-git-creds
              key: token
        - name: "GIT_BRANCH"
          value: "dev"
        - name: "ENV"
          value: "dev"
        - name: "TIMEOUT"
          value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-minio
  - name: infra-install-redis
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=redis-ha-$ENV && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace $ENV --path helm/charts/redis-ha --values values-$ENV.yaml --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
        - name: "GIT_TOKEN"
          valueFrom:
            secretKeyRef:
              name: argo-git-creds
              key: token
        - name: "GIT_BRANCH"
          value: "dev"
        - name: "ENV"
          value: "dev"
        - name: "TIMEOUT"
          value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-redis
  - name: infra-install-prometheus-operator
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=prometheus-operator && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace monitoring --path helm/charts/prometheus-operator --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
      - name: "GIT_TOKEN"
        valueFrom:
          secretKeyRef:
            name: argo-git-creds
            key: token
      - name: "GIT_BRANCH"
        value: "dev"
      - name: "TIMEOUT"
        value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-prometheus-operator
...

Argo + ArgoCD

Result

Name:                create-infra-l89gx
Namespace:           dev
ServiceAccount:      init-infra
Status:              Succeeded
Created:             Tue Nov 06 20:21:39 +0800 (1 minute ago)
Started:             Tue Nov 06 20:21:39 +0800 (1 minute ago)
Finished:            Tue Nov 06 20:22:32 +0800 (15 seconds ago)
Duration:            53 seconds

STEP                                          PODNAME                        DURATION  MESSAGE
 ✔ create-infra-l89gx
 ├---✔ infra-preparation                      create-infra-l89gx-1326669295  3s
 ├-·-○ infra-install-elasticsearch                                                     when 'false' evaluated false
 | ├-○ infra-install-elasticsearch-curator                                             when 'false' evaluated false
 | ├-✔ infra-install-external-dns             create-infra-l89gx-3052507158  9s
 | ├-○ infra-install-fluentd                                                           when 'false' evaluated false
 | ├-○ infra-install-kibana                                                            when 'false' evaluated false
 | ├-✔ infra-install-minio                    create-infra-l89gx-2877376357  19s
 | ├-✔ infra-install-mongodb-replicaset       create-infra-l89gx-682468434   14s
 | ├-✔ infra-install-nginx                    create-infra-l89gx-2368329705  11s
 | ├-✔ infra-install-openresty                create-infra-l89gx-465791308   17s
 | ├-○ infra-install-prometheus-operator                                               when 'false' evaluated false
 | ├-✔ infra-install-rabbitmq                 create-infra-l89gx-4269141425  23s
 | └-✔ infra-install-redis                    create-infra-l89gx-42796882    6s
 └-·-✔ infra-install-argo-rbmq-build          create-infra-l89gx-1052447225  6s
   ├-✔ infra-install-argo-rbmq-consumer       create-infra-l89gx-4148577219  15s
   ├-✔ infra-install-argo-workflow-generator  create-infra-l89gx-2290595715  12s
   ├-✔ infra-install-cache-processor          create-infra-l89gx-2554795665  9s
   ├-✔ infra-install-gateway-middleware       create-infra-l89gx-4193299339  18s
   ├-✔ infra-install-jenkins                  create-infra-l89gx-2848110700  23s
   ├-✔ infra-install-loader                   create-infra-l89gx-1539524629  21s
   ├-○ infra-install-services-creation                                                 when 'false' evaluated false
   ├-○ infra-install-services-manager                                                  when 'false' evaluated false
   └-○ infra-install-shutdown-consumer                                                 when 'false' evaluated false

{DemoTime}

What's next ?

Argo-CD https://github.com/argoproj/argo-cd

Argo-CI https://github.com/argoproj/argo-ci

Argo-Events https://github.com/argoproj/argo-events

https://kintohub.com

ben@kintohub.com

Workflow on K8s

Argo

What's KintoHub ?

Integration / Build

The Problems

Argo

Argo Workflow

UseCase

UseCase

Result

Infra Automation

The Problems

KintoHub Stack

Infra Deployment

Result

What's next ?

Integration
/
Build

Infra
Automation