Workflow on K8s

Argo

What's KintoHub ?

 

Connect Repo

Build

Combine

Deploy 

Call

KintoHub Flow

Integration
/
Build

The Problems

Using Jenkins

Learning Curve

Scalability

Too many plugins

Slowness

Kubernetes fit

Jenkins Slave - Many languages

    Docker in Docker

    Mount docker socket

    Install language packages

- https://github.com/argoproj/argo -> Open Source

- Kubernetes Custom Resource Definition
- Cloud agnostic

- Designed for container
- Graph or Step definition of workflows

- Workflow is a YAML file

- Easy to deploy / Easy to use

- Artifact / Timeout / TTL / Retry / Exit Handler / ...


- Get logs by workflow/step

- Labels / Volumes / Env vars / Secrets / Scheduling / ...

Argo

Spec body

    - Entrypoint invocation

    - List of template definitions

        - Name of the template

        - Optionally a list of inputs /outputs

        - List of steps

            - Container/Pod invocation

Argo Workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: coinflip-recursive-
spec:
  entrypoint: coinflip
  templates:
  - name: coinflip
    steps:
    - - name: flip-coin
        template: flip-coin
    - - name: heads
        template: heads
        when: "{{steps.flip-coin.outputs.result}} == heads"
      - name: tails
        template: coinflip
        when: "{{steps.flip-coin.outputs.result}} == tails"

  - name: flip-coin
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import random
        result = "heads" if random.randint(0,1) == 0 else "tails"
        print(result)
  - name: heads
    container:
      image: alpine:3.6
      command: [sh, -c]
      args: ["echo \"it was heads\""]

UseCase

 

Build NodeJS Application

    - `git clone`
    - `npm install`
    - `npm build`
    - `npm run test`
    - `push [artifact]`
pipeline {
  agent any
  stages {
    stage('Cloning Git') {
      steps { sh 'git clone ...' }
    }
    stage('Install dependencies') {
      steps { sh 'npm install' }
    }
    stage('Testing') {
      steps { sh 'npm run test' }
    }
    stage('Push') {
      steps { sh '[push artifacts]' }
    }
  }
}

Jenkins

UseCase

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: k8s-meetup-
spec:
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 300
  entrypoint: main
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  imagePullSecrets:
  - name: regsecret
  templates:
  - name: main
    steps:
    - - name: git-init
        template: git-init
    - - name: build-app
        template: build-app
    - - name: upload-to-minio
        template: upload-to-minio
  - name: git-init
    container:
      image: alpine/git:latest
      command:
      - "sh"
      - "-c"
      args:
      - "rm -rf * && git clone https://${GIT_REPO} . && git checkout ${GIT_COMMIT} || exit 1"
      workingDir: /build
      env:
      - name: "GIT_COMMIT"
        value: "b8167974a9efb7a2d0e1e4b87d7cfcdf009eff2e"
      - name: "GIT_REPO"
        value: "github.com/kintohub/node-examples.git"
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: git-init
  - name: build-app
    container:
      image: kintocloud.azurecr.io/node894-alpine:latest
      command:
      - "sh"
      - "-c"
      args:
      - "npm install || exit 1"
      workingDir: /build
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: build-app
  - name: upload-to-minio
    container:
      image: minio/mc:RELEASE.2018-06-22T23-32-12Z
      command:
      - "sh"
      - "-c"
      args:
      - "mc config host add k8s http://${MINIO_HOST} ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY} S3v4 && mc mb --ignore-existing k8s/artifact && mc cp -r * k8s/artifact || exit 1"
      workingDir: /build
      env:
      - name: "MINIO_ACCESS_KEY"
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: accessKey
      - name: "MINIO_SECRET_KEY"
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: secretKey
      - name: "MINIO_HOST"
        value: "minio-dev:9000"
      volumeMounts:
        - name: workdir
          mountPath: /build
    metadata:
      labels:
        workflowId: k8s-meetup
        step: upload-to-minio

Argo

Result

Name:                k8s-meetup-cvgvh
Namespace:           dev
ServiceAccount:      default
Status:              Succeeded
Created:             Tue Nov 06 20:50:01 +0800 (41 seconds ago)
Started:             Tue Nov 06 20:50:01 +0800 (41 seconds ago)
Finished:            Tue Nov 06 20:50:42 +0800 (now)
Duration:            41 seconds

STEP                    PODNAME                      DURATION  MESSAGE
 ✔ k8s-meetup-cvgvh
 ├---✔ git-init         k8s-meetup-cvgvh-678981496   27s
 ├---✔ build-app        k8s-meetup-cvgvh-2911520064  7s
 └---✔ upload-to-minio  k8s-meetup-cvgvh-3551202965  4s

Infra
Automation

The Problems

How to deploy easily our K8s resources?

    - Terraform

        - No support for beta resources.
        - No Deployment, StatefulSet, DeamonSet, ...
    - Helm

       - Tiller

    - Both ?
    - Spinnaker
    - ...

KintoHub Stack

Gateway Middleware

Cache Processor

Loader

...

...

Infra Deployment

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: create-infra-
spec:
  ttlSecondsAfterFinished: 600
  activeDeadlineSeconds: 600
  entrypoint: main
  serviceAccountName: init-infra
  imagePullSecrets:
  - name: regsecret
  templates:
  - name: main
    steps:
    - - name: infra-preparation
        template: infra-preparation
    - - name: infra-install-minio
        template: infra-install-minio
      - name: infra-install-redis
        template: infra-install-redis
      - name: infra-install-rabbitmq
        template: infra-install-rabbitmq
      - name: infra-install-mongodb-replicaset
        template: infra-install-mongodb-replicaset
      - name: infra-install-nginx
        template: infra-install-nginx
      - name: infra-install-external-dns
        template: infra-install-external-dns
      - name: infra-install-openresty
        template: infra-install-openresty
      - name: infra-install-elasticsearch
        template: infra-install-elasticsearch
        when: "false"
      - name: infra-install-elasticsearch-curator
        template: infra-install-elasticsearch-curator
        when: "false"
      - name: infra-install-fluentd
        template: infra-install-fluentd
        when: "false"
      - name: infra-install-kibana
        template: infra-install-kibana
        when: "false"
      - name: infra-install-prometheus-operator
        template: infra-install-prometheus-operator
        when: "false"
    - - name: infra-install-cache-processor
        template: infra-install-cache-processor
      - name: infra-install-gateway-middleware
        template: infra-install-gateway-middleware
      - name: infra-install-loader
        template: infra-install-loader
      - name: infra-install-jenkins
        template: infra-install-jenkins
      - name: infra-install-argo-rbmq-build
        template: infra-install-argo-rbmq-build
      - name: infra-install-argo-rbmq-consumer
        template: infra-install-argo-rbmq-consumer
      - name: infra-install-argo-workflow-generator
        template: infra-install-argo-workflow-generator
      - name: infra-install-services-creation
        template: infra-install-services-creation
        when: "false"
      - name: infra-install-services-manager
        template: infra-install-services-manager
        when: "false"
      - name: infra-install-shutdown-consumer
        template: infra-install-shutdown-consumer
        when: "false"
  - name: infra-preparation
    outputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "argocd login argocd-server.argocd.svc.cluster.local --config /argocd/.argocd --username admin --password $ARGOCD_PWD --insecure || exit 1"
      env:
        - name: ARGOCD_PWD
          value: admin123
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-preparation
  - name: infra-install-minio
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=minio-$ENV && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace $ENV --path helm/charts/minio --values values-$ENV.yaml --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
        - name: "GIT_TOKEN"
          valueFrom:
            secretKeyRef:
              name: argo-git-creds
              key: token
        - name: "GIT_BRANCH"
          value: "dev"
        - name: "ENV"
          value: "dev"
        - name: "TIMEOUT"
          value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-minio
  - name: infra-install-redis
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=redis-ha-$ENV && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace $ENV --path helm/charts/redis-ha --values values-$ENV.yaml --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
        - name: "GIT_TOKEN"
          valueFrom:
            secretKeyRef:
              name: argo-git-creds
              key: token
        - name: "GIT_BRANCH"
          value: "dev"
        - name: "ENV"
          value: "dev"
        - name: "TIMEOUT"
          value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-redis
  - name: infra-install-prometheus-operator
    inputs:
      artifacts:
        - name: argocd
          path: /argocd
          s3:
            endpoint: storage.googleapis.com
            bucket: install-infra
            key: argocd.tgz
            accessKeySecret:
              name: my-gcs-s3-credentials
              key: accessKey
            secretKeySecret:
              name: my-gcs-s3-credentials
              key: secretKey
    container:
      image: kintocloud.azurecr.io/kinto-argo:latest
      command:
      - "sh"
      - "-c"
      args:
      - "APP=prometheus-operator && argocd app create $APP --config /argocd/.argocd --repo https://${GIT_TOKEN}@github.com/kintohub/KintoInfra.git --dest-namespace monitoring --path helm/charts/prometheus-operator --revision $GIT_BRANCH --upsert --env default --dest-server https://kubernetes.default.svc --sync-policy automated && argocd app wait $APP --config /argocd/.argocd --timeout $TIMEOUT || exit 1"
      env:
      - name: "GIT_TOKEN"
        valueFrom:
          secretKeyRef:
            name: argo-git-creds
            key: token
      - name: "GIT_BRANCH"
        value: "dev"
      - name: "TIMEOUT"
        value: "600"
    metadata:
      labels:
        workflowId: create-infra
        step: kinto-infra-install-prometheus-operator
...

Argo + ArgoCD

Result

Name:                create-infra-l89gx
Namespace:           dev
ServiceAccount:      init-infra
Status:              Succeeded
Created:             Tue Nov 06 20:21:39 +0800 (1 minute ago)
Started:             Tue Nov 06 20:21:39 +0800 (1 minute ago)
Finished:            Tue Nov 06 20:22:32 +0800 (15 seconds ago)
Duration:            53 seconds

STEP                                          PODNAME                        DURATION  MESSAGE
 ✔ create-infra-l89gx
 ├---✔ infra-preparation                      create-infra-l89gx-1326669295  3s
 ├-·-○ infra-install-elasticsearch                                                     when 'false' evaluated false
 | ├-○ infra-install-elasticsearch-curator                                             when 'false' evaluated false
 | ├-✔ infra-install-external-dns             create-infra-l89gx-3052507158  9s
 | ├-○ infra-install-fluentd                                                           when 'false' evaluated false
 | ├-○ infra-install-kibana                                                            when 'false' evaluated false
 | ├-✔ infra-install-minio                    create-infra-l89gx-2877376357  19s
 | ├-✔ infra-install-mongodb-replicaset       create-infra-l89gx-682468434   14s
 | ├-✔ infra-install-nginx                    create-infra-l89gx-2368329705  11s
 | ├-✔ infra-install-openresty                create-infra-l89gx-465791308   17s
 | ├-○ infra-install-prometheus-operator                                               when 'false' evaluated false
 | ├-✔ infra-install-rabbitmq                 create-infra-l89gx-4269141425  23s
 | └-✔ infra-install-redis                    create-infra-l89gx-42796882    6s
 └-·-✔ infra-install-argo-rbmq-build          create-infra-l89gx-1052447225  6s
   ├-✔ infra-install-argo-rbmq-consumer       create-infra-l89gx-4148577219  15s
   ├-✔ infra-install-argo-workflow-generator  create-infra-l89gx-2290595715  12s
   ├-✔ infra-install-cache-processor          create-infra-l89gx-2554795665  9s
   ├-✔ infra-install-gateway-middleware       create-infra-l89gx-4193299339  18s
   ├-✔ infra-install-jenkins                  create-infra-l89gx-2848110700  23s
   ├-✔ infra-install-loader                   create-infra-l89gx-1539524629  21s
   ├-○ infra-install-services-creation                                                 when 'false' evaluated false
   ├-○ infra-install-services-manager                                                  when 'false' evaluated false
   └-○ infra-install-shutdown-consumer                                                 when 'false' evaluated false

{DemoTime}

What's next ?

Argo-CD           https://github.com/argoproj/argo-cd

 

 

 

 

 

 

 

 


Argo-CI            https://github.com/argoproj/argo-ci

Argo-Events   https://github.com/argoproj/argo-events

https://kintohub.com

ben@kintohub.com

Argo - Workflow on K8s

By Benjamin

Made with Slides.com

Argo - Workflow on K8s

How, at KintoHub, we are not only using Argo to run all users builds, but we also use it to orchestrate the deployment of all of KintoHub's initial services within our cluster.

  • 217

More from Benjamin