CICD on OKD

Warszawa 11-02-2020 @ <WJUG>

Andrzej Goławski

(Origin Community Distribution of Kubernetes)

- Szkolenia

- Konsultacje

- Wdrożenia

- Subskrypcje/Licencje

- Support

Lest make IT better

Zapraszamy

Warszawa 02.06.2020

Application Delivery Cycle

Dev

Code

GIT

Build

Artifact

Deploy Test

Deploy Green

Testing

Internal Testing

Switch Traffic to Green

Monitoring

Deploy Blue

Switch Traffic to Blue

OpenShift

The Origin Community Distribution of Kubernetes

OpenShift

Concepts:

- Source to Image (s2i)

OpenShift

Concepts:

- Source to Image (s2i)

- POD

POD

OpenShift

Concepts:

- Source to Image (s2i)

- POD

- Resource/Object

Resource

apiVersion: v1
kind: Pod
metadata:
  name: simple-app
spec:
  containers:
  - name: simple-app-container
    image: andipansa/courses-service

oc create -f simple-pod.yaml

OpenShift

Concepts:

- Source to Image (s2i)

- POD

- Resource/Object

- Image Stream

Image Stream

kind: Deployment
....
  containers:
  - image: mailcatcher

---

kind: Pod
....
  containers:
  - image: mailcatcher

kubectl run mailcatcher --image=mailcatcher

oc new-app mailcatcher

mailcatcher

DOCKERHUB

is

kind: ImageStream
....
tags:
  from:
    kind: DockerImage
    name: mailcatcher
  name: latest

---

kind: DeploymentConfig
....
from:
  kind: ImageStreamTag
  name: mailcatcher:latest
---

kind: Pod
....
  containers:
  - image: mailcatcher@sha256:4....

OpenShift

Concepts:

- Source to Image (s2i)

- POD

- Resource/Object

- Image Stream

- Project

Application Delivery Cycle

GIT

Code

Build

Deploy

APP

inner registry

push

pull

clone

app

app-test

app-prod

Deploy

Build Process on OpenShift

openshift

app

inner registry

s2i-java11-maven

bc

s2i

application

is

GIT

push

clone

oc new-build --name=app s2i-java11-maven~http://URL-TO-REPO

oc new-project app

Build Process on OpenShift

#!/bin/bash

#Create app project
oc new-project app

#Create Build Configuration 
oc new-build --name=app s2i-java11-maven~$APP_GIT_URL -n app

Build Application script: build-app.sh

Deploy Process on OpenShift

app

app-test

inner registry

bc

is

app:latest

dc

rc

pod

pull

app:test

oc new-app --name=app-test --image-stream=app/app:test --allow-missing-imagestream-tags

oc tag app:latest app:test -n app

oc new-project app-test

oc policy add-role-to-user system:image-puller system:serviceaccount:app-test:default -n app

Deploy Process on OpenShift

#!/bin/bash

#Create app-test project
oc new-project app-test

#Add permissions to pull image from app project
oc adm policy add-role-to-user system:image-puller \
   system:serviceaccount:app-test:default -n app

#Create DeploymentConfiguration
oc new-app --name=app-test --image-stream=app/app:test --allow-missing-imagestream-tags 

#Remove unnecessary Image Stream
oc delete is app-test -n app-test

#Add readiness probe
oc set probe dc/app-test --readiness --get-url=http://:8080/actuator/health -n app-test

Deploy Application on Test script: create-app-test.sh

Network on OpenShift

app-test

pod

svc

route

http request

oc expose dc/app-test

oc expose scv/app-test

Deploy Production

#!/bin/bash

#Create project prod
oc new-project app-prod

#Add permissions to pull image from app project
oc policy add-role-to-user system:image-puller \
  system:serviceaccount:app-prod:default -n app

#Create blue and green deployments	
oc new-app --name=app-blue --image-stream=app/app:prod-blue \ 
  --allow-missing-imagestream-tags -n app-prod
oc new-app --name=app-green --image-stream=app/app:prod-green \
  --allow-missing-imagestream-tags -n app-prod

#Remove unnecessary Image Streams
oc delete is --all -n app-prod

#Expose prod applications
oc expose deploymentconfig.apps.openshift.io/app-blue --port=8080 -n app-prod
oc expose deploymentconfig.apps.openshift.io/app-green --port=8080 -n app-prod
oc expose service app-blue --hostname=blue.apps-crc.testing -n app-prod
oc expose service app-green --hostname=green.apps-crc.testing -n app-prod

#Add readiness probe
oc set probe dc/app-blue --readiness --get-url=http://:8080/actuator/health -n app-prod
oc set probe dc/app-green --readiness --get-url=http://:8080/actuator/health -n app-prod

Deploy Production script: create-app-prod.sh

Split traffic

app-test

pod

svc

route

http request

pod

svc

70%

30%

oc expose svc/app-green

oc set route-backends app-green app-green=70 app-blue=30

Expose Production

#!/bin/bash

#Create production Route
oc expose service app-blue --name prod \
  --hostname=prod.apps-crc.testing -n app-prod
oc set route-backends prod app-green=0 app-blue=100 -n app-prod

Expose production script: expose-prod.sh

"CICD" on OpenShift without Jenkis

oc start-build app -n app

build project

oc tag app:latest app:test -n app

deploy test

oc tag app:test app:prod-green -n app

deploy green

oc tag app:test app:prod-blue -n app

switch traffic to green

oc set route-backends prod \
  app-green=100 app-blue=0 -n app-prod

deploy blue

oc set route-backends prod \
  app-green=0 app-blue=100 -n app-prod

switch traffic to blue

Jenkins on OpenShift

Install Jenkins on OpenShift

Install using CLI

oc new-app jenkins-persistent

oc get templates -n openshift | grep jenkins

jenkins-ephemeral      Jenkins service, without persistent storage....
jenkins-persistent     Jenkins service, with persistent storage....

Available Templates

Install using Web Console

Jenkins on OpenShift

cicd

J

bc

kind: BuildConfig
apiVersion: v1
metadata:
  name: app-pipeline
  namespace: cicd
  labels:
    name: app-pipeline
spec:
  strategy:
    type: JenkinsPipeline
    jenkinsPipelineStrategy:
      jenkinsfile: |-
		echo 'hello world!'

oc start-build app-pipeline -n cicd

oc apply -f app-pipeline.yaml -n cicd

Jenkins on OpenShift

cicd

J

bc

stage('build') {
  echo 'build'  
} 

stage('deploy') {
  echo 'deploy'
}

jenkinsfile

oc start-build app-pipeline -n cicd

Jenkins on OpenShift

cicd

J

bc

node {
  stage('build') {
    echo 'build'  
  } 

  stage('deploy') {
    echo 'deploy'
  }
}

jenkinsfile

oc start-build app-pipeline -n cicd

node('slave') {
  stage('build') {
    echo 'build'  
  } 

  stage('deploy') {
    echo 'deploy'
  }
}

slave

execute build

execute deploy

Jenkins on OpenShift

cicd

J

bc

node {
  stage('build') {
    sh 'oc start-build app -n app'  
  } 

  stage('deploy test') {
    sh 'oc tag app:latest app:test -n app'
    sh 'oc rollout latest app-test -n app-test'
  }
}

jenkinsfile

oc start-build app-pipeline -n cicd

app

app-test

build

deploy

dc

bc

OpenShift Jenkins Pipeline (DSL) Plugin

Build Configuration Definition

kind: BuildConfig
apiVersion: v1
metadata:
  name: app-pipeline
  namespace: cicd
  labels:
    name: app-pipeline
spec:
  strategy:
    type: JenkinsPipeline
    jenkinsPipelineStrategy:
      jenkinsfile: |-
        node {
          stage("build") {
            .....
          }
        }

OpenShift Jenkins Pipeline (DSL) Plugin

Build

        node {
          stage("build") {
            openshift.withCluster() {
              openshift.withProject("app") {
                def buildConfig = openshift.selector("bc", "app")
                def build = buildConfig.startBuild("--incremental")
                build.untilEach() {
                  return it.object().status.phase == "Complete"
                }
              }
            }
          }
        }

OpenShift 3.x

        node {
          stage("build") {
          	openshiftBuild bldCfg: 'app', namespace: 'app'
          }
        }

OpenShift Jenkins Pipeline (DSL) Plugin

Deploy Test

        node {
          stage("build") {.....}
          stage("deploy to test") {
            openshift.withCluster() {
              openshift.withProject("app-test") {
                openshift.tag( "app/app:latest", "app/app:test")
                def deployConfig = openshift.selector("dc", "app-test")
                def replicasNumber = deployConfig.object().spec.replicas
                deployConfig.rollout().latest()
                def latestDeploymentVersion = 
                  deployConfig.object().status.latestVersion
                def replicationController = 
                  openshift.selector('rc', "app-test-${latestDeploymentVersion}")
                replicationController.watch {
                  def replicationControllerObject = it.object()
                  return (replicasNumber.equals(
                    replicationControllerObject.status.readyReplicas))
                }
              }
            }
          }
        }

OpenShift Jenkins Pipeline (DSL) Plugin

        node {
          stage("build") {.....}
          stage("deploy to test") {
            openshiftDeploy depCfg: 'app-test', namespace: 'app-test'
          }
        }

OpenShift 3.x

OpenShift Jenkins Pipeline (DSL) Plugin

Extract Deploy to method

        node {
          stage("build") {.....}
          stage("deploy to test") {
            openshift.withCluster() {
              openshift.withProject("app-test") {
                openshift.tag( "app/app:latest", "app/app:test")
                deploy("app-test")
              }
            }
          }
        }
        def deploy(String deploymentConfigurationName) {
          def deployConfig = openshift.selector("dc", deploymentConfigurationName)
          def replicasNumber = deployConfig.object().spec.replicas
          deployConfig.rollout().latest()
          def latestDeploymentVersion = deployConfig.object().status.latestVersion
          def replicationController = 
            openshift.selector('rc', "${deploymentConfigurationName}-${latestDeploymentVersion}")
          replicationController.watch {
            def replicationControllerObject = it.object()
            return (replicasNumber.equals(
              replicationControllerObject.status.readyReplicas))
          }
        }

OpenShift Jenkins Pipeline (DSL) Plugin

Testing

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {
            input message: 'Should deploy to production as green?', ok: 'Yes'
          }
        }
        def deploy(String deploymentConfigurationName) {.....}

OpenShift Jenkins Pipeline (DSL) Plugin

Deploy Green

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {
            openshift.withCluster() {
              openshift.withProject("app-prod") {
                openshift.tag("app/app:latest", "app/app:prod-green")
                openshift.selector("dc", "app-green").scale("--replicas=1")
                deploy("app-green")
              }
            }
          }
        }
        def deploy(String deploymentConfigurationName) {.....}

OpenShift Jenkins Pipeline (DSL) Plugin

Internal testing Green

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {.....}
          stage("internal testing green") {
            input message: 'Should swich traffic to green?', ok: 'Yes'
          }
        }
        def deploy(String deploymentConfigurationName) {.....}

OpenShift Jenkins Pipeline (DSL) Plugin

Switch traffic to Green

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy on production as green") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {
            openshift.withCluster() {
              openshift.withProject('app-prod') {
               def routeObject = openshift.selector("route", "prod").object()
               routeObject.spec.alternateBackends[0].weight=100
               routeObject.spec.to.weight=0
               openshift.apply(routeObject)
              }
            }
          }
        }
        def deploy(String deploymentConfigurationName) {.....}

or

          stage("swich traffic to green") {
            sh 'oc set route-backends prod app-green=100 app-blue=0 -n app-prod'
          }

OpenShift Jenkins Pipeline (DSL) Plugin

Extract switch traffic to method

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy on production as green") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {
            openshift.withCluster() {
              openshift.withProject('app-prod') {
                switchTraffic('prod', 100, 0)
            }
          }
        }
        def deploy(String deploymentConfigurationName) {.....}
        def switchTraffic(String routeName, int greenWeight, int blueWeight) {
          def routeObject = openshift.selector("route", routeName).object()
          routeObject.spec.alternateBackends[0].weight=blueWeight
          routeObject.spec.to.weight=greenWeight
          openshift.apply(routeObject)
        }

OpenShift Jenkins Pipeline (DSL) Plugin

Testing Green on production

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {.....}
          stage("testing green on production") {
            input message: 'Should deploy to production as blue?', ok: 'Yes'
          }
        }
        def deploy(String deploymentConfigurationName) {.....}
        def switchTraffic(String routeName, int greenWeight, int blueWeight) {....}

OpenShift Jenkins Pipeline (DSL) Plugin

Deploy Blue

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {.....}
          stage("testing green on production") {.....}
          stage("deploy blue on production") {
            openshift.withCluster() {
              openshift.withProject('app-prod') {
                openshift.tag('app/app:latest', 'app/app:prod-blue')
                deploy("app-blue")
              }
            }
          }
        }
        def deploy(String deploymentConfigurationName) {.....}
        def switchTraffic(String routeName, int greenWeight, int blueWeight) {....}

OpenShift Jenkins Pipeline (DSL) Plugin

Switch traffic to Blue

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {.....}
          stage("testing green on production") {.....}
          stage("deploy blue on production") {.....}
          stage("swich traffic to blue") {
            openshift.withCluster() {
              openshift.withProject('app-prod') {
                switchTraffic('prod', 0, 100)
              }
            }
          }
        }
        def deploy(String deploymentConfigurationName) {.....}
        def switchTraffic(String routeName, int greenWeight, int blueWeight) {....}

OpenShift Jenkins Pipeline (DSL) Plugin

Turn off Geen

        node {
          stage("build") {.....}
          stage("deploy to test") {.....}
          stage("manual testing") {.....}
          stage("deploy green on production") {.....}
          stage("internal testing green") {.....}
          stage("swich traffic to green") {.....}
          stage("testing green on production") {.....}
          stage("deploy blue on production") {.....}
          stage("swich traffic to blue") {.....}
          stage("turn off green") {
          	openshift.selector("dc", 'app-green').scale('--replicas=0')
          }
        }
        def deploy(String deploymentConfigurationName) {.....}
        def switchTraffic(String routeName, int greenWeight, int blueWeight) {....}

OpenShift Jenkins Pipeline (DSL) Plugin

Let's run IT

#!/bin/bash

#Add permissions for jenkins
oc policy add-role-to-user edit system:serviceaccount:cicd:jenkins -n app
oc policy add-role-to-user edit system:serviceaccount:cicd:jenkins -n app-test
oc policy add-role-to-user edit system:serviceaccount:cicd:jenkins -n app-prod

Setup permissions for Jenkins

OpenShift Jenkins Pipeline (DSL) Plugin

Turn off the triggers

#!/bin/bash

#Turn off the triggers for dc in project app-test
oc set triggers dc/app-test --from-config --remove -n app-test
oc patch dc/app-test \
  -p '[{"op": "replace", "path": "/spec/triggers/0/imageChangeParams/automatic", "value":false}]' \ 
  --type=json -n app-test

#Turn off the triggers for dc in project app-prod
oc set triggers dc/app-green --from-config --remove -n app-prod
oc set triggers dc/app-blue --from-config --remove -n app-prod
oc patch dc/app-green -p ....
oc patch dc/app-blue -p ....

oc apply -f app-pipeline.yaml