Kubernetes

~ Management Strategies ~

Florian Dambrine - Principal Engineer - @GumGum

Cluster Operator

> K8s Stack Provisioning

Spotinst Ocean

  • Ease of use

  • Provides GUI

  • Smart Autoscaler

  • Spot / OnDemand

EKS

  • Managed Control Plane
  • Integrated with AWS services
terraform {
  source = "git::ssh://git@bitbucket.org/gumgum/terraform-verity-modules.git//k8s"
}

inputs = {
  eks_cluster_name = "verity-onprem"

  eks_cluster_subnets = [
    "vpc-sx-va-private-us-east-1b",
    "vpc-sx-va-private-us-east-1d",
    "vpc-sx-va-private-us-east-1f",
    "vpc-sx-va-public-us-east-1b",
    "vpc-sx-va-public-us-east-1d",
    "vpc-sx-va-public-us-east-1f"
  ]

  eks_worker_subnets = [
    "vpc-sx-va-private-us-east-1b",
    "vpc-sx-va-private-us-east-1d",
    "vpc-sx-va-private-us-east-1f",
  ]

  eks_worker_scaling = {
    min = 1
    max = 20
  }
}

> K8s Stack Provisioning

EKS Control Plane

Worker Nodes

Cluster Admin

> K8s Custom Resource Definition (CRD)

A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.


A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
 

Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.

> K8s Monitoring

# kube-prometheus-stack

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitoring
  namespace: default
  labels:
    release: prometheus--production   # << ServiceMonitorSelector (Configured in Prometheus)
spec:
  endpoints:
  - interval: 30s
    targetPort: 9090
    path: /metrics
  selector:
    matchLabels:
      release: myapp--production # << ServiceMonitor will apply to services with the following labels
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    name: my-grafana-dashboard
    grafana_dashboard: "1" # << Indicates that this ConfigMap object contains
data:
  my-grafana-dashboard.json: |
  	<grafana-dashboard-as-json>

> K8s Logging

# kube-fluentd-operator​

apiVersion: logs.vdp.vmware.com/v1beta1
kind: FluentdConfig
metadata:
  name: fluentd-config
spec:
  fluentconf: |
    <match $labels(release=myapp)>
	    @type sumologic
        endpoint https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/<token>
	    source_category /eks/verity-onprem/myapp
	    log_format json
    </match>

> K8s Secrets

# sealed-secret

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: mynamespace
spec:
  encryptedData:
    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

> K8s Secrets - In Practice

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: mynamespace
spec:
  encryptedData:
    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq... # Public Key encryption
AWS Parameter Store
/eks/verity/us-east-1/prod/secret
ref+awsssm:///eks/verity/us-east-1/prod/secret
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  foo: Y2xvdWR5dWdh # Base64 encoded

> K8s DNS

# external-dns

---

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.com
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

> K8s GitOps

# argo-cd

apiVersion: "argoproj.io/v1alpha1"
kind: "Application"
metadata:
  name: verity-gitops--production-monochart-verity-go-api--production
  labels:
    app: monochart
    chart: monochart-0.25.0
    heritage: "Helm"
    release: "verity-gitops--production"
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    directory:
      recurse: true
    path: verity-go-api--production
    repoURL: https://bitbucket.org/gumgum/verity-onprem-gitops.git
    targetRevision: HEAD

EKS Control Plane

Worker Nodes

Cluster Administrator

GitOps Repo

Cluster Users

EKS Control Plane

Worker Nodes

Cluster Administrator

GitOps Repo

Developer Space

> K8s Application Chart

# monochart

replicaCount: 1

dockercfg: {}

image: {}

configMaps: {}

secrets: {}

env: {}

deployment: {}
statefulset: {}
daemonset: {}

job: {}
cronjob: {}

service: {}

serviceMonitors: {}
prometheusRules: {}

probes: {}
resources: {}

persistence: {}

ingress: {}

crd: {}

> K8s Application Repository

# helmfile

#TheGlue
.
├── README.md
├── environments.yaml
├── common.yaml
├── helmfile.yaml
└──  releases
    ├── gitops
    │   ├── confluent-platform
    │   ├── helmfile.yaml
    │   ├── kaf
    │   ├── vegeta
    │   └── verity-go-poc
    └── ops
        ├── argo-cd
        ├── helmfile.yaml
        ├── kube-fluentd-operator
        ├── kube-prometheus-stack
        ├── metrics-server
        ├── sealed-secrets
        └── traefik

> K8s GitOps Cluster Repository

# argocd

#TheClusterState

1/ Code changes

Application Repo

CI / CD

Container Registry

GitOps Repo

2/ Config Changes

Config Repo

CI / CD

Sync
Source of truth
Push
Push
Push
Push

Deploy Operator

> CI/CD Plugins for GitOps

# drone-helm-semver

#!/usr/bin/env bash

function print_version {
  cat $1 | yq e "${VERSION_PATH}" -
}

# Step 1 - Generate new values file
yq e ".image.tag = \"${VERSION}\"" ${RELEASE_ORIG_VALUES} > ${RELEASE_NEW_VALUES}

# Step 2 - Diff new values with the old one to preserve original file
diff -U0 -w -b --ignore-blank-lines ${RELEASE_ORIG_VALUES} ${RELEASE_NEW_VALUES} | tee ${RELEASE_DIFF_VALUES} | colordiff || true

# Step 3 - Apply the patch with new values
echo "|- 📋 $(patch -o ${RELEASE_DEST_VALUES} ${RELEASE_ORIG_VALUES} < ${RELEASE_DIFF_VALUES})"

echo "|- 🎏 Version changed from $(print_version ${RELEASE_ORIG_VALUES}) to $(print_version ${RELEASE_NEW_VALUES})"

# Step 4 - Override the current values file with the newly generated file
cp ${RELEASE_DEST_VALUES} ${RELEASE_ORIG_VALUES}

# drone-git-push

> K8s GitOps Cluster Repository

Business Applications
 

  • CI/CD with Docker image publish step

Application Helm Charts & Values
 

  • Defines all micro services required for the application to work
     
  • Render templated charts into Gitops

Cluster State

 

  • Stores the current state of the cluster (Flat YAML files)

GitOps Repo

Config Repo

Application Repo

> ArgoProject

ArgoWorflows

Argo CD

Argo Events

Argo Rollouts

Appendix

> Propose helmfile repo as cookiecutter 

.
├── README.md
├── environments.yaml
├── helmfile.yaml
└──  releases
    ├── apps
    │   ├── argocd-apps.gotmpl
    │   └── helmfile.yaml
    ├── gitops
    │   ├── confluent-platform
    │   ├── helmfile.yaml
    │   ├── kaf
    │   ├── vegeta
    │   └── verity-go-poc
    └── ops
        ├── argo-cd
        ├── helmfile.yaml
        ├── kube-fluentd-operator
        ├── kube-prometheus-stack
        ├── metrics-server
        ├── sealed-secrets
        └── traefik

helmfile-gitops-cookiecutter

$ helmfile --environment sync
<repo>
├── Makefile
├── README.md
├── common.yaml
├── environments.yaml
├── helmfile.yaml
└── releases
    └── gitops # Placeholder for developers
    └── ops    # Cluster Administrator namepace
        ├── argo-cd
        │   ├── helmfile.yaml
        │   └── values
        ├── external-dns
        │   ├── helmfile.yaml
        │   └── values
        ├── helmfile.yaml
        ├── kube-fluentd-operator
        │   ├── charts
        │   └── helmfile.yaml
        ├── kube-prometheus-stack
        │   ├── helmfile.yaml
        │   └── values
        └── sealed-secrets
            └── helmfile.yaml

1/ Code changes

Application Repo

CI / CD

Container Registry

GitOps Repo

2/ Config Changes

Helmfile Config Repo

CI / CD

Sync
Source of truth
Push
Publish
Push
Push

Deploy Operator

Push

Kubernetes Management Strategy

By Florian Dambrine

Made with Slides.com

Kubernetes Management Strategy

  • 1,074

More from Florian Dambrine