BENEFITS & BEST PRACTICES: TESTING  CONTAINERS & k8s MANIFESTS

Hello,

Srinivasan Sekar

Open Source Enthusiast

@srinivasanskr

srinivasanTarget

Lead Consultant

https://srini.codes

Agenda,

  • Docker
  • Best Practices
  • Demo
  • Kubernetes and manifests
  • Failure stories
  • Static checks
  • Demo
Credits: Sunil Kumar

Docker Best Practices

  • Trusted Base Image

  • Avoid using the latest tag for the base image

  • Use COPY instead of ADD

  • Do not run as root, use USER instead

  • Avoid using the Sudo command

  • Avoid storing secrets as ENV variables

  • Avoid curl bashing

  • Use multi-stage builds

Conftest

  • Conftest is a utility to help you write tests against structured configuration data.

  • Conftest relies on the Rego language from Open Policy Agent for writing policies.

Open Policy Agent

  • OPA gives you the ability to offload the decision-making process from policy enforcement to a dedicated engine.

  • Domain agnostic general-purpose agent

  • Unified toolset and framework for policy across the cloud native stack.

DEMO

Kubernetes Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-deployment
  labels:
    app: orders
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
      - name: orders
        image: registry.com/orders:tag
        ports:
        - containerPort: 8080

  • Kubernetes workloads are described in YAML manifests

apiVersion: v1
kind: Service
metadata:
  name: orders-service
spec:
  selector:
    app: orders
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Kubernetes Best Practices

  • Health checks

  • Graceful shutdown

  • Fault tolerance

  • Resource Utilisation

  • Resource tagging (labels)

  • Configuration and Secrets

  • Pod security policy

  • Namespaces, etc

Static Checks

  • Integrating Static checks allows catching policy violations and errors close to dev cycle.

  • Static checks can be categorised as:

    • API Validators

    • Built-in Checkers

    • Custom Validators

API Validators

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: one
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

API Validators

  • Validates k8s manifest against the k8s API server

  • Using schemas generated from the Kubernetes OpenAPI specification

Built In Checkers

  • Validates k8s manifest against the in-built opinionated checks

One size doesn't fit all:
Datadog ensures that they don't use ImagePullPolicy as "Always"

Custom Validators

  • Conftest is a utility to help you write tests against structured configuration data.

  • Write tests for your Kubernetes configurations, Terraform code, Serverless configs or any other structured data.

DEMO

Resources

  • https://k8s.af/
  • https://github.com/open-policy-agent/conftest/tree/master/examples
  • https://learnk8s.io/
  • https://kubernetes.io/blog/2019/03/22/kubernetes-end-to-end-testing-for-everyone/
  • https://kube-score.com/
  • https://www.mobilise.cloud/kubernetes-deployment-strategies/
  • https://www.youtube.com/watch?v=QXApVwRBeys
  • https://www.youtube.com/watch?v=_wySvT2uqyM

Testing Containers & k8s Manifests

By Srinivasan Sekar

Made with Slides.com

Testing Containers & k8s Manifests

  • 535

More from Srinivasan Sekar