BENEFITS & BEST PRACTICES: TESTING CONTAINERS & k8s MANIFESTS

Hello,

Srinivasan Sekar

Open Source Enthusiast

@srinivasanskr

srinivasanTarget

Lead Consultant

https://srini.codes

Agenda,

Docker
Best Practices
Demo
Kubernetes and manifests
Failure stories
Static checks
Demo

Credits: Sunil Kumar

Docker Best Practices

Trusted Base Image
Avoid using the latest tag for the base image
Use COPY instead of ADD
Do not run as root, use USER instead
Avoid using the Sudo command
Avoid storing secrets as ENV variables
Avoid curl bashing
Use multi-stage builds

Conftest

Conftest is a utility to help you write tests against structured configuration data.
Conftest relies on the Rego language from Open Policy Agent for writing policies.

Open Policy Agent

OPA gives you the ability to offload the decision-making process from policy enforcement to a dedicated engine.
Domain agnostic general-purpose agent
Unified toolset and framework for policy across the cloud native stack.

DEMO

Kubernetes Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-deployment
  labels:
    app: orders
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
      - name: orders
        image: registry.com/orders:tag
        ports:
        - containerPort: 8080

Kubernetes workloads are described in YAML manifests

apiVersion: v1
kind: Service
metadata:
  name: orders-service
spec:
  selector:
    app: orders
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Kubernetes Best Practices

Health checks
Graceful shutdown
Fault tolerance
Resource Utilisation
Resource tagging (labels)
Configuration and Secrets
Pod security policy
Namespaces, etc

Static Checks

Integrating Static checks allows catching policy violations and errors close to dev cycle.
Static checks can be categorised as:
- API Validators
- Built-in Checkers
- Custom Validators

API Validators

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: one
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

API Validators

Validates k8s manifest against the k8s API server

Using schemas generated from the Kubernetes OpenAPI specification

Built In Checkers

Validates k8s manifest against the in-built opinionated checks

One size doesn't fit all:
Datadog ensures that they don't use ImagePullPolicy as "Always"

Custom Validators

Conftest is a utility to help you write tests against structured configuration data.
Write tests for your Kubernetes configurations, Terraform code, Serverless configs or any other structured data.

DEMO

Resources

https://k8s.af/
https://github.com/open-policy-agent/conftest/tree/master/examples
https://learnk8s.io/
https://kubernetes.io/blog/2019/03/22/kubernetes-end-to-end-testing-for-everyone/
https://kube-score.com/
https://www.mobilise.cloud/kubernetes-deployment-strategies/
https://www.youtube.com/watch?v=QXApVwRBeys
https://www.youtube.com/watch?v=_wySvT2uqyM