K8s from the Inside

Ibrahim AshShohail

@ibrasho

Ibrahim AshShohail

VP Engineering @ HungerStation
(We are growing and hiring!)
 

Open-source contributor
(Kubernetes, Go, go/dep ... etc)
 

"TechTalks Saudi" organizer

@ibrasho

HungerStation

Outline

  • What is Kubernetes?
     
  • Kubernetes components
     
  • Kubernetes core controllers
     
  • Operators

What do you know about Kubernetes?

I'll give a brief anyway...

What is Kubernetes?

Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.

The Control Plane

Control Components

https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/

Control Components

  • kube-apiserver
     
  • kube-scheduler
     
  • kube-controller-manager
     
  • cloud-controller-manager (still beta in v1.13)
     
  • kubelet & kube-proxy?

What is the API Server?

The Kubernetes API is a HTTP API with JSON as its primary serialization schema, however it also supports Protocol Buffers, mainly for cluster-internal communication.

 

For extensibility reasons Kubernetes supports multiple API versions at different API paths, such as  /api/v1 or /apis/extensions/v1beta1. Different API versions imply different levels of stability and support.

https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/

The API Server

Different API versions imply different levels of stability and support:

  • Alpha (e.g. v1alpha1): disabled by default
  • Beta (e.g. v1beta1): enabled by default
  • Stable level (e.g. v1)

The API Server

https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/

The API Server Endpoints

kubectl get --raw /

{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1",
    "/apis/apiregistration.k8s.io/v1beta1",
    "/apis/apps",
    "/apis/apps/v1",
    "/apis/apps/v1beta1",
    "/apis/apps/v1beta2",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/autoscaling/v2beta1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v1beta1",
    "/apis/cloud.google.com",
    "/apis/cloud.google.com/v1beta1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/metrics.k8s.io",
    "/apis/metrics.k8s.io/v1beta1",
    "/apis/networking.k8s.io",
    "/apis/networking.k8s.io/v1",
    "/apis/policy",
    "/apis/policy/v1beta1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1",
    "/apis/rbac.authorization.k8s.io/v1beta1",
    "/apis/scalingpolicy.kope.io",
    "/apis/scalingpolicy.kope.io/v1alpha1",
    "/apis/scheduling.k8s.io",
    "/apis/scheduling.k8s.io/v1beta1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/SSH Tunnel Check",
    "/healthz/autoregister-completion",
    "/healthz/etcd",
    "/healthz/ping",
    "/healthz/poststarthook/apiservice-openapi-controller",
    "/healthz/poststarthook/apiservice-registration-controller",
    "/healthz/poststarthook/apiservice-status-available-controller",
    "/healthz/poststarthook/bootstrap-controller",
    "/healthz/poststarthook/ca-registration",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/kube-apiserver-autoregistration",
    "/healthz/poststarthook/rbac/bootstrap-roles",
    "/healthz/poststarthook/scheduling/bootstrap-system-priority-classes",
    "/healthz/poststarthook/start-apiextensions-controllers",
    "/healthz/poststarthook/start-apiextensions-informers",
    "/healthz/poststarthook/start-kube-aggregator-informers",
    "/healthz/poststarthook/start-kube-apiserver-admission-initializer",
    "/healthz/poststarthook/start-kube-apiserver-informers",
    "/logs",
    "/metrics",
    "/openapi/v2",
    "/swagger-2.0.0.json",
    "/swagger-2.0.0.pb-v1",
    "/swagger-2.0.0.pb-v1.gz",
    "/swagger.json",
    "/swaggerapi",
    "/version"
  ]
}

K8s Resources

kubectl get --raw /api/v1

{
  "kind": "APIResourceList",
  "groupVersion": "v1",
  "resources": [..., {
    "name": "configmaps",
    "singularName": "",
    "namespaced": true,
    "kind": "ConfigMap",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["cm"]
  }, {
    "name": "endpoints",
    "singularName": "",
    "namespaced": true,
    "kind": "Endpoints",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["ep"]
  }, ..., {
    "name": "limitranges",
    "singularName": "",
    "namespaced": true,
    "kind": "LimitRange",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["limits"]
  }, {
    "name": "namespaces",
    "singularName": "",
    "namespaced": false,
    "kind": "Namespace",
    "verbs": ["create", "delete", "get", "list", "patch", "update", "watch"],
    "shortNames": ["ns"]
  }, ..., {
    "name": "nodes",
    "singularName": "",
    "namespaced": false,
    "kind": "Node",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["no"]
  }, ..., {
    "name": "persistentvolumeclaims",
    "singularName": "",
    "namespaced": true,
    "kind": "PersistentVolumeClaim",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["pvc"]
  }, ..., {
    "name": "persistentvolumes",
    "singularName": "",
    "namespaced": false,
    "kind": "PersistentVolume",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["pv"]
  }, ..., {
    "name": "pods",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["po"],
    "categories": ["all"]
  }, {
    "name": "pods/attach",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": []
  }, ..., {
    "name": "pods/exec",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": []
  }, {
    "name": "pods/log",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": ["get"]
  }, {
    "name": "pods/portforward",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": []
  }, {
    "name": "pods/proxy",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": []
  }, {
    "name": "pods/status",
    "singularName": "",
    "namespaced": true,
    "kind": "Pod",
    "verbs": ["get", "patch", "update"]
  }, {
    "name": "podtemplates",
    "singularName": "",
    "namespaced": true,
    "kind": "PodTemplate",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  }, {
    "name": "replicationcontrollers",
    "singularName": "",
    "namespaced": true,
    "kind": "ReplicationController",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["rc"],
    "categories": ["all"]
  }, ..., {
    "name": "resourcequotas",
    "singularName": "",
    "namespaced": true,
    "kind": "ResourceQuota",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["quota"]
  }, ..., {
    "name": "secrets",
    "singularName": "",
    "namespaced": true,
    "kind": "Secret",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  }, {
    "name": "serviceaccounts",
    "singularName": "",
    "namespaced": true,
    "kind": "ServiceAccount",
    "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
    "shortNames": ["sa"]
  }, {
    "name": "services",
    "singularName": "",
    "namespaced": true,
    "kind": "Service",
    "verbs": ["create", "delete", "get", "list", "patch", "update", "watch"],
    "shortNames": ["svc"],
    "categories": ["all"]
  }, ...]
}

K8s Resources

 kubectl get --raw /apis/apps/v1

{
   "kind": "APIResourceList",
   "apiVersion": "v1",
   "groupVersion": "apps/v1",
   "resources": [{
     "name": "controllerrevisions",
     "singularName": "",
     "namespaced": true,
     "kind": "ControllerRevision",
     "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   }, {
     "name": "daemonsets",
     "singularName": "",
     "namespaced": true,
     "kind": "DaemonSet",
     "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
     "shortNames": ["ds"],
     "categories": ["all"]
   }, {
     "name": "daemonsets/status",
     "singularName": "",
     "namespaced": true,
     "kind": "DaemonSet",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "deployments",
     "singularName": "",
     "namespaced": true,
     "kind": "Deployment",
     "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
     "shortNames": ["deploy"],
     "categories": ["all"]
   }, {
     "name": "deployments/scale",
     "singularName": "",
     "namespaced": true,
     "group": "autoscaling",
     "version": "v1",
     "kind": "Scale",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "deployments/status",
     "singularName": "",
     "namespaced": true,
     "kind": "Deployment",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "replicasets",
     "singularName": "",
     "namespaced": true,
     "kind": "ReplicaSet",
     "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
     "shortNames": ["rs"],
     "categories": ["all"]
   }, {
     "name": "replicasets/scale",
     "singularName": "",
     "namespaced": true,
     "group": "autoscaling",
     "version": "v1",
     "kind": "Scale",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "replicasets/status",
     "singularName": "",
     "namespaced": true,
     "kind": "ReplicaSet",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "statefulsets",
     "singularName": "",
     "namespaced": true,
     "kind": "StatefulSet",
     "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
     "shortNames": ["sts"],
     "categories": ["all"]
   }, {
     "name": "statefulsets/scale",
     "singularName": "",
     "namespaced": true,
     "group": "autoscaling",
     "version": "v1",
     "kind": "Scale",
     "verbs": ["get", "patch", "update"]
   }, {
     "name": "statefulsets/status",
     "singularName": "",
     "namespaced": true,
     "kind": "StatefulSet",
     "verbs": ["get", "patch", "update"]
   }]
 }

The API Server

{
  "kind": "APIGroupList",
  "apiVersion": "v1",
  "groups": [{
    "name": "apiregistration.k8s.io",
    "versions": [{
      "groupVersion": "apiregistration.k8s.io/v1",
      "version": "v1"
    }, {
      "groupVersion": "apiregistration.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "apiregistration.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "extensions",
    "versions": [{
      "groupVersion": "extensions/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "extensions/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "apps",
    "versions": [{
      "groupVersion": "apps/v1",
      "version": "v1"
    }, {
      "groupVersion": "apps/v1beta2",
      "version": "v1beta2"
    }, {
      "groupVersion": "apps/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "apps/v1",
      "version": "v1"
    }
  }, {
    "name": "authentication.k8s.io",
    "versions": [{
      "groupVersion": "authentication.k8s.io/v1",
      "version": "v1"
    }, {
      "groupVersion": "authentication.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "authentication.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "authorization.k8s.io",
    "versions": [{
      "groupVersion": "authorization.k8s.io/v1",
      "version": "v1"
    }, {
      "groupVersion": "authorization.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "authorization.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "autoscaling",
    "versions": [{
      "groupVersion": "autoscaling/v1",
      "version": "v1"
    }, {
      "groupVersion": "autoscaling/v2beta1",
      "version": "v2beta1"
    }],
    "preferredVersion": {
      "groupVersion": "autoscaling/v1",
      "version": "v1"
    }
  }, {
    "name": "batch",
    "versions": [{
      "groupVersion": "batch/v1",
      "version": "v1"
    }, {
      "groupVersion": "batch/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "batch/v1",
      "version": "v1"
    }
  }, {
    "name": "certificates.k8s.io",
    "versions": [{
      "groupVersion": "certificates.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "certificates.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "networking.k8s.io",
    "versions": [{
      "groupVersion": "networking.k8s.io/v1",
      "version": "v1"
    }],
    "preferredVersion": {
      "groupVersion": "networking.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "policy",
    "versions": [{
      "groupVersion": "policy/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "policy/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "rbac.authorization.k8s.io",
    "versions": [{
      "groupVersion": "rbac.authorization.k8s.io/v1",
      "version": "v1"
    }, {
      "groupVersion": "rbac.authorization.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "rbac.authorization.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "storage.k8s.io",
    "versions": [{
      "groupVersion": "storage.k8s.io/v1",
      "version": "v1"
    }, {
      "groupVersion": "storage.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "storage.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "admissionregistration.k8s.io",
    "versions": [{
      "groupVersion": "admissionregistration.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "admissionregistration.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "apiextensions.k8s.io",
    "versions": [{
      "groupVersion": "apiextensions.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "apiextensions.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "scheduling.k8s.io",
    "versions": [{
      "groupVersion": "scheduling.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "scheduling.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "prow.k8s.io",
    "versions": [{
      "groupVersion": "prow.k8s.io/v1",
      "version": "v1"
    }],
    "preferredVersion": {
      "groupVersion": "prow.k8s.io/v1",
      "version": "v1"
    }
  }, {
    "name": "bitnami.com",
    "versions": [{
      "groupVersion": "bitnami.com/v1alpha1",
      "version": "v1alpha1"
    }],
    "preferredVersion": {
      "groupVersion": "bitnami.com/v1alpha1",
      "version": "v1alpha1"
    }
  }, {
    "name": "certmanager.k8s.io",
    "versions": [{
      "groupVersion": "certmanager.k8s.io/v1alpha1",
      "version": "v1alpha1"
    }],
    "preferredVersion": {
      "groupVersion": "certmanager.k8s.io/v1alpha1",
      "version": "v1alpha1"
    }
  }, {
    "name": "scalingpolicy.kope.io",
    "versions": [{
      "groupVersion": "scalingpolicy.kope.io/v1alpha1",
      "version": "v1alpha1"
    }],
    "preferredVersion": {
      "groupVersion": "scalingpolicy.kope.io/v1alpha1",
      "version": "v1alpha1"
    }
  }, {
    "name": "cloud.google.com",
    "versions": [{
      "groupVersion": "cloud.google.com/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "cloud.google.com/v1beta1",
      "version": "v1beta1"
    }
  }, {
    "name": "metrics.k8s.io",
    "versions": [{
      "groupVersion": "metrics.k8s.io/v1beta1",
      "version": "v1beta1"
    }],
    "preferredVersion": {
      "groupVersion": "metrics.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  }]
}

Terminology​

  • Kind: a string that defines the type of a resource in K8s API. We have 3 variants:
    • Objects: represent entities. (e.g. Pod, Service)
    • Lists: represent lists of Objects. (e.g. PodList)
    • Special kinds:
       
  • API Group: is a string that identifies a collection of related Kinds. (e.g. v1, apps/v1, extensions/v1beta1)
     
  • Version: Each API Group can have multiple versions. (e.g. v1alpha1 -> v1beta2 -> v1)

Extension Methods

  • Custom Resource Definition (CRDs):
    Formerly were called Third Party Resources (TPRs).
    simple and fairly flexible
    Define your own object kinds and let the API Server handle the entire lifecycle
    Deploy your own controllers to reacts to these objects.
     
  • User API Servers (UAS):
    Also called the aggregated API server.
    They run in parallel to the main API Server.
    Require more development effort but have more flexibility and control.

CRDs and Operators are wining

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: meetups.cncf.io
spec:
  group: cncf.io
  version: v1
  names:
    kind: Meetup
    plural: meetups
  scope: Namespaced

https://schd.ws/hosted_files/kccnceu18/4b/Sig%20API%20Machinery%20Deep%20Dive.pdf

Admission Controllers

  • a piece of code that intercepts requests to the Kubernetes API server
    • prior to persistence of the object
    • after authentication and authorization.
       
  • Admission controllers may be “validating”, “mutating”, or both.
     
  • The admission control process proceeds in two phases. In the first phase, mutating admission controllers are run. In the second phase, validating admission controllers are run. Note again that some of the controllers are both.

Admission Controllers

  • Enabled by default
    • NamespaceLifecycle
    • LimitRanger
    • ServiceAccount
    • PersistentVolumeClaimResize
    • DefaultStorageClass
    • DefaultTolerationSeconds
    • MutatingAdmissionWebhook
    • ValidatingAdmissionWebhook
    • ResourceQuota
    • Priority

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do

Dynamic Admission Controllers

ValidatingAdmissionWebhook and the

kube-controller-manager

  • Node Controller: Responsible for noticing and responding when nodes go down.
     
  • Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
     
  • Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
  •  
  • Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.

cloud-controller-manager

  • Node Controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
     
  • Route Controller: For setting up routes in the underlying cloud infrastructure
     
  • Service Controller: For creating, updating and deleting cloud provider load balancers
     
  • Volume Controller: For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes

Controller Loops

https://blog.couchbase.com/kubernetes-operators-game-changer/

Deployment Controller

  • A Deployment is a collection of ReplicaSets.
     
  • Deployment controller listens for created or updated Deployment resources (via an informer).
     
  • The change will be queued for reconciliation.
     
  • Reconcile the state of ReplicaSets.
    • Create new ReplicaSet if none exists.
    • Create new one if Spec was changed and delete the old one.

 

 

ReplicaSet Controller

  • A ReplicaSet is are a collection of Pods.
     
  • ReplicaSet controller listens for created or updated ReplicaSet resources (via an informer).
     
  • The change will be queued for reconciliation.
     
  • Reconcile the state of Pods.
    • Create new Pod if none exists matching the spec until reaching replica count.
    • Delete old Pods.

ReplicaSet Controller (2)

  • Create operations for Pods are batched
    • to avoid thundering herd issues
    • to mitigate the risk of swamping kube-apiserver with unnecessary HTTP requests when there are numerous pod bootup failures

Informers

  • An informer is a pattern
    • allows controllers to subscribe to storage events and easily list resources they're interested in
    • provides an abstraction which is nice to work with
    • takes care of a lot of the nuts and bolts such as caching (caching is important because it reduces unnecessary kube-apiserver connections, and reduces duplicate serialization costs server- and controller-side).

Want more?

https://github.com/jamiehannaford/what-happens-when-k8s#kube-apiserver

Questions?

Break?

Operators

History

It was back in 2016 when CoreOs first announced Operators and provided examples for etcd and Prometheus. The Tectonic platform was based on this model and it was apparently working extremely well. It wasn’t long after that when people started to build their own.

What is an operator?

An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs.

 

It's a replication of the controller pattern in Kubernetes.

Thank you

What do you want to hear more about?

Made with Slides.com