K8s from the Inside
Ibrahim AshShohail
@ibrasho
Ibrahim AshShohail
VP Engineering @ HungerStation
(We are growing and hiring!)
Open-source contributor
(Kubernetes, Go, go/dep ... etc)
"TechTalks Saudi" organizer
@ibrasho
HungerStation
Outline
- What is Kubernetes?
-
Kubernetes components
- Kubernetes core controllers
- Operators
What do you know about Kubernetes?
I'll give a brief anyway...
What is Kubernetes?
Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.
The Control Plane
Control Components
https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/
Control Components
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- cloud-controller-manager (still beta in v1.13)
- kubelet & kube-proxy?
What is the API Server?
The Kubernetes API is a HTTP API with JSON as its primary serialization schema, however it also supports Protocol Buffers, mainly for cluster-internal communication.
For extensibility reasons Kubernetes supports multiple API versions at different API paths, such as /api/v1 or /apis/extensions/v1beta1. Different API versions imply different levels of stability and support.
https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/
The API Server
Different API versions imply different levels of stability and support:
- Alpha (e.g. v1alpha1): disabled by default
- Beta (e.g. v1beta1): enabled by default
- Stable level (e.g. v1)
The API Server
https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/
The API Server Endpoints
kubectl get --raw /
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/",
"/apis/admissionregistration.k8s.io",
"/apis/admissionregistration.k8s.io/v1beta1",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1beta1",
"/apis/apiregistration.k8s.io",
"/apis/apiregistration.k8s.io/v1",
"/apis/apiregistration.k8s.io/v1beta1",
"/apis/apps",
"/apis/apps/v1",
"/apis/apps/v1beta1",
"/apis/apps/v1beta2",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/autoscaling/v2beta1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v1beta1",
"/apis/cloud.google.com",
"/apis/cloud.google.com/v1beta1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/metrics.k8s.io",
"/apis/metrics.k8s.io/v1beta1",
"/apis/networking.k8s.io",
"/apis/networking.k8s.io/v1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1",
"/apis/rbac.authorization.k8s.io/v1beta1",
"/apis/scalingpolicy.kope.io",
"/apis/scalingpolicy.kope.io/v1alpha1",
"/apis/scheduling.k8s.io",
"/apis/scheduling.k8s.io/v1beta1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/SSH Tunnel Check",
"/healthz/autoregister-completion",
"/healthz/etcd",
"/healthz/ping",
"/healthz/poststarthook/apiservice-openapi-controller",
"/healthz/poststarthook/apiservice-registration-controller",
"/healthz/poststarthook/apiservice-status-available-controller",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/ca-registration",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/healthz/poststarthook/kube-apiserver-autoregistration",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/healthz/poststarthook/scheduling/bootstrap-system-priority-classes",
"/healthz/poststarthook/start-apiextensions-controllers",
"/healthz/poststarthook/start-apiextensions-informers",
"/healthz/poststarthook/start-kube-aggregator-informers",
"/healthz/poststarthook/start-kube-apiserver-admission-initializer",
"/healthz/poststarthook/start-kube-apiserver-informers",
"/logs",
"/metrics",
"/openapi/v2",
"/swagger-2.0.0.json",
"/swagger-2.0.0.pb-v1",
"/swagger-2.0.0.pb-v1.gz",
"/swagger.json",
"/swaggerapi",
"/version"
]
}
K8s Resources
kubectl get --raw /api/v1
{
"kind": "APIResourceList",
"groupVersion": "v1",
"resources": [..., {
"name": "configmaps",
"singularName": "",
"namespaced": true,
"kind": "ConfigMap",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["cm"]
}, {
"name": "endpoints",
"singularName": "",
"namespaced": true,
"kind": "Endpoints",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["ep"]
}, ..., {
"name": "limitranges",
"singularName": "",
"namespaced": true,
"kind": "LimitRange",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["limits"]
}, {
"name": "namespaces",
"singularName": "",
"namespaced": false,
"kind": "Namespace",
"verbs": ["create", "delete", "get", "list", "patch", "update", "watch"],
"shortNames": ["ns"]
}, ..., {
"name": "nodes",
"singularName": "",
"namespaced": false,
"kind": "Node",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["no"]
}, ..., {
"name": "persistentvolumeclaims",
"singularName": "",
"namespaced": true,
"kind": "PersistentVolumeClaim",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["pvc"]
}, ..., {
"name": "persistentvolumes",
"singularName": "",
"namespaced": false,
"kind": "PersistentVolume",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["pv"]
}, ..., {
"name": "pods",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["po"],
"categories": ["all"]
}, {
"name": "pods/attach",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": []
}, ..., {
"name": "pods/exec",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": []
}, {
"name": "pods/log",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": ["get"]
}, {
"name": "pods/portforward",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": []
}, {
"name": "pods/proxy",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": []
}, {
"name": "pods/status",
"singularName": "",
"namespaced": true,
"kind": "Pod",
"verbs": ["get", "patch", "update"]
}, {
"name": "podtemplates",
"singularName": "",
"namespaced": true,
"kind": "PodTemplate",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
}, {
"name": "replicationcontrollers",
"singularName": "",
"namespaced": true,
"kind": "ReplicationController",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["rc"],
"categories": ["all"]
}, ..., {
"name": "resourcequotas",
"singularName": "",
"namespaced": true,
"kind": "ResourceQuota",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["quota"]
}, ..., {
"name": "secrets",
"singularName": "",
"namespaced": true,
"kind": "Secret",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
}, {
"name": "serviceaccounts",
"singularName": "",
"namespaced": true,
"kind": "ServiceAccount",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["sa"]
}, {
"name": "services",
"singularName": "",
"namespaced": true,
"kind": "Service",
"verbs": ["create", "delete", "get", "list", "patch", "update", "watch"],
"shortNames": ["svc"],
"categories": ["all"]
}, ...]
}
K8s Resources
kubectl get --raw /apis/apps/v1
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "apps/v1",
"resources": [{
"name": "controllerrevisions",
"singularName": "",
"namespaced": true,
"kind": "ControllerRevision",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
}, {
"name": "daemonsets",
"singularName": "",
"namespaced": true,
"kind": "DaemonSet",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["ds"],
"categories": ["all"]
}, {
"name": "daemonsets/status",
"singularName": "",
"namespaced": true,
"kind": "DaemonSet",
"verbs": ["get", "patch", "update"]
}, {
"name": "deployments",
"singularName": "",
"namespaced": true,
"kind": "Deployment",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["deploy"],
"categories": ["all"]
}, {
"name": "deployments/scale",
"singularName": "",
"namespaced": true,
"group": "autoscaling",
"version": "v1",
"kind": "Scale",
"verbs": ["get", "patch", "update"]
}, {
"name": "deployments/status",
"singularName": "",
"namespaced": true,
"kind": "Deployment",
"verbs": ["get", "patch", "update"]
}, {
"name": "replicasets",
"singularName": "",
"namespaced": true,
"kind": "ReplicaSet",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["rs"],
"categories": ["all"]
}, {
"name": "replicasets/scale",
"singularName": "",
"namespaced": true,
"group": "autoscaling",
"version": "v1",
"kind": "Scale",
"verbs": ["get", "patch", "update"]
}, {
"name": "replicasets/status",
"singularName": "",
"namespaced": true,
"kind": "ReplicaSet",
"verbs": ["get", "patch", "update"]
}, {
"name": "statefulsets",
"singularName": "",
"namespaced": true,
"kind": "StatefulSet",
"verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"],
"shortNames": ["sts"],
"categories": ["all"]
}, {
"name": "statefulsets/scale",
"singularName": "",
"namespaced": true,
"group": "autoscaling",
"version": "v1",
"kind": "Scale",
"verbs": ["get", "patch", "update"]
}, {
"name": "statefulsets/status",
"singularName": "",
"namespaced": true,
"kind": "StatefulSet",
"verbs": ["get", "patch", "update"]
}]
}
The API Server
{
"kind": "APIGroupList",
"apiVersion": "v1",
"groups": [{
"name": "apiregistration.k8s.io",
"versions": [{
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}, {
"groupVersion": "apiregistration.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}
}, {
"name": "extensions",
"versions": [{
"groupVersion": "extensions/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "extensions/v1beta1",
"version": "v1beta1"
}
}, {
"name": "apps",
"versions": [{
"groupVersion": "apps/v1",
"version": "v1"
}, {
"groupVersion": "apps/v1beta2",
"version": "v1beta2"
}, {
"groupVersion": "apps/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "apps/v1",
"version": "v1"
}
}, {
"name": "authentication.k8s.io",
"versions": [{
"groupVersion": "authentication.k8s.io/v1",
"version": "v1"
}, {
"groupVersion": "authentication.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "authentication.k8s.io/v1",
"version": "v1"
}
}, {
"name": "authorization.k8s.io",
"versions": [{
"groupVersion": "authorization.k8s.io/v1",
"version": "v1"
}, {
"groupVersion": "authorization.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "authorization.k8s.io/v1",
"version": "v1"
}
}, {
"name": "autoscaling",
"versions": [{
"groupVersion": "autoscaling/v1",
"version": "v1"
}, {
"groupVersion": "autoscaling/v2beta1",
"version": "v2beta1"
}],
"preferredVersion": {
"groupVersion": "autoscaling/v1",
"version": "v1"
}
}, {
"name": "batch",
"versions": [{
"groupVersion": "batch/v1",
"version": "v1"
}, {
"groupVersion": "batch/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "batch/v1",
"version": "v1"
}
}, {
"name": "certificates.k8s.io",
"versions": [{
"groupVersion": "certificates.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "certificates.k8s.io/v1beta1",
"version": "v1beta1"
}
}, {
"name": "networking.k8s.io",
"versions": [{
"groupVersion": "networking.k8s.io/v1",
"version": "v1"
}],
"preferredVersion": {
"groupVersion": "networking.k8s.io/v1",
"version": "v1"
}
}, {
"name": "policy",
"versions": [{
"groupVersion": "policy/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "policy/v1beta1",
"version": "v1beta1"
}
}, {
"name": "rbac.authorization.k8s.io",
"versions": [{
"groupVersion": "rbac.authorization.k8s.io/v1",
"version": "v1"
}, {
"groupVersion": "rbac.authorization.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "rbac.authorization.k8s.io/v1",
"version": "v1"
}
}, {
"name": "storage.k8s.io",
"versions": [{
"groupVersion": "storage.k8s.io/v1",
"version": "v1"
}, {
"groupVersion": "storage.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "storage.k8s.io/v1",
"version": "v1"
}
}, {
"name": "admissionregistration.k8s.io",
"versions": [{
"groupVersion": "admissionregistration.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "admissionregistration.k8s.io/v1beta1",
"version": "v1beta1"
}
}, {
"name": "apiextensions.k8s.io",
"versions": [{
"groupVersion": "apiextensions.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "apiextensions.k8s.io/v1beta1",
"version": "v1beta1"
}
}, {
"name": "scheduling.k8s.io",
"versions": [{
"groupVersion": "scheduling.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "scheduling.k8s.io/v1beta1",
"version": "v1beta1"
}
}, {
"name": "prow.k8s.io",
"versions": [{
"groupVersion": "prow.k8s.io/v1",
"version": "v1"
}],
"preferredVersion": {
"groupVersion": "prow.k8s.io/v1",
"version": "v1"
}
}, {
"name": "bitnami.com",
"versions": [{
"groupVersion": "bitnami.com/v1alpha1",
"version": "v1alpha1"
}],
"preferredVersion": {
"groupVersion": "bitnami.com/v1alpha1",
"version": "v1alpha1"
}
}, {
"name": "certmanager.k8s.io",
"versions": [{
"groupVersion": "certmanager.k8s.io/v1alpha1",
"version": "v1alpha1"
}],
"preferredVersion": {
"groupVersion": "certmanager.k8s.io/v1alpha1",
"version": "v1alpha1"
}
}, {
"name": "scalingpolicy.kope.io",
"versions": [{
"groupVersion": "scalingpolicy.kope.io/v1alpha1",
"version": "v1alpha1"
}],
"preferredVersion": {
"groupVersion": "scalingpolicy.kope.io/v1alpha1",
"version": "v1alpha1"
}
}, {
"name": "cloud.google.com",
"versions": [{
"groupVersion": "cloud.google.com/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "cloud.google.com/v1beta1",
"version": "v1beta1"
}
}, {
"name": "metrics.k8s.io",
"versions": [{
"groupVersion": "metrics.k8s.io/v1beta1",
"version": "v1beta1"
}],
"preferredVersion": {
"groupVersion": "metrics.k8s.io/v1beta1",
"version": "v1beta1"
}
}]
}
Terminology
-
Kind: a string that defines the type of a resource in K8s API. We have 3 variants:
- Objects: represent entities. (e.g. Pod, Service)
- Lists: represent lists of Objects. (e.g. PodList)
-
Special kinds:
-
API Group: is a string that identifies a collection of related Kinds. (e.g. v1, apps/v1, extensions/v1beta1)
- Version: Each API Group can have multiple versions. (e.g. v1alpha1 -> v1beta2 -> v1)
Extension Methods
- Custom Resource Definition (CRDs):
Formerly were called Third Party Resources (TPRs).
simple and fairly flexible
Define your own object kinds and let the API Server handle the entire lifecycle
Deploy your own controllers to reacts to these objects.
- User API Servers (UAS):
Also called the aggregated API server.
They run in parallel to the main API Server.
Require more development effort but have more flexibility and control.
CRDs and Operators are wining
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: meetups.cncf.io
spec:
group: cncf.io
version: v1
names:
kind: Meetup
plural: meetups
scope: Namespaced
https://schd.ws/hosted_files/kccnceu18/4b/Sig%20API%20Machinery%20Deep%20Dive.pdf
Admission Controllers
- a piece of code that intercepts requests to the Kubernetes API server
- prior to
persistence of the object - after authentication and authorization.
- prior to
-
Admission controllers may be “validating”, “mutating”, or both.
- The admission control process proceeds in two phases. In the first phase, mutating admission controllers are run. In the second phase, validating admission controllers are run. Note again that some of the controllers are both.
Admission Controllers
- Enabled by default
- NamespaceLifecycle
- LimitRanger
- ServiceAccount
- PersistentVolumeClaimResize
- DefaultStorageClass
- DefaultTolerationSeconds
- MutatingAdmissionWebhook
- ValidatingAdmissionWebhook
- ResourceQuota
- Priority
Dynamic Admission Controllers
ValidatingAdmissionWebhook and the
kube-controller-manager
-
Node Controller: Responsible for noticing and responding when nodes go down.
-
Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
- Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
- Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.
cloud-controller-manager
-
Node Controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
-
Route Controller: For setting up routes in the underlying cloud infrastructure
-
Service Controller: For creating, updating and deleting cloud provider load balancers
- Volume Controller: For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes
Controller Loops
Deployment Controller
- A Deployment is a collection of ReplicaSets.
- Deployment controller listens for created or updated Deployment resources (via an informer).
- The change will be queued for reconciliation.
- Reconcile the state of ReplicaSets.
- Create new ReplicaSet if none exists.
- Create new one if Spec was changed and delete the old one.
ReplicaSet Controller
- A ReplicaSet is are a collection of Pods.
-
ReplicaSet controller listens for created or updated ReplicaSet resources (via an informer).
- The change will be queued for reconciliation.
- Reconcile the state of Pods.
- Create new Pod if none exists matching the spec until reaching replica count.
- Delete old Pods.
ReplicaSet Controller (2)
- Create operations for Pods are batched
- to avoid thundering herd issues
- to mitigate the risk of swamping kube-apiserver with unnecessary HTTP requests when there are numerous pod bootup failures
Informers
- An informer is a pattern
- allows controllers to subscribe to storage events and easily list resources they're interested in
- provides an abstraction which is nice to work with
- takes care of a lot of the nuts and bolts such as caching (caching is important because it reduces unnecessary kube-apiserver connections, and reduces duplicate serialization costs server- and controller-side).
Want more?
Questions?
Break?
Operators
History
It was back in 2016 when CoreOs first announced Operators and provided examples for
What is an operator?
An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs.
It's a replication of the controller pattern in Kubernetes.
Thank you
What do you want to hear more about?
K8s from the Inside
By Ibrahim AshShohail
