Running Virtual Machines On Kubernetes with libvirt and kvm
Roman Mohr & Fabian Deutsch, Red Hat, KVM Forum, 2017
Fedora user and former package maintainer
oVirt and KubeVirt Contributor
Working at Red Hat
oVirt and KubeVirt Contributor
Working at Red Hat
Virtualization is omnipresent. today.
(drome, CC BY-NC 2.0)
Containers as well.
(davehamster, CC BY 2.0)
Containers look, taste, and
smell the same - just better
"Versatile, scalable, hyped, community driven, devops, …"
Take this with a grain of salt.
(colinwarren, CC BY-NC 2.0)
"How do we get there?"
"can I replace my VMs with containers? HOW!?"
Are they REALLY substitutes?
Is the one like the other?
Technology? Features? Feeling? Tools? Requirements?
⇝ It depends
If workloads can be moved to containers, then it's a migration
If not, then we still want convergence
on the same infrastructure.
Keep your VMs …
… Transition what you need …
… And stick to VMs as needed.
Tell me more.
“Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.” *
kind: Pod metadata: name: nginx labels: name: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 nodeSelector: cpu: fast status: phase: Running
“A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.” *
CRI? Can a pod be a VM?
- Add device details as annotations.
- Modify the container runtime on every node.
- Deal with the fact that there are two Pods when you do migrations.
- Implement as much functionality as possible from the Kubelet, since there is not way to distinguish from outside what your VM Pod supports, compared to a normal Pod.
- Are we talking about a VM Pod or a Pod?
Can VMs just be pods?
Or: Implementing a CRI
- Allows a proper Virtual Machine Specification
- We can ship KubeVirt as a pure add-on. No Node modifications are necessary.
- No matter, how much Pods are necessary to perform a migration, we have one single entrypoint to the Virtual Machine.
- Reuse all of the kubelet and Pod Spec functionality, by running a Virtual Machine inside the Pod
- Talk about VMs when they are VMs, talk about Pods when they are Pods.
A VM API?
No, an explicit virtualization API
VMs are different to pods, they require a dedicated API.
API & runtime
on top of kubernetes
(NOT A CRI)
- Feature-wise comparable to domxml
- Some features are node specific - they need to be abstracted
- Needs to be married with Kubernetes concepts (PVs, networks)
- Needs additional data for cluster-only features like scheduling
- Imperative vs. Declarative
kind: VirtualMachine metadata: name: testvm spec: domain: devices: type: PersistentVolumeClaim device: disk source: name: myVolumeClaim nodeSelector: cpu: fast status: phase: Running
We have the typical Pod like structure:
Typical Pod features like
Behind the scene a Pod is created, scheduled and we make sure that the VM starts correctly inside.
kind: VirtualMachine metadata: name: testvm spec: domain: devices: graphics: - type: spice consoles: - type: pty
Typical Pod commands:
TYPICAL KUBECTL FEELING
$ kubectl create -f mypodspec.yaml $ kubectl delete mypod $ kubectl exec mypod -it /bin/bash
$ kubectl create -f myvmspec.yaml $ kubectl delete testvm $ kubectl plugin virt console testvm $ kubectl plugin virt spice testvm
Typical VirtualMachine commands:
kind: Migration metadata: generateName: my-migration spec: nodeSelector: kubevirt.io/hostname: node1 selector: name: testvm status: phase: Succeeded
Backed by a controller:
On object create, schedules a new Pod
On successful Pod start, it triggers the migration
At the end of the migration the object is moved to a final state
Always one VirtualMachine object you reference
The objects Migration with VirtualMachine provide a consistent entry point to anything VirtualMachine related, like the Pod does for Kubernetes.
Properly integrate the VirtualMachine lifecycle in a Pod lifecycle.
- qemu with libvirt in a Pod
- cgroups and namespaces
- Migrations on top of Kubernetes
- Cloud Init
- Console/Spice access
- Cloud Provider
- Nested K8s nodes for workload isolation
- More to come ...
$ minikube start --vm-driver kvm --network-plugin cni
$ git clone https://github.com/kubevirt/demo.git
$ cd demo
$ bash run-demo.sh
Try (with minikube)
$ bash run-demo.sh # Deploying KubeVirt ... vm "testvm" created Waiting for KubeVirt to be ready ... Waiting for KubeVirt to be ready ... Waiting for KubeVirt to be ready ... # KubeVirt is now ready. Try: # $ kubectl get vms $ kubectl get vms NAME KIND testvm VM.v1alpha1.kubevirt.io $ kubectl get pods NAME READY STATUS RESTARTS AGE haproxy-723816479-wcblm 1/1 Running 1 49s iscsi-demo-target-tgtd-1270025779-nckbh 1/1 Running 0 48s libvirt-8zj1k 2/2 Running 0 48s spice-proxy-3525077118-fswn9 1/1 Running 0 47s virt-api-1956313626-t9rhj 1/1 Running 0 46s virt-controller-2251532855-tfm9f 1/1 Running 0 45s virt-handler-s7g76 1/1 Running 0 43s virt-launcher-testvm-----q05vh 1/1 Running 0 38s virt-manifest-1665692876-cs8wp 2/2 Running 0 42s $ kubectl exec -it libvirt-8zj1k bash Defaulting container name to libvirtd. Use 'kubectl describe pod/libvirt-8zj1k' to see all of the containers in this pod. # virsh list Id Name State ---------------------------------------------------- 1 default_testvm running # exit
Contribute to Kubernetes
(cuatrok, CC BY SA 2.0)
WIP and R&D
Summary: Our Pillars and effects.
libvirt, … everything in pods
New resource type for VMs
Operator pattern to manage VMs
VMs live inside pods
Native Kubernetes add-on
API server with VM functionality
Declarative, like everything else
Kubernetes' infrastructure is leveraged
(tabor-roeder, CC BY 2.0)
WIP Running Virtual Machines on Kubernetes
By Fabian Deutsch