Running Virtual Machines On Kubernetes with libvirt and kvm

Roman Mohr & Fabian Deutsch, Red Hat, KVM Forum, 2017

Fabian Deutsch

Fedora user and former package maintainer

oVirt and KubeVirt Contributor

Working at Red Hat

Roman Mohr

oVirt and KubeVirt Contributor

Working at Red Hat

fabiand@redhat.com

@dummdida

fabiand

https://dummdida.tumblr.com

rmohr@redhat.com

@rfenkhuber

rmohr

Virtualization is omnipresent. today.

(drome, CC BY-NC 2.0)

Containers as well.

(davehamster, CC BY 2.0)

Containers look, taste, and
smell the same - just better

"Versatile, scalable, hyped, community driven, devops, …"
Take this with a grain of salt.

(colinwarren, CC BY-NC 2.0)

"How do we get there?"

 

"can I replace my VMs with containers? HOW!?"

Are they REALLY substitutes?
Is the one like the other?

Technology? Features? Feeling? Tools? Requirements?

⇝ It depends

Not yet?
"never"?

Yes

No

Cool

Replace?

Migration

If workloads can be moved to containers, then it's a migration

Convergence

If not, then we still want convergence

Yes

No

Replace?

xx%

YY%

Yes

Migration Path?

Replace?

Doubled Infrastructure?

No

Management Plane

Storage

Network

…

 

Virtual Machines

 

Management Plane

Storage

Network

…

 

Containers

 

2x Infrastructure?

Virtualization
and
containers

(giphy)

KubeVirt

Containers
&
Virtual Machines

on the same infrastructure.

Management Plane

Storage

Network

…

 

Virtual Machines

 

 

Containers

 

Keep your VMs …

Management Plane

Storage

Network

…

 

Virtual Machines

 

 

Containers

 

… Transition what you need …

Management Plane

Storage

Network

…

 

Virtual Machines

 

 

Containers

 

… And stick to VMs as needed.

Woot?

Tell me more.

(giphy)

Kubernetes

Storage

Network

…

 

Virtual Machines

 

 

Containers

 

Kubernetes

Storage

Network

…

 

Virtual Machines

 

 

Containers

 

+ KubeVirt

Kubernetes integration

Kubernetes

“Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.” *

 

 

 

 

* https://kubernetes.io/

kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
  nodeSelector:
    cpu: fast
status:
  phase: Running

Kubernetes API

“A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.” *

 

 

 

 

* https://kubernetes.io/docs/concepts/workloads/pods/pod/#what-is-a-pod

CRI? Can a pod be a VM?

  • Add device details as annotations.
  • Modify the container runtime on every node.
  • Deal with the fact that there are two Pods when you do migrations.
  • Implement as much functionality as possible from the Kubelet, since there is not way to distinguish from outside what your VM Pod supports, compared to a normal Pod.
  • Are we talking about a VM Pod or a Pod?

Can VMs just be pods?

Or: Implementing a CRI

  • Allows a proper Virtual Machine Specification
  • We can ship KubeVirt as a pure add-on. No Node modifications are necessary.
  • No matter, how much Pods are necessary to perform a migration, we have one single entrypoint to the Virtual Machine.
  • Reuse all of the kubelet and Pod Spec functionality, by running a Virtual Machine inside the Pod
  • Talk about VMs when they are VMs, talk about Pods when they are Pods.

A VM API?

No, an explicit virtualization API

VMs are different to pods, they require a dedicated API.

Kubevirt

 

Virtual machine

API & runtime

on top of kubernetes

(NOT A CRI)

(source)

* https://github.com/kubevirt/kubevirt/tree/master/docs

  • Feature-wise comparable to domxml
    • Some features are node specific - they need to be abstracted
    • Needs to be married with Kubernetes concepts (PVs, networks)
    • Needs additional data for cluster-only features like scheduling
  • Imperative vs. Declarative

API CHALLENGES

kind: VirtualMachine
metadata:
  name: testvm
spec:
  domain:
    devices:
      type: PersistentVolumeClaim
      device: disk
      source:
        name: myVolumeClaim
  nodeSelector:
    cpu: fast
status:
  phase: Running

KUBEVIRT API

We have the typical Pod like structure:

  • Metadata section

  • Specification section

  • Typical Pod features like

    • nodeSelector

    • affinity

  • Status section


Behind the scene a Pod is created, scheduled  and we make sure that the VM starts correctly inside.

kind: VirtualMachine
metadata:
  name: testvm
spec:
  domain:
    devices:
      graphics:
      - type: spice
      consoles:
      - type: pty

Typical Pod commands:​

TYPICAL KUBECTL FEELING

$ kubectl create -f mypodspec.yaml
$ kubectl delete mypod
$ kubectl exec mypod -it /bin/bash
$ kubectl create -f myvmspec.yaml
$ kubectl delete testvm
$ kubectl plugin virt console testvm
$ kubectl plugin virt spice testvm

Typical VirtualMachine commands:

kind: Migration
metadata:
  generateName: my-migration
spec:
  nodeSelector:
    kubevirt.io/hostname: node1
  selector:
    name: testvm
status:
  phase: Succeeded

MIgrations

Backed by a controller:

  • On object create, schedules a new Pod

  • On successful Pod start, it triggers the migration

  • At the end of the migration the object is moved to a final state

  • Always one VirtualMachine object you reference

 

The objects Migration with VirtualMachine provide a consistent entry point to anything VirtualMachine related, like the Pod does for Kubernetes.

 

Properly integrate the VirtualMachine lifecycle in a Pod lifecycle.

 

  • Disks
  • Networking
  • qemu with libvirt in a Pod
  • cgroups and namespaces
  • Migrations on top of Kubernetes

INTEGRATION CHALLENGES

  • Cloud Init
  • Console/Spice access
  • VirtualMachineReplicaSet
  • Cloud Provider
  • Nested K8s nodes for workload isolation
  • More to come ...

features

$ minikube start --vm-driver kvm --network-plugin cni

$ git clone https://github.com/kubevirt/demo.git

$ cd demo

$ bash run-demo.sh

Try (with minikube) 

$ bash run-demo.sh
# Deploying KubeVirt
...
vm "testvm" created
Waiting for KubeVirt to be ready ...
Waiting for KubeVirt to be ready ...
Waiting for KubeVirt to be ready ...
# KubeVirt is now ready. Try:
# $ kubectl get vms

$ kubectl get vms
NAME      KIND
testvm    VM.v1alpha1.kubevirt.io

$ kubectl get pods
NAME                                      READY     STATUS    RESTARTS   AGE
haproxy-723816479-wcblm                   1/1       Running   1          49s
iscsi-demo-target-tgtd-1270025779-nckbh   1/1       Running   0          48s
libvirt-8zj1k                             2/2       Running   0          48s
spice-proxy-3525077118-fswn9              1/1       Running   0          47s
virt-api-1956313626-t9rhj                 1/1       Running   0          46s
virt-controller-2251532855-tfm9f          1/1       Running   0          45s
virt-handler-s7g76                        1/1       Running   0          43s
virt-launcher-testvm-----q05vh            1/1       Running   0          38s
virt-manifest-1665692876-cs8wp            2/2       Running   0          42s

$ kubectl exec -it libvirt-8zj1k bash
Defaulting container name to libvirtd.
Use 'kubectl describe pod/libvirt-8zj1k' to see all of the containers in this pod.
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     default_testvm                 running

# exit

Add-On

Stabilize

 Contribute to Kubernetes

Going forward

(cuatrok, CC BY SA 2.0)

WIP and R&D

Unified API

Converged infrastructure

Summary

Containers
&
Virtual

Machines

Thank you.

http://kubevirt.io/

https://github.com/kubevirt/kubevirt/

@kubevirt

Summary: Our Pillars and effects.

libvirt, … everything in pods

 

New resource type for VMs

 

Operator pattern to manage VMs

 

VMs live inside pods

Native Kubernetes add-on

 

API server with VM functionality

 

Declarative, like everything else

 

Kubernetes' infrastructure is leveraged

⇝

 

⇝

 

⇝

 

⇝

(tabor-roeder, CC BY 2.0)

Made with Slides.com