Running Virtual Machines On Kubernetes with libvirt and kvm

Roman Mohr & Fabian Deutsch, Red Hat, KVM Forum, 2017

Fabian Deutsch

Fedora user and former package maintainer

oVirt and KubeVirt Contributor

Working at Red Hat

Roman Mohr

oVirt and KubeVirt Contributor

Working at Red Hat

Warning!

This is research, are you sure you want to continue?

Virtualization is omnipresent. today.

(drome, CC BY-NC 2.0)

Containers as well.

(davehamster, CC BY 2.0)

Containers look, taste, and
smell the same - just better

"Versatile, scalable, hyped, community driven, devops, …"
Take this with a grain of salt.

(colinwarren, CC BY-NC 2.0)

"How do we get there?"

 

"can I replace my VMs with containers? HOW!?"

Are they REALLY substitutes?
Is the one like the other?

Technology? Features? Feeling? Tools? Requirements?

⇝ It depends

Not yet?
"never"?

Yes

No

Cool

Replace?

Migration

If workloads can be moved to containers, then it's a migration

Convergence

If not, then we still want convergence

Yes

No

Replace?

xx%

YY%

Yes

Migration Path?

Replace?

Doubled Infrastructure?

No

Management Plane

Storage

Network

 

Virtual Machines

 

Management Plane

Storage

Network

 

Containers

 

2x Infrastructure?

Virtualization
and
containers

(giphy)

KubeVirt

Containers
&
Virtual Machines

on the same infrastructure.

Management Plane

Storage

Network

 

Virtual Machines

 

 

Containers

 

Keep your VMs …

Management Plane

Storage

Network

 

Virtual Machines

 

 

Containers

 

… Transition what you need …

Management Plane

Storage

Network

 

Virtual Machines

 

 

Containers

 

… And stick to VMs as needed.

Woot?

Tell me more.

(giphy)

Kubernetes

Storage

Network

 

Virtual Machines

 

 

Containers

 

Kubernetes

Storage

Network

 

Virtual Machines

 

 

Containers

 

+ KubeVirt

Kubernetes integration

kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
  nodeSelector:
    cpu: fast
status:
  phase: Running

Kubernetes API

“A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.” *

 

 

 

 

* https://kubernetes.io/docs/concepts/workloads/pods/pod/#what-is-a-pod

CRI? Can a pod be a VM?

  • Add device details as annotations.
  • Modify the container runtime on every node.
  • Deal with the fact that there are two Pods when you do migrations.
  • Implement as much functionality as possible from the Kubelet, since there is not way to distinguish from outside what your VM Pod supports, compared to a normal Pod.
  • Are we talking about a VM Pod or a Pod?

Can VMs just be pods?

Or: Implementing a CRI

  • Allows a proper Virtual Machine Specification
  • We can ship KubeVirt as a pure add-on. No Node modifications are necessary.
  • No matter, how much Pods are necessary to perform a migration, we have one single entrypoint to the Virtual Machine.
  • Reuse all of the kubelet and Pod Spec functionality, by running a Virtual Machine inside the Pod
  • Talk about VMs when they are VMs, talk about Pods when they are Pods.

A VM API?

No, an explicit virtualization API

VMs are different to pods, they require a dedicated API.

API & runtime

(source)

API
challenges

  • Nothing Node-ish
  • Add Kube concepts
  • Add cluster concepts
  • Imperative vs. Declarative
  • How opinionated?

API challenges

  • Feature wise comparable to domxml
  • But
    • Some features are node specific - they need to be abstracted
    • Needs to be married with Kubernetes concepts (pv, networks)
    • Needs additional data for cluster-only features like scheduling
  • Imperative vs. Declarative
    • Actions


kind: Migration

metadata:
  generateName: my-migration

spec:
  nodeSelector:
    kubevirt.io/hostname: node1

  selector:
    name: testvm

status:
  phase: Succeeded


kind: VirtualMachine

metadata:
  name: testvm

spec:
  domain:
    devices:
      type: PersistentVolumeClaim
      device: disk
      source:
        name: myVolumeClaim

  nodeSelector:
    cpu: fast

status:
  phase: Running
kind: Migration
metadata:
  generateName: my-migration
spec:
  nodeSelector:
    kubevirt.io/hostname: node1
  selector:
    name: testvm
status:
  phase: Succeeded

MIgrations

Backed by a controller:

  • On object create, schedules a new Pod

  • On successful Pod start, it triggers the migration

  • At the end of the migration the object is moved to a final state

  • Always one VirtualMachine object you reference

 

The objects Migration with VirtualMachine provide a consistent entry point to anything VirtualMachine related, like the Pod does for Kubernetes.

 

kind: VirtualMachine
metadata:
  name: testvm
spec:
  domain:
    devices:
      type: PersistentVolumeClaim
      device: disk
      source:
        name: myVolumeClaim
  nodeSelector:
    cpu: fast
status:
  phase: Running

KUBEVIRT API

We have the typical Pod like structure:

  • Metadata section

  • Specification section

  • Typical Pod features like

    • nodeSelector

    • affinity

  • Status section


Behind the scene a Pod is created, scheduled  and we make sure that the VM starts correctly inside.

# Typical Pod commands:

kubectl create -f mypodspec.yaml
kubectl delete mypod
kubectl exec mypod -it /bin/bash

# Typical VirtualMachine commands:

kubectl create -f myvmspec.yaml

kubectl plugin virt console testvm
kubectl plugin virt spice testvm

kubectl create -f myvmmigration.yaml

kubectl delete testvm

Workflow

kind: VirtualMachine
metadata:
  name: testvm
spec:
  domain:
    devices:
      graphics:
      - type: spice
      consoles:
      - type: pty


      

Typical Pod commands:

  • kubectl create -f mypodspec.yaml

  • kubectl delete mypod

  • kubectl exec mypod -it /bin/bash

 

Typical VirtualMachine commands:

  • kubectl create -f myvmspec.yaml

  • kubectl delete testvm

  • kubectl plugin virt console testvm

  • kubectl plugin virt spice testvm

TYPICAL KUBECTL FEELING

Integration challenges

  • qemu and libvirt in a Pod
  • cgroups and namespaces
  • Disk and networks
  • VM migrations
  • Functionality ownership

INTEGRATION Challenges

Properly integrate the VirtualMachine lifecycle in a Pod lifecycle.

 

  • Disks
  • Networking
  • qemu with libvirt in a Pod
  • cgroups and Namespaces
  • Migrations on top of Kubernetes

Features

  • Cloud Init
  • Console/Spice access
  • VirtualMachineReplicaSet
  • Cloud Provider
  • Nested K8s nodes for workload isolation
  • More to come ...

$ minikube start --vm-driver kvm --network-plugin cni

$ git clone https://github.com/kubevirt/demo.git

$ cd demo

$ bash run-demo.sh

Try (with minikube)

$ bash run-demo.sh
# Deploying KubeVirt
...
vm "testvm" created
Waiting for KubeVirt to be ready ...
Waiting for KubeVirt to be ready ...
Waiting for KubeVirt to be ready ...
# KubeVirt is now ready. Try:
# $ kubectl get vms

$ kubectl get vms
NAME      KIND
testvm    VM.v1alpha1.kubevirt.io

$ kubectl get pods
NAME                                      READY     STATUS    RESTARTS   AGE
haproxy-723816479-wcblm                   1/1       Running   1          49s
iscsi-demo-target-tgtd-1270025779-nckbh   1/1       Running   0          48s
libvirt-8zj1k                             2/2       Running   0          48s
spice-proxy-3525077118-fswn9              1/1       Running   0          47s
virt-api-1956313626-t9rhj                 1/1       Running   0          46s
virt-controller-2251532855-tfm9f          1/1       Running   0          45s
virt-handler-s7g76                        1/1       Running   0          43s
virt-launcher-testvm-----q05vh            1/1       Running   0          38s
virt-manifest-1665692876-cs8wp            2/2       Running   0          42s

$ kubectl exec -it libvirt-8zj1k bash
Defaulting container name to libvirtd.
Use 'kubectl describe pod/libvirt-8zj1k' to see all of the containers in this pod.
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     default_testvm                 running

# exit

Add-On

Stabilize

Upstream Kubernetes

Going forward

(cuatrok, CC BY SA 2.0)

WIP and R&D

Unified API

Converged infrastructure

Summary

Thank you.

Learn and contribute at http://kubevirt.io

Summary: Our Pillars and effects.

libvirt, … everything in pods

 

New resource type for VMs

 

Operator pattern to manage VMs

 

VMs live inside pods

Native Kubernetes add-on

 

API server with VM functionality

 

Declarative, like everything else

 

Kubernetes' infrastructure is leveraged

 

 

 

(tabor-roeder, CC BY 2.0)

Made with Slides.com