Pet VMs in Kubernetes? WTH

Fabian Deutsch, Red Hat

devconf.cz, 2017

@dummdida - dummida.tumblr.com

https://www.flickr.com/photos/infinitelydigital/3763912574/

Agenda

• Pet VMs
• Kubernetes & Containers
• What they share
• How they differ
• A first try
• A good representation
• And how to add the functionality
• KubeVirt
• Gaps

Workloads: Differ

• The assumptions of Kubernetes do not apply to pet VMs
• Pet VMs can

Strongest point: The assumptions about them differ

Infrastructure: Similar

Container Cluster

VM Cluster

Infrastructure: Similar

… they have much in common: Infrastructure.

Cluster, Schedule, Storage, Network, …

Sharing?

Even if the workload is different, can both workloads share the same infrastructure?

Let's see how we can run both workloads on the same infra. Would allow sharing.

s /docker/qemu/

Try: Don't run a pod but a pet VM instead

Keep everything Kubernetes, just replace the launcher

"VM Driver"

Let's see pods as VM representations and let's get the kubelet to launch VMs.

Representation: Inadequat

Handling: Ugly

Ugh.

(Fail)

Representation + Handling

Some issues:

• Tried to use Pod Spec to launch a VM
• Before CRI ...
• Representation difficulties
  • Display
  • Boot order
  • …
• Technical difficulties
  • Storage - file-system vs block storage
• Assumption: Stateless

KubeVirt

• Virtualization add-on to Kubernetes
• Resources, controllers & daemons

$ kubectl create -f - <<EOF
kind: VM

apiVersion: kubevirt.io/v1alpha1

metadata:
  name: testvm

spec:
  nodeSelector:
    kubernetes.io/hostname: master
  domain:
    devices:
      interfaces:
      - source:
          network: default
        type: network
      video:
      - model:
          type: qxl
          heads: 1
      graphics:
      - autoPort: 'yes'
        defaultMode: any
        type: spice
    memory:
      unit: KiB
      value: 8192
    os:
      type:
        os: hvm
    type: qemu

RE: VM Representation

VM: 3rd Party Resource

VM TPR

• Pet VMs have different knobs than Pods
• Custom Type to represent VMs
• Defined and hosted by Kubernetes.
• Can be manipulated through the native Kubernetes REST API

RE: VM Handling

Controllers & Daemons

(Operator Pattern)

Controllers

Watch for VMs, create associated pod: Ability to reuse pod features.

Handle virtualization specific cluster tasks: live-migration

Daemons

Handle virtualization functionality on the node side.

Responsible for launching VMs.

RE: Handling

Pet VMs can not be handled by Kubernetes.

Kubernetes has a core assumption of stateless workloads - this shines through everywhere.

Some functionality is missing - live migration.

Needs to be extended.

Kubernetes

virt-controller

VM (TPR)

virt-handler

VM

kubelet

Cluster

Host

Pod

Kubernetes

virt-controller

VMSpec (TPR)

Pod Spec

virt-handler

VM

Ouch. Issues. Ouch. Gaps.

Ouch. Gaps.

Everywhere.

Design allows us to solve in Kubernetes if it can be solved there.

Otherwise workaround/bypass in Controllers and Daemons