Pet VMs in Kubernetes? WTH

Fabian Deutsch, Red Hat

devconf.cz, 2017

@dummdida - dummida.tumblr.com

https://www.flickr.com/photos/infinitelydigital/3763912574/

Agenda

  • Pet VMs
  • Kubernetes & Containers
  • What they share
  • How they differ
  • A first try
  • A good representation
  • And how to add the functionality
  • KubeVirt
  • Gaps

Why?

Pet VMs

  • Many knobs and switches
  • Hand crafted - VM and its contents

 

"Oh! I deleted the VM. Argh."

Kubernetes

  • It's all about running containers
  • Images become ready-to-use containers

 

 

"Oh - I deleted a container?"

Container Cluster

VM Cluster

Workloads: Differ

  • The assumptions of Kubernetes do not apply to pet VMs
  • Pet VMs can

 

Strongest point: The assumptions about them differ

Workloads: Differ

Container Cluster

VM Cluster

Infrastructure: Similar

Container Cluster

VM Cluster

Scheduling ~ Storage ~ Network ~ Host life-cycle

Infrastructure: Similar

… they have much in common: Infrastructure.

Cluster, Schedule, Storage, Network, …

Sharing?

Even if the workload is different, can both workloads share the same infrastructure?

 

Let's see how we can run both workloads on the same infra. Would allow sharing.

Same infrastructure & different workloads

Container and VM Cluster

Scheduling ~ Storage ~ Network ~ Host life-cycle

First Try

s /docker/qemu/

 

Try: Don't run a pod but a pet VM instead

Keep everything Kubernetes, just replace the launcher

"VM Driver"

Let's see pods as VM representations and let's get the kubelet to launch VMs.

 

Representation: Inadequat

Handling: Ugly

Ugh.

(Fail)

Representation + Handling

Some issues:

  • Tried to use Pod Spec to launch a VM
  • Before CRI ...
  • Representation difficulties
    • Display
    • Boot order
  • Technical difficulties
    • Storage - file-system vs block storage
  • Assumption: Stateless

Try Again

KubeVirt

  • Virtualization add-on to Kubernetes
  • Resources, controllers & daemons
$ kubectl create -f - <<EOF
kind: VM
apiVersion: kubevirt.io/v1alpha1
metadata:
  name: testvm
spec:
  nodeSelector:
    kubernetes.io/hostname: master
  domain:
    devices:
      interfaces:
      - source:
          network: default
        type: network
      video:
      - model:
          type: qxl
          heads: 1
      graphics:
      - autoPort: 'yes'
        defaultMode: any
        type: spice
    memory:
      unit: KiB
      value: 8192
    os:
      type:
        os: hvm
    type: qemu

RE: VM Representation

VM: 3rd Party Resource

VM TPR

  • Pet VMs have different knobs than Pods
  • Custom Type to represent VMs
  • Defined and hosted by Kubernetes.
  • Can be manipulated through the native Kubernetes REST API

RE: VM Handling

Controllers & Daemons

(Operator Pattern)

Controllers

Watch for VMs, create associated pod: Ability to reuse pod features.

 

Handle virtualization specific cluster tasks: live-migration

Daemons

Handle virtualization functionality on the node side.

Responsible for launching VMs.

RE: Handling

Pet VMs can not be handled by Kubernetes.

Kubernetes has a core assumption of stateless workloads - this shines through everywhere.

Some functionality is missing - live migration.

 

Needs to be extended.

Kubernetes

virt-controller

VM (TPR)

.

.

virt-handler

VM

kubelet

Cluster

Host

Pod

 

 

 

Pod

 

 

 

Pod

 

 

 

Kubernetes

virt-controller

VMSpec (TPR)

Pod Spec

.

.

virt-handler

VM

$ curl run.kubevirt.io/demo.sh | bash

(Fedora 25, Ubuntu 16.04.1 LTS)

Up Next

  • Disks

  • Networks

  • Improved Scheduling

Beyond

  • Live Migration

  • Templating

  • Host API

  • K8s TPR Improvements

  • K8s Snapshot Improvements

  • K8s Volume Cloning

Ouch. Issues. Ouch. Gaps.

Ouch. Gaps.

Everywhere.

Design allows us to solve in Kubernetes if it can be solved there.

Otherwise workaround/bypass in Controllers and Daemons

Opportunities

Common ground for cluster wide virtualization management.

  • Focus on virtualization and consume infrastructure
  • Embrace Kubernetes as a cluster manager
  • Improve Kubernetes in supporting other workloads

Questions?

http://kubevirt.io

@kubevirt

irc: fabiand

fabiand@redhat.com

@dummdida

Join the KubeVirt Gathering on

Saturday, 12 noon, C236

Pet VMs in Kubernetes? WTH

By Fabian Deutsch

Pet VMs in Kubernetes? WTH

  • 2,750