Pet VMs in Kubernetes? WTH
Fabian Deutsch, Red Hat
devconf.cz, 2017
@dummdida - dummida.tumblr.com
https://www.flickr.com/photos/infinitelydigital/3763912574/
Agenda
- Pet VMs
- Kubernetes & Containers
- What they share
- How they differ
- A first try
- A good representation
- And how to add the functionality
- KubeVirt
- Gaps
Why?
Pet VMs
- Many knobs and switches
- Hand crafted - VM and its contents
"Oh! I deleted the VM. Argh."
Kubernetes
- It's all about running containers
- Images become ready-to-use containers
"Oh - I deleted a container?"
Container Cluster
VM Cluster
Workloads: Differ
- The assumptions of Kubernetes do not apply to pet VMs
- Pet VMs can
Strongest point: The assumptions about them differ
Workloads: Differ
Container Cluster
VM Cluster
Infrastructure: Similar
Container Cluster
VM Cluster
Scheduling ~ Storage ~ Network ~ Host life-cycle
Infrastructure: Similar
… they have much in common: Infrastructure.
Cluster, Schedule, Storage, Network, …
Sharing?
Even if the workload is different, can both workloads share the same infrastructure?
Let's see how we can run both workloads on the same infra. Would allow sharing.
Same infrastructure & different workloads
Container and VM Cluster
Scheduling ~ Storage ~ Network ~ Host life-cycle
First Try
s /docker/qemu/
Try: Don't run a pod but a pet VM instead
Keep everything Kubernetes, just replace the launcher
"VM Driver"
Let's see pods as VM representations and let's get the kubelet to launch VMs.
Representation: Inadequat
Handling: Ugly
Ugh.
(Fail)
Representation + Handling
Some issues:
- Tried to use Pod Spec to launch a VM
- Before CRI ...
- Representation difficulties
- Display
- Boot order
- …
- Technical difficulties
- Storage - file-system vs block storage
- Assumption: Stateless
Try Again
KubeVirt
- Virtualization add-on to Kubernetes
- Resources, controllers & daemons
$ kubectl create -f - <<EOF
kind: VM
apiVersion: kubevirt.io/v1alpha1
metadata:
name: testvm
spec:
nodeSelector:
kubernetes.io/hostname: master
domain:
devices:
interfaces:
- source:
network: default
type: network
video:
- model:
type: qxl
heads: 1
graphics:
- autoPort: 'yes'
defaultMode: any
type: spice
memory:
unit: KiB
value: 8192
os:
type:
os: hvm
type: qemu
RE: VM Representation
VM: 3rd Party Resource
VM TPR
- Pet VMs have different knobs than Pods
- Custom Type to represent VMs
- Defined and hosted by Kubernetes.
- Can be manipulated through the native Kubernetes REST API
RE: VM Handling
Controllers & Daemons
(Operator Pattern)
Controllers
Watch for VMs, create associated pod: Ability to reuse pod features.
Handle virtualization specific cluster tasks: live-migration
Daemons
Handle virtualization functionality on the node side.
Responsible for launching VMs.
RE: Handling
Pet VMs can not be handled by Kubernetes.
Kubernetes has a core assumption of stateless workloads - this shines through everywhere.
Some functionality is missing - live migration.
Needs to be extended.
Kubernetes
virt-controller
VM (TPR)
.
.
virt-handler
VM
kubelet
Cluster
Host
Pod
Pod
Pod
Kubernetes
virt-controller
VMSpec (TPR)
Pod Spec
.
.
virt-handler
VM
$ curl run.kubevirt.io/demo.sh | bash
Up Next →
-
Disks
-
Networks
-
Improved Scheduling
…
Beyond
-
Live Migration
-
Templating
-
Host API
-
K8s TPR Improvements
-
K8s Snapshot Improvements
-
K8s Volume Cloning
…
Ouch. Issues. Ouch. Gaps.
Ouch. Gaps.
Everywhere.
Design allows us to solve in Kubernetes if it can be solved there.
Otherwise workaround/bypass in Controllers and Daemons
Opportunities
Common ground for cluster wide virtualization management.
- Focus on virtualization and consume infrastructure
- Embrace Kubernetes as a cluster manager
- Improve Kubernetes in supporting other workloads
Questions?
http://kubevirt.io
@kubevirt
irc: fabiand
fabiand@redhat.com
@dummdida
Join the KubeVirt Gathering on
Saturday, 12 noon, C236