Look ma, no OS!

An introduction to unikernels and how to deploy an Erlang application to EC2 as a LING unikernel
A presentation for the Denver Erlang / Elixir Meetup

By: Matt Bajor

  • Infrastructure Automation Engineer on 10ft Pole
  • Rally Software, Boulder CO
  • Originally from New Hampshire
  • Currently from Denver
  • IT Helpdesk -> SA -> Webmaster -> DevOps -> Autom8

Intro to Unikernels

The first part of the presentation will be an introduction to unikernels and their benefits, a comparison to our existing 'stacks', as well as a brief overview of the current ecosystem.

Deploying LING to EC2

The second part of the presentation will be done in live demo mode. We will be building, packaging, and deploying an Erlang application (an overly simple webserver) as a LING unikernel on EC2. This will involve creating an AMI and provisioning an instance.

TL;DR:

This presentation consists of two parts:

Part 1: Intro to Unikernels

What do our existing environments look like?

Our current stack is:

  • Designed to support a wide array of hardware:
brw-rw---- 1 root floppy 2, 0 May  1 15:19 /dev/fd0
  • Designed to support a vast array of applications (POSIX).
  • Designed to maintain backwards compatibility with existing, iron based systems.
  • Sort of a forklift from the 90s (move it to the cloud Jim!).
  • Essentially thicker than it needs to be.
  • Ripe for a refactor!

Look at all of the things!

But we get all that stuff for free right?

(Thanks RMS!)

No.

  • Staffing and running ops teams is tricky.
  • Ops is required because:
    • Linux is a black box to a lot of devs.
    • Linux very generalized with support for lots of stuff, much of which is no longer needed.
    • There are tons of things running that a feature team does not know, care about, or use explicitly.
    • Maintaining all of these things takes a lot of (risky) work.
  • Security teams are also expensive:
    • Auditing 3rd party code is a daunting task.
    • Let's only use code (explicitly) if we need it.
    • Limit exposure to vulnerabilities:
      • no shell == no shellshock
      • no libopenss = no heartbleed

So what do we really need?

The things we are actually using:

  • Virtualized Disk I/O Drivers.
  • Virtualized Network I/O Drivers.
  • Virtualized CPU / RAM Access.
  • The runtime env (Erlang VM / JVM / HHVM).
  • Explicitly included libraries that provide
    functionality our app requires.
  • External, network-based services.
  • Our application.

Not:

  • Tape drive drivers
  • Python
  • libpango

  • SSH (we're immutable right?)

  • kmod

  • ** Multi-user support **

    • It's only the app running bro!

Keeping these things around does cost $$$

So let's only deploy and maintain what we are actively using.

What is a unikernel?

A unikernel is...

An application compiled with everything it requires to operate including:

  • application code
  • required app libraries
  • configuration

in addition to

  • the runtime (JVM / BEAM)
  • required system libraries*
  • a RAMdisk and PV kernel

and packaged as an artifact that can be run against a H/V

*In the form of a Library OS

More specifically, an Erlang (LING) unikernel contains

  • Your application's compiled Erlang bytecode.
  • Your application's compiled dependencies.
  • A version of the Erlang runtime (LING).
  • Any explicitly included parts of Erlang's stdlib:
    • mnesia
    • inets
    • ssl
  • A block device if the app requires it (mnesia).*
  • Any configuration information the app needs.

* goofs is a flat (think s3) plan 9 based filesystem that can be used for non-persistent block storage. ie: mnesia tables

and then...

  • Is packaged using LING's `railing` tool:
    • The output is a Xen image.
    • This artifact can be run against an internal Xen server, but we're going to further package it.
  • Is packaged using Amazon's ec2 tools:
    • This process outputs an (environment specific) AMI
  • Is deployed to EC2
    • Manually for now

There are two main types of unikernels:

Purpose built
(like an ASIC)

Generalized

(POSIX Support)

Specialized & purpose built

  • These true unikernels were made to utilize all the modern features they can in lieu of backwards compatibility. They are leaner, meaner, and (in general) more performant than the second type of unikernels.
    • Examples include:
      • LING
      • HalVM
      • MirageOS
      • Clive
  • MirageOS's OCaml compiler will even optimize your code based on the instance's runtime configuration!

Generalized 'fat' unikernels

  • These unikernels are much fatter and target running single process instances of unmodified Linux applications.
  • Examples include:
  • These are fatter than specialized unikernels and not true 'unikernels' as they support more than just the specific compiled application deployed.

I like the specialized ones

We have been moving our 1990's workloads around from mainframes to servers to clouds to containers without changing the architecture much. Now that we have something that works (cloud like platforms), let's invest in making applications that fully utilize the potential of our fancy ass platforms.

But how is that possible?

Paravirtualization

Paravirtualization (the type of virtualization Xen provides) basically exposes hardware in a similar, but consistent manner to the guests.

This means there is no need to support specific hardware in guests; only the virtualized hardware interfaces which remain consistent.

Network Interfaces

No longer do we need to run applications co-located on the same hardware in order to communicate. Almost all inter-process comms in an SOA is done over the network allowing each part of the architecture to run in isolation with all comms happening on the network layer.

...oh yeah. and dropping backwards compability

In order to reap the benefits that a true unikernel architecture offers, it is required to design (or redesign) the application to run in such an environment. This lack of general POSIX support is why a true unikernel approach can be so much leaner and less complex than existing Linux based stacks.

 

Luckily with languages like Java and Erlang, this can mostly be done at the runtime level requiring minimal application code changes if your app supports running in an ephemeral SOA environment.

Just like your app!

Specifically, why is this better?

In three words:

  • Efficiency
  • Security
  • Complexity

Efficiency

  • Images go from GB to MB (or even KB!) reducing the overall storage footprint of a given application.
    • We no longer need a full OS stack for each app making the images very, very small.
  • Since there is no unnecessary duplication of services, the RAM and CPU footprint of a running app is significantly reduced as well.
  • With startup times in the hundreds of milliseconds, it is possible to run a zero footprint cloud. This means that there are no services running when there are no requests to be served. When a request is received, a unikernel is spun up to handle the request, hangs out for a bit to handle any others that may come in, and then shuts down within a few seconds. 

Security

  • The attack surface of a running application is significantly reduced when you remove the operating system. Now an attacker must rely on attack vectors in your application code and included libraries instead of the OS which is much more common across deployments.
    • We are now out of the script kiddie land and into the land of professional hackers.
  • The security patch problem becomes much smaller:
    • Not every single app server is running BASH etc..
  • There are no tools on the running instance to help in the attack. An attacker must bring along their own compiler, libraries to support it, and a shell to run it.
  • Type safety/checking makes it even more secure (OCaml).
  •  

Complexity

  • Removing the OS removes the bulk of complexity from a system.
  • The majority of the remaining complexity is moved to build/compile time where there is no risk of an outage.
  • The team that built the app knows it's exact requirements and how to meet them
    • No more black-box Linux stack
  • All development teams can now code against the same interface (virtualized drivers) creating re-usable libraries that can be shared across teams.

It's also the next logical step from containerization

Docker and other containerization techniques were a drastic improvement with regards to efficiency when compared to traditional VMs, but this is not the end-all be-all. Currently the shared nature of containers and the reliance on shared system libraries/kernels creates a few problems. Unikernels attempt to address the major faults of containers. Also there's security...

What are some of the challenges when adopting unikernels?

Feature teams have more responsibility..

  • No OS maintained by another team.
  • Deeper knowledge of what libraries are included and how to patch them.
  • More ownership through all environments.
  • On-call rotations.

..so give them a DevOps type

  • Easier to staff DevOps on a feature team.
  • More fun, less stress, working as a team to deliver a feature from dev to prod.
  • Transfers opsy type knowledge to the rest of the team.
  • Builds tools to support the feature/app.
  • Is part of a larger operations group.*

*see Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds

Configuration management is way different...

Current best practices* dictate building a single artifact that is then moved around through environments that have different configurations. By compiling configurations into the binary this flow no longer works.

Luckily this configuration check is happening in the build phase which is much less risky than if it were changing production systems!

...but much more powerful

The major benefit here is when using a compiler that supports compiling in configuration information (like OCaml), the compiler sees exactly what is being setup, checks to confirm it can compile, and removes anything that is left unconfigured, further reducing the actual code being deployed and thus the security footprint etc.. This also creates a unique footprint for each instance. When is the last time Puppet removed packages that were not in use?

* 12 Factor Apps

Cya backwards compatability!

This next evolution in architecture will not be a fork-lift. We have done that from mainframes to iron to the cloud. It is now time to re-architect our applications to make better and more efficient use of our limited resources.

Immutable infra is different.

Adopting the mentality that the system running will never change, but only be replaced is a hard shift in thought. It requires thinking about where the app requires persistence and making that happen which can be more difficult than just writing to local disk.

...but if we can do it

  • Development teams will have a much better understanding of the full application they are developing.
  • Operations teams can focus on tool building and escalations.
  • Business units will have a better understanding of the real cost of software development.
  • Companies can horizontally scale feature teams as each team is self-sustainable.
  • Infrastructure Engineering / Ops will not be the blocking resource!

What's out there right now?

(In alphabetical order)

  • Name: Clive
  • Description: Clive is a research project that takes a modified Go compiler and couples it with a runtime kernel that will allow the final artifact to be run against a hypervisor or bare metal blade. This is still certainly a research project, but it is a very, very interesting project.
  • Homepage: http://lsub.org/ls/clive.html
  • Project URL: git://git.lsub.org/go.git
  • Language support: Go
  • Hypervisor support: Unclear, but looks like they are targetting bare metal (which implies KVM and Xen)
  • License: MIT http://opensource.org/licenses/MIT
  • Cool links:
  • Name: Microsoft Drawbridge
  • Description: Drawbridge is a research prototype of a new form of virtualization for Microsoft application sandboxing. Drawbridge combines two core technologies: First, a picoprocess, which is a process-based isolation container with a minimal kernel API surface. Second, a library OS, which is a version of Windows enlightened to run efficiently within a picoprocess.
  • Homepage: http://research.microsoft.com/en-us/projects/drawbridge/
  • Project URL: No code available at this time
  • Language support: Windoze
  • Hypervisor support: Windoze
  • License: No license specified, but probably not BSD
  • Cool links:
  • Name: OSv
  • Description: OSv is a new open-source operating system for virtual machines. OSv was designed from the ground up to execute a single application on top of a hypervisor. OSv has new APIs for new applications, but also runs unmodified Linux applications (most of Linux's ABI is supported) and in particular can run an unmodified JVM, and applications built on top of one.
  • Homepage: http://osv.io/
  • Project URL: https://github.com/cloudius-systems/osv
  • Language support: Lots including the JVM (OSv runs unmodified Linux applications)
  • Hypervisor support: EC2, GCE
  • License: 3-clause BSD
  • Cool links:

..and many more to come!

Part 2: Building and deploying a LING unikernel to EC2

What are we going to be doing exactly?

  • Write (paste) an Erlang app in IntelliJ.
  • Compile and run the app in IntelliJ to test it.
  • Package the app as a Xen image with LING's railing.
  • Run the Xen image on Xen in Virtualbox to test.
  • Package the Xen image as an AMI.
  • Upload and register the AMI.
  • Deploy an EC2 instance from AMI.
  • ...
  • Profit.

And what will we be using to do it?

  • Erlang/OTP 17
  • IntelliJ IDEA & Erlang/OTP Plugin
  • railing: LING's packaging utility
  • rebar: Basho's Erlang build tool
  • Vagrant (to run Ubuntu
  • Ubuntu (to run Xen)
  • Xen (to run our LING artifact)
  • AWS CLI tools
  • AWS GUI

Thank you!

Interesting Links

05/01/2015

  1. My Github
  2. OSv ACEU14 (presentation)
  3. Unikernels: Rise of the Virtual Library Operating System
  4. OSv: Optimizing the Operating System for Virtual Machines
  5. Asplos Mirage Whitepaper
  6. PV-GRUB (The kernel loader we use with LING)
  7. 7Unikernel projects to take on Docker
  8. OCaml Labs Mirage Page
  9. Rise of the unikernel (excellent presentation)