Beyond multitenancy 
container-based application factory

 

Damien Metzler

dmetzler@nuxeo.com

https://github.com/dmetzler/

Thierry Delprat
tdelprat@nuxeo.com

https://github.com/tiry/

About This talK

This is our story / technical adventure

  • multi-tenancy with advanced per-tenant customization 
  • build a Software factory using VMs, Docker and now CaaS
     

but, hopefully, this should also make sense for

  • applications that are modular and configurable
  • applications moving to the cloud / SaaS model

2012/13 - IdeA & first try

Multi-Tenants, Cloud & Application Factory

Some Context

Nuxeo









 

About Me  

​ we provide a Platform that developers can use to
build highly customized Content Applications

we provide components, and the tools to assemble the
everything we do is open source

our customers are people building software
 in-house - software vendors - SaaS

Nuxeo Platform

Repository









 

Services

Workflows, Conversions, Diff, Notifications, Activity  ...

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Customer

1 Customer

More Customers

Customers

APPLICATION LEVEL MULTI-TENANTS 

Document Store 
Security
Life Cycle
Indexing
Versioning

all clients share the same application

application manages data and configuration partitionning

Application Level Multi-Tenants 

Shallow isolation

  • quota management is not efficient 
  • customization options are limited
     

Align all customers 

  • same version, same component set
  • same upgrade and maintenance policy
  • per-tenant Backup/Restore is not easy
     

Management

  • ​scale out is not that easy (i.e. move a tenant)
  • Heterogeneous deployment units
    ​              VM level / JVM level / App level 

X

250+ Extension Points
-> Big Pain / Risk

Container Level Multi-tenants

rely on infrastructure to provide tenants isolation

application does not need to be impacted
 

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer​

Some key advantages

Flexibility

  • safe, easy testing  & upgrade
  • no impact on custom code

Strong isolation

  • quota and QS management is easy

Management

  • backup / restore is easy

But it's really a different approach

Application Factory

Application Factory

Application Factory

Application Factory

Application Factory

Application Factory

Let's do this ...

... for our own use cases !

Nuxeo Trials Use Case

We build a platform and customization tools
 

We want people to be able to taste the full experience

  • choose their components
  • configure business rules
  • run the app they build

​ Online demo site
                          with full per-tenant customization

PaaS vs Automation

Early testing with CloudFoundry / OpenShift

  • Fast moving ground
  • Our app does not fit in Java PaaS

​Go with Deployment Automation on IaaS

  • ​Seems easier
  • Better match for sys-admin / devops

Some Challenges

Prospects do not pay

Prospects want to access all features

We want them to have a great experience​​  

free customers with high expectations !

Cost is a major concern !

Cutting Costs

EC2 Spot instances

 

 

​Passivation


 

​Leverage AWS Services

Early Results

 

A lot of moving parts

  • no clear deployment unit
  • lot of scripting

     

​Slow

  • provisioning is slow
  • passivation is not usable  

works but ...

DEAD END ?

We need lighter foot print

We need faster startup

Hope Comes from 

Some developers have started playing
with lightweight containers because
VMs are too fat & slow !

and Docker !

the developers 

[dev] Cool new stuff on cloud related techs

...

http://www.docker.io/ :

a command line tool for launching and managing
arbitrary processes using LXC.
Open Source project contributed by the dotcloud guys.

...

03/2013

2014 - Nuxeo.io v1

Build the Application Factory on Docker

Arken

First target is to power Nuxeo Trials

need a smooth UX
 

but also build a generic infrastructure

publish work as opensource
 

Assign a team of java developers

not system-administrators
 

Expect results in 3 months !

rebuild / refactor every 3 months 

Build Your Own Application  

Build Your Own Application  

Select
target Platform

Build Your Own Application  

Pick additional components

Build Your Own Application  

Build Your Own Application  

Build new components

Build Your Own Application  

Select custom Components

Build Your Own Application  

Define Application Model

Build Your Own Application  

Build Your Own Application  

Choose deployment

environment

Build Your Own Application  

Deploy & Run !

Nuxeo.io Factory

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

Principles

Docker containers !

Leverage AWS infrastructure 

Principles

Passivation 

  • Less than 5% of trials are active at a given time
  • Fast start/stop (no data)
  • High density hosting on AWS

Principles

Dual state orchestration

  • Expected vs Actual
  • Decoupling & Monitoring



     

Use a distributed registry 

 

CASTING

Containers infrastructure:
 Docker + CoreOS
 

Scheduler:
Fleet
 

Distributed registry: 
etcd

​​​Monitoring:

DataDog

CASTING

 

​The Containers:
 

Nuxeo Servers

 

Manager Application (AngularJS)

Passivator (Go Service)

ArkenCtl (Go Cmd)

Dynamic reverse proxy Gogeta

 

Big Picture

Big Picture

Big Picture

Big Picture

Deploy new tenant

Big Picture

Register new tenant

Big Picture

Deploy more tenants

Big Picture

Route request to Customer X

Big Picture

Route request to Customer X

PASSIVATE

ACTIVATE

ACTIVATE

ACTIVATE

ACTIVATE

started

Monitoring

Some Good Results

 

1000+ instances/month managed on 4 EC2 VMs (m3.2xlarge)

Production hosting for some Nuxeo based applications

Eventually stable


Docker and Go are really great

The vision is good ! 

All Good ?

Almost !

Miseries

  • CoreOS updates
  • btrfs fragmentation and IO issues
  • etcd stability
  • fleet "shortest path" scheduling
  • ...

Lots of moving parts + Lots of young solutions

Experience a new type of failure every day !

Structural Issues

Lot of boring glue code

  • Networking, Port Mapping, provisionning ...
  • ​developers are sick of shell scripts
     

Storage management is an issue

  • asymmetrical model & scalability
  • Security concerns on shared infrastructure

CoMplexity

 

Troubleshooting is tricky

Command line tools
 

Still too scary for a customer 

​the system is complex to setup

now - Nuxeo.io v2

Leverage our experience and Docker evolutions

Drivers

Docker ecosystem evolved

Kubernetes, ​Swarm, Compose, Rancher ...


 

We still believe in the initial vision

Customers starts to like the idea of Containers


 

We have learned a lot from Nuxeo.io v1 

​Time for a reboot

Additional contraints

Cluster configurations

1 node, 2 nodes, 7 nodes ...



 

Customer compliant 

avoid or hide and package
the glue code 

X

Leverage Docker evolution

Networking in Docker / Rancher / Kubernetes

  • ​No more "manual" port & DNS mapping
  • Use Software Defined Network​
s/port-mapping/SDN/g
s/Ansible/Swarm/g
s/Scripting/Compose/g

​Clustering in Docker-Swarm / Rancher / Kubernetes

  • No more scripting automation

Stack templating in Docker-Compose / Rancher / Kubernetes 

  • ​Manage an application as a set of containers​

​Free from the Shell !

​less glue code to write / debug

focus on application level

Leverage Docker evolution

Volumes in Docker

storage can now be provisionned as a container

​containers can now be statefull

Can provision Storage nodes exactly as Processing nodes

Streamline architecture:     everything is container

                         all tenant resources are provisioned the same way 

CaaS ?

 

  • Avoid building a CaaS infrastructure
     
  • Focus on Application Customization & Templating
     
  • Keep some flexibility on Cluster Orchestration


RANCHER
 

Caas Choice

Nice ​high level REST API

abstraction on CaaS provider 
Swarm/Kubernetes/Cattle
 

Administration UI

  • Infrastructure vs Application view
  • Monitoring and SSH access​
     

Close to Docker 

  • Built on top of Docker and Compose 
  • Contribute to Docker

CaaS Choice

 

Application Management 

  • Stack definition & templating
     

Provide additional features

  • Health Monitoring
  • ​Load Balancing and DNS
  • Rolling upgrades

​

Volume plugins :

convoy: NFS / GlusterFS

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Direct Gains

Rancher & Docker
do the heavy lifting




 

One unique API
to deploy new tenants

Arken Contents

Application Templating

Package Selection : Wizard + Config + Docker File 

Deployment template :  Compose + Rancher

 

Passivation Management

Passivation aware Routing

State management

API & Adapter

Independent

Go Based

OpenSource

multi-tenants ?

 

data isolation : separated data containers (docker)
 

processing isolation : separated containers (docker)
 

configuration : separated Nuxeo config + stack (compose + rancher) 
 

infrastructure isolation: separated environments (rancher)

 

Application is not even aware about tenants !

Tenants sharing an Application

Tenants sharing Infrastructure

Take aways

Software based multi-tenant application

  • lots of limitations                                                 

Docker

  • build and ship reusable executions units

Compose 

  • software defined infrastructure                   

Volume plugin

  • open the way to stateful containers           

Container based multi-tenancy  

  • per tenant customization                                 

 

Any Questions ?

Thank You !

http://www.nuxeo.com/careers/

We are hiring !

New York, Paris, Lisboa