2012/13 - IdeA & first try

Multi-Tenants, Cloud & Application Factory

Some Context

Nuxeo









 

About Me  

​ we provide a Platform that developers can use to
build highly customized Content Applications

we provide components, and the tools to assemble the
everything we do is open source

our customers are people building software
 in-house - software vendors - SaaS

Nuxeo Platform

Repository









 

Services

Workflows, Conversions, Diff, Notifications, Activity  ...

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Customer

1 Customer

More Customers

Customers

APPLICATION LEVEL MULTI-TENANTS 

all clients share the same application

application manages data and configuration partitionning

Application Level Multi-Tenants 

Shallow isolation

• quota management is not efficient 
• customization options are limited
   

Align all customers 

• same version, same component set
• same upgrade and maintenance policy
• per-tenant Backup/Restore is not easy
   

Management

• ​scale out is not that easy (i.e. move a tenant)
• Heterogeneous deployment units
  ​          VM level / JVM level / App level 

X

Container Level Multi-tenants

rely on infrastructure to provide tenants isolation

application does not need to be impacted
 

Some key advantages

Flexibility

• safe, easy testing  & upgrade
• no impact on custom code

Strong isolation

• quota and QS management is easy

Management

• backup / restore is easy

Application Factory

Application Factory

Application Factory

Application Factory

Application Factory

Let's do this ...

... for our own use cases !

Nuxeo Trials Use Case

We build a platform and customization tools
 

We want people to be able to taste the full experience

• choose their components
• configure business rules
• run the app they build

​ Online demo site
                          with full per-tenant customization

PaaS vs Automation

Some Challenges

Prospects do not pay

Prospects want to access all features

We want them to have a great experience​​  

free customers with high expectations !

Cost is a major concern !

Cutting Costs

EC2 Spot instances

 

 

​Passivation


 

​Leverage AWS Services

Early Results

 

A lot of moving parts

• no clear deployment unit
• lot of scripting
  
   

​Slow

• provisioning is slow
• passivation is not usable  

works but ...

DEAD END ?

We need lighter foot print

We need faster startup

Hope Comes from 

Some developers have started playing
with lightweight containers because
VMs are too fat & slow !

and Docker !

the developers 

[dev] Cool new stuff on cloud related techs

...

http://www.docker.io/ :

a command line tool for launching and managing
arbitrary processes using LXC.
Open Source project contributed by the dotcloud guys.

...

03/2013

2014 - Nuxeo.io v1

Build the Application Factory on Docker

Arken

First target is to power Nuxeo Trials

need a smooth UX
 

but also build a generic infrastructure

publish work as opensource
 

Assign a team of java developers

not system-administrators
 

Expect results in 3 months !

rebuild / refactor every 3 months 

Build Your Own Application  

Build Your Own Application  

Select
target Platform

Build Your Own Application  

Pick additional components

Build Your Own Application  

Build Your Own Application  

Build new components

Build Your Own Application  

Select custom Components

Build Your Own Application  

Define Application Model

Build Your Own Application  

Build Your Own Application  

Choose deployment

environment

Build Your Own Application  

Deploy & Run !

Nuxeo.io Factory

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

NUXEO.IO FACTORY

Principles

Docker containers !

Leverage AWS infrastructure 

Principles

Passivation 

• Less than 5% of trials are active at a given time
• Fast start/stop (no data)
• High density hosting on AWS

Principles

Dual state orchestration

• Expected vs Actual
• Decoupling & Monitoring
  
  
  
   

Use a distributed registry 

 

CASTING

CASTING

 

​The Containers:
 

Big Picture

Big Picture

Big Picture

Big Picture

Deploy new tenant

Big Picture

Register new tenant

Big Picture

Deploy more tenants

Big Picture

Route request to Customer X

Big Picture

Route request to Customer X

PASSIVATE

ACTIVATE

ACTIVATE

ACTIVATE

ACTIVATE

started

Monitoring

Some Good Results

 

1000+ instances/month managed on 4 EC2 VMs (m3.2xlarge)

Production hosting for some Nuxeo based applications

Eventually stable


Docker and Go are really great

The vision is good ! 

All Good ?

Almost !

Miseries

• CoreOS updates
• btrfs fragmentation and IO issues
• etcd stability
• fleet "shortest path" scheduling
• ...

Lots of moving parts + Lots of young solutions

Experience a new type of failure every day !

Structural Issues

Lot of boring glue code

• Networking, Port Mapping, provisionning ...
• ​developers are sick of shell scripts
   

Storage management is an issue

• asymmetrical model & scalability
• Security concerns on shared infrastructure

CoMplexity

 

Troubleshooting is tricky

Command line tools
 

Still too scary for a customer 

​the system is complex to setup

now - Nuxeo.io v2

Leverage our experience and Docker evolutions

Drivers

Docker ecosystem evolved

Kubernetes, ​Swarm, Compose, Rancher ...

 

We still believe in the initial vision

Customers starts to like the idea of Containers

 

We have learned a lot from Nuxeo.io v1 

​Time for a reboot

Additional contraints

Cluster configurations

1 node, 2 nodes, 7 nodes ...



 

Customer compliant 

avoid or hide and package
the glue code 

X

Leverage Docker evolution

Networking in Docker / Rancher / Kubernetes

• ​No more "manual" port & DNS mapping
• Use Software Defined Network​

Leverage Docker evolution

Volumes in Docker

storage can now be provisionned as a container

​containers can now be statefull

CaaS ?

 

• Avoid building a CaaS infrastructure
   
• Focus on Application Customization & Templating
   
• Keep some flexibility on Cluster Orchestration

Caas Choice

Nice ​high level REST API

abstraction on CaaS provider 
Swarm/Kubernetes/Cattle
 

Administration UI

• Infrastructure vs Application view
• Monitoring and SSH access​
   

Close to Docker 

• Built on top of Docker and Compose 
• Contribute to Docker

CaaS Choice

 

Application Management 

• Stack definition & templating
   

Provide additional features

• Health Monitoring
• ​Load Balancing and DNS
• Rolling upgrades

​

Volume plugins :

convoy: NFS / GlusterFS

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Direct Gains

Rancher & Docker
do the heavy lifting




 

One unique API
to deploy new tenants

Arken Contents

Application Templating

Package Selection : Wizard + Config + Docker File 

Deployment template :  Compose + Rancher

 

Passivation Management

Passivation aware Routing

State management

API & Adapter

multi-tenants ?

Take aways

Software based multi-tenant application

• lots of limitations                                                 

Docker

• build and ship reusable executions units

Compose 

• software defined infrastructure                   

Volume plugin

• open the way to stateful containers           

Container based multi-tenancy  

• per tenant customization