About This talK

This is our technical adventure

• multi-tenancy with advanced per-tenant customization 
• build a Software factory using VMs, Docker and now CaaS
   

This is our story but, hopefully, this should also make sense for

• applications that are modular and configurable
• applications moving to the cloud / SaaS model

a long road

2012/2013 
 

2014
 

now 

initial idea and first tentative
 

first viable version

current development
and future architecture 

2012/13 - IdeA & first try

Multi-Tenants, Cloud & Application Factory

Some Context

Nuxeo









 

 

​ we provide a Platform that developers can use to
build highly customized Content Applications

we provide components, and the tools to assemble 
everything we do is open source

our customers are people building software
 in-house - software vendors - SaaS

Nuxeo Platform

Repository









 

Services

Workflows, Conversions, Diff, Notifications, Activity  ...

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Project Management

Building ON TOP of a Platform

Digital Assets Management

Building ON TOP of a Platform

Case Management

Building ON TOP of a Platform

Collaboration

Building ON TOP of a Platform

Customer

1 Customer

Custom deployment

More Customers

Industrialization

Customers

PLM Solution 

Customer/Tenant 1

Customer/Tenant 2

Styles
Screen Layouts
Meta-Data
Workflows
Business Logics

Connectors
 

Styles
Screen Layouts
Meta-Data
Workflows
Business Logics

Connectors
 

APPLICATION LEVEL MULTI-TENANTS 

all clients share the same application

application manages data and configuration partitionning

Application Level Multi-Tenants 

Shallow isolation

• quota management is not efficient 
• customization options are limited
   

Monolithic 

• same version, same component set
• same upgrade and maintenance policy
   

Not even simple

• ​scale out is not that easy (i.e. move a tenant)
• per-tenant Backup/Restore is not easy
• Heterogeneous deployment units
  ​          VM level / JVM level / App level 

Can not leverage OSGi / Extension Point model
 

Not "Cloud Native approach"
 

Container Level Multi-tenants

rely on infrastructure to provide tenants isolation

application does not need to be impacted
 

Application Factory

Create "on demand" application for each customer

• use Container level isolation
• provision infrastructure from the Cloud 
• custom assembly for each customer​

Application Factory

Create "on demand" application for each customer

• use Container level isolation
• provision infrastructure from the Cloud 
• custom assembly for each customer​

Application Factory

Create "on demand" application for each customer

• use Container level isolation
• provision infrastructure from the Cloud 
• custom assembly for each customer​

Application Factory

Create "on demand" application for each customer

• use Container level isolation
• provision infrastructure from the Cloud 
• custom assembly for each customer​

Application Factory

Create "on demand" application for each customer

• use Container level isolation
• provision infrastructure from the Cloud 
• custom assembly for each customer​

Let's do this ...

... for our own use cases !

Nuxeo Trials Use Case

We build a platform and customization tools
 

We want people to be able to taste the full experience

• choose their components
• configure business rules
• run the app they build

​ Online demo site
                          with full per-tenant customization

PaaS vs Automation

Some Challenges

Prospects do not pay

Prospects want to access all features

We want them to have a great experience​​  

free customers with high expectations !

Cost is a major concern   

Optimize hosting infrastructure

Cutting Costs

EC2 Spot instances

 

 

​Start/Stop instances when needed


 

 

​Leverage AWS Services

Early Results

 

A lot of moving parts

• no clear deployment unit
• lot of scripting
  
   

​Slow

• provisioning is slow
• start/stop too slow to be usable  

works but ...

DEAD END ?

We need lighter foot print

We need faster startup

Hope Comes from 

Some developers have started playing
with lightweight containers because
VMs are too fat & slow !

and Docker !

the developers 

[dev] Cool new stuff on cloud related techs

...

http://www.docker.io/ :

a command line tool for launching and managing
arbitrary processes using LXC.
Open Source project contributed by the dotcloud guys.

...

03/2013

2014 - Nuxeo.io v1

Build the Application Factory on Docker

Arken

First target is to power Nuxeo Trials

need a smooth UX
 

but also build a generic infrastructure

publish work as opensource
 

Assign a team of java developers

not system-administrators
 

Expect results in 3 months !

rebuild / refactor every 3 months 

Build Your Own Application  

Build Your Own Application  

Select
target Platform

Build Your Own Application  

Pick additional components

Build Your Own Application  

Build Your Own Application  

Build new components

Build Your Own Application  

Select custom Components

Build Your Own Application  

Define Application Model

Build Your Own Application  

Build Your Own Application  

Choose deployment

environment

Build Your Own Application  

Deploy & Run !

Make your own container recipe

• choose your ingredients
• assemble
• run 

 

Available to the web user

 

Build Your Own Application  

Nuxeo.io Factory

Principles

Docker containers !

Leverage AWS infrastructure 

Principles

Passivation 

• Less than 5% of trials are active at a given time
• Fast start/stop (no data)
• High density hosting on AWS

Principles

Dual state orchestration

• Expected vs Actual
• Decoupling & Monitoring
  
  
  
   

Use a distributed registry 

 

CASTING

CASTING

CASTING

CASTING

 

​The Containers:
 

Big Picture

Big Picture

Big Picture

Big Picture

Deploy new tenant

Big Picture

Register new tenant

Big Picture

Deploy more tenants

Big Picture

Route request to Customer X

Big Picture

Route request to Customer X

PASSIVATE

ACTIVATE

ACTIVATE

ACTIVATE

ACTIVATE

started

Monitoring

Some Good Results

 

1000+ instances/month managed on 4 EC2 VMs (m3.2xlarge)

Production hosting for some Nuxeo based applications

Eventually stable


Docker and Go are really great

The vision is good ! 

All Good ?

Almost !

Miseries

• CoreOS updates
• btrfs fragmentation and IO issues
• etcd stability
• fleet "shortest path" scheduling
• ...

Lots of moving parts + Lots of young solutions

Experience a new type of failure every day !

Structural Issues

Lot of boring glue code

• Networking, Port Mapping, provisionning ...
• ​developers are sick of shell scripts
   

Storage management is an issue

• asymmetrical model & scalability
• Security concerns on shared infrastructure

CoMplexity

 

Troubleshooting is tricky

Command line tools
 

Still too scary for a customer 

​the system is complex to setup

now - Nuxeo.io v2

Leverage our experience and Docker evolutions

Drivers

Docker ecosystem evolved

Kubernetes, ​Swarm, Compose, Rancher ...

 

We believe in the initial vision is good

Customers starts to like the idea of Containers

 

We have learned a lot from Nuxeo.io v1 

​Time for a reboot

Additional contraints

Cluster configurations

1 node, 2 nodes, 7 nodes ...


Production hosting

take over Nuxeo-Cloud
 

Customer compliant 

avoid or hide and package
the glue code 

X

Remove as much code as we can

Leverage Docker evolution

Networking in Docker / Rancher / Kubernetes

• ​No more "manual" port & DNS mapping
• Use Software Defined Network​

Leverage Docker evolution

Volumes in Docker

storage can now be provisionned as a container

​containers can now be statefull

CaaS ?

 

• Avoid building a CaaS infrastructure
   
• Focus on Application Customization & Templating
   
• Keep some flexibility on Cluster Orchestration

Caas Choice

Nice ​high level REST API

abstraction on CaaS provider 
Swarm/Kubernetes/Cattle
 

Administration UI

• Infrastructure vs Application view
• Monitoring and SSH access​
   

Close to Docker 

• Built on top of Docker and Compose 
• Contribute to Docker

CaaS Choice

 

Application Management 

• Stack definition & templating
   

Provide additional features

• Health Monitoring
• ​Load Balancing and DNS
• Rolling upgrades

​

Volume plugins :

convoy: NFS / GlusterFS

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher - DEMO

Direct Gains

Rancher & Docker
do the heavy lifting

 

Deploying Arken is easy


 

One unique API
to deploy new tenants

Arken Contents

Application Templating

Package Selection : Wizard + Config + Docker File 

Deployment template :  Compose + Rancher

 

Passivation Management

Passivation aware Routing

State management

API & Adapter

multi-tenants ?

multi-tenants ?

Breaking the Monolith ?