Beyond multitenancy 
container-based application factory

 

Damien Metzler

dmetzler@nuxeo.com

https://github.com/dmetzler/

Thierry Delprat
tdelprat@nuxeo.com

https://github.com/tiry/

About This talK

This is our technical adventure

  • multi-tenancy with advanced per-tenant customization 
  • build a Software factory using VMs, Docker and now CaaS
     

This is our story but, hopefully, this should also make sense for

  • applications that are modular and configurable
  • applications moving to the cloud / SaaS model

a long road

2012/2013 
 

2014
 

now 

initial idea and first tentative
 

first viable version


current development
and future architecture 

2012/13 - IdeA & first try

Multi-Tenants, Cloud & Application Factory

Some Context

Nuxeo









 

 

we provide a Platform that developers can use to
build highly customized Content Applications

we provide components, and the tools to assemble 
everything we do is open source

our customers are people building software
 in-house - software vendors - SaaS

Nuxeo Platform

Repository









 

Services

Workflows, Conversions, Diff, Notifications, Activity  ...

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Building ON TOP of a Platform

Project Management

Building ON TOP of a Platform

Digital Assets Management

Building ON TOP of a Platform

Case Management

Building ON TOP of a Platform

Collaboration

Building ON TOP of a Platform

Customer

1 Customer

Custom deployment

More Customers

Industrialization

Customers

PLM Solution 

Customer/Tenant 1

Customer/Tenant 2

Styles
Screen Layouts
Meta-Data
Workflows
Business Logics

Connectors
 

Styles
Screen Layouts
Meta-Data
Workflows
Business Logics

Connectors
 

APPLICATION LEVEL MULTI-TENANTS 

Document Store 
Security
Life Cycle
Indexing
Versioning

all clients share the same application

application manages data and configuration partitionning

Application Level Multi-Tenants 

Shallow isolation

  • quota management is not efficient 
  • customization options are limited
     

Monolithic 

  • same version, same component set
  • same upgrade and maintenance policy
     

Not even simple

  • scale out is not that easy (i.e. move a tenant)
  • per-tenant Backup/Restore is not easy
  • Heterogeneous deployment units
    ​          
    VM level / JVM level / App level 

Can not leverage OSGi / Extension Point model
 

Not "Cloud Native approach"
 

Container Level Multi-tenants

rely on infrastructure to provide tenants isolation

application does not need to be impacted
 

Flexible
Unlimited Customization
Full Isolation & Quotas

Application Factory

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer

Application Factory

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer

Application Factory

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer

Application Factory

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer

Application Factory

Create "on demand" application for each customer

  • use Container level isolation
  • provision infrastructure from the Cloud 
  • custom assembly for each customer

Let's do this ...

... for our own use cases !

Nuxeo Trials Use Case

We build a platform and customization tools
 

We want people to be able to taste the full experience

  • choose their components
  • configure business rules
  • run the app they build

​ Online demo site
                          with full per-tenant customization

PaaS vs Automation

Early testing with CloudFoundry / OpenShift

  • Fast moving ground
  • Very opinionated
  • Our app does not fit in Java PaaS

​Go with Deployment Automation on IaaS

  • ​Seems easier
  • Better match for sys-admin / devops

Some Challenges

Prospects do not pay

Prospects want to access all features

We want them to have a great experience​​  

free customers with high expectations !

Cost is a major concern   

Optimize hosting infrastructure

Cutting Costs

EC2 Spot instances

 

 

​Start/Stop instances when needed


 

 

​Leverage AWS Services

Early Results

 

A lot of moving parts

  • no clear deployment unit
  • lot of scripting

     

​Slow

  • provisioning is slow
  • start/stop too slow to be usable  

works but ...

DEAD END ?

We need lighter foot print

We need faster startup

Hope Comes from 

Some developers have started playing
with lightweight containers because
VMs are too fat & slow !

and Docker !

the developers 

[dev] Cool new stuff on cloud related techs

...

http://www.docker.io/ :

a command line tool for launching and managing
arbitrary processes using LXC.
Open Source project contributed by the dotcloud guys.

...

03/2013

2014 - Nuxeo.io v1

Build the Application Factory on Docker

Arken

First target is to power Nuxeo Trials

need a smooth UX
 

but also build a generic infrastructure

publish work as opensource
 

Assign a team of java developers

not system-administrators
 

Expect results in 3 months !

rebuild / refactor every 3 months 

Build Your Own Application  

Build Your Own Application  

Select
target Platform

Build Your Own Application  

Pick additional components

Build Your Own Application  

Build Your Own Application  

Build new components

Build Your Own Application  

Select custom Components

Build Your Own Application  

Define Application Model

Build Your Own Application  

Build Your Own Application  

Choose deployment

environment

Build Your Own Application  

Deploy & Run !

Make your own container recipe

  • choose your ingredients
  • assemble
  • run 

 

Available to the web user

 

Build Your Own Application  

Nuxeo.io Factory

Principles

Docker containers !

Leverage AWS infrastructure 

Principles

Passivation 

  • Less than 5% of trials are active at a given time
  • Fast start/stop (no data)
  • High density hosting on AWS

Principles

Dual state orchestration

  • Expected vs Actual
  • Decoupling & Monitoring



     

Use a distributed registry 

 

CASTING

Containers infrastructure:
 Docker + CoreOS
 

Scheduler:
Fleet
 

Distributed registry: 
etcd

​Monitoring:

DataDog

CASTING

CASTING

CASTING

 

The Containers:
 

Nuxeo Servers

 

Manager.io (Nuxeo +AngularJS)

Passivator (Go Service)

ArkenCtl (Go Cmd)

Dynamic reverse proxy Gogeta

 

(1 per tenant)

Big Picture

Big Picture

Big Picture

Big Picture

Deploy new tenant

Big Picture

Register new tenant

Big Picture

Deploy more tenants

Big Picture

Route request to Customer X

Big Picture

Route request to Customer X

PASSIVATE

ACTIVATE

ACTIVATE

ACTIVATE

ACTIVATE

started

Monitoring

Some Good Results

 

1000+ instances/month managed on 4 EC2 VMs (m3.2xlarge)

Production hosting for some Nuxeo based applications

Eventually stable


Docker and Go are really great

The vision is good ! 

All Good ?

Almost !

Miseries

  • CoreOS updates
  • btrfs fragmentation and IO issues
  • etcd stability
  • fleet "shortest path" scheduling
  • ...

Lots of moving parts + Lots of young solutions

Experience a new type of failure every day !

Structural Issues

Lot of boring glue code

  • Networking, Port Mapping, provisionning ...
  • developers are sick of shell scripts
     

Storage management is an issue

  • asymmetrical model & scalability
  • Security concerns on shared infrastructure

CoMplexity

 

Troubleshooting is tricky

Command line tools
 

Still too scary for a customer 

the system is complex to setup

now - Nuxeo.io v2

Leverage our experience and Docker evolutions

Drivers

Docker ecosystem evolved

Kubernetes, ​Swarm, Compose, Rancher ...


 

We believe in the initial vision is good

Customers starts to like the idea of Containers


 

We have learned a lot from Nuxeo.io v1 

​Time for a reboot

Additional contraints

Cluster configurations

1 node, 2 nodes, 7 nodes ...


Production hosting

take over Nuxeo-Cloud
 

Customer compliant 

avoid or hide and package
the glue code 

X

Remove as much code as we can

Leverage Docker evolution

Networking in Docker / Rancher / Kubernetes

  • No more "manual" port & DNS mapping
  • Use Software Defined Network​
s/port-mapping/SDN/g
s/Ansible/Swarm/g
s/Scripting/Compose/g

Clustering in Docker-Swarm / Rancher / Kubernetes

  • No more scripting automation

Stack templating in Docker-Compose / Rancher / Kubernetes 

  • ​Manage an application as a set of containers

​Free from the Shell !

less glue code to write / debug

focus on application level

Leverage Docker evolution

Volumes in Docker

storage can now be provisionned as a container

containers can now be statefull

Can provision Storage nodes exactly as Processing nodes

Streamline architecture:     everything is container

                         all tenant resources are provisioned the same way 

CaaS ?

 

  • Avoid building a CaaS infrastructure
     
  • Focus on Application Customization & Templating
     
  • Keep some flexibility on Cluster Orchestration


RANCHER
 

Caas Choice

Nice high level REST API

abstraction on CaaS provider 
Swarm/Kubernetes/Cattle

 

Administration UI

  • Infrastructure vs Application view
  • Monitoring and SSH access
     

Close to Docker 

  • Built on top of Docker and Compose 
  • Contribute to Docker

CaaS Choice

 

Application Management 

  • Stack definition & templating
     

Provide additional features

  • Health Monitoring
  • ​Load Balancing and DNS
  • Rolling upgrades

Volume plugins :

convoy: NFS / GlusterFS

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher

Nuxeo.io with Rancher - DEMO

Direct Gains

Rancher & Docker
do the heavy lifting

 


Deploying Arken is easy


 

One unique API
to deploy new tenants

Arken Contents

Application Templating

Package Selection : Wizard + Config + Docker File 

Deployment template :  Compose + Rancher

 

Passivation Management

Passivation aware Routing

State management

API & Adapter

Independent

Go Based

OpenSource

multi-tenants ?

Application Level Multi-Tenancy 

  • data only isolation

Container Level Multi-Tenancy - Nuxeo.io v1 

  • processing isolation - shared data sources         

Container Level Multi-Tenancy - Nuxeo.io v2

  • complete data and process isolation

multi-tenants ?

Tenants sharing an Application

Tenants sharing Infrastructure
+ dedicated applications

processing isolation : docker containers

  • reusable execution units

data isolation : docker and volumes 

  • stateful containers         

application configuration: Rancher + Arken + Nuxeo

  • software defined tenant

Application Factory: Rancher + Arken

Breaking the Monolith ?

Stack of OSGi Services     =   MicroServices

Break deployment by tenants

Break deployment by containers

Deploy different service stacks          Different containers

Converters, Async Processing, IDE , Account management ...

Leverage Arken architecture

Introduce new granularity level

Any Questions ?

Thank You !

http://www.nuxeo.com/careers/

We are hiring !

New York, Paris, Lisboa

Beyond Multitenancy

By Thierry Delprat

Beyond Multitenancy

O'Reilly Software Architecture 2016

  • 3,213
Loading comments...

More from Thierry Delprat