- Who are we?

- What is HubSpot?

- HubSpot's architecture

- Load balancing at HubSpot

Before...

And after...

HubSpot

- An All-In-One marketing (and sales) platform

PaaS (Platform as a Service)

- Manage build and deploy

- Developer productivity

- 1484 Deployable artifacts

- 150+ Production deploys every day

- ~8900 Req/sec accross our load balancers

- 100+ developers

- 250+ web services / 200+ static apps

Singularity

- HubSpot's open source mesos scheduler

Deploy Service

- Stores build and deploy data

Orion

- Deploy UI/API, talks to Singularity + Baragon

Baragon

- Open source API for manipulating load balancers

app.hubspot.com/hello

app.hubspot.com

/hello

'app'

- /hello

- app

- host:port

- Decrease number of servers

(almost 200 at HubSpot)

- Speed up replacement

1) condense a new instance

2) configure instance

3) wait for puppet to run...

...

-

- Mesos is great for app reliability, why not load balancers?

VS

1) Start new mesos task

- Efficient resource usage

(individual machines were using 50% or less of available memory)

- How do I docker?

- Can we smush it all into an image?

- Restarts must be seamless

- Run nginx on a consistent port for the ELB

- Singularity custom mesos executor

- Embedded artifacts

- S3 download service

- thread count monitoring

- logrotate

- S3 upload service

**Can't get all this and use mesos docker containerizer

- Use mesos' docker containerizer?

- Containerizer vs Executor

- custom executor writes to files outside of the mesos

  sandbox for the s3 uploader to use

Slave Machine

Containerizer

Executor

- Container lifecycle

- Pull    create    start    attach    stop    remove

- Map ENV, volumes, and ports

- Resource Monitoring

- Thread count

   docker inspect to find the root pid

- CPU / memory usage

- mesos looks at cgroup of the executor

- manually moving processes around cgroups...

use --cgroup-parent to run as a child of the executor cgroup

enable hierarchy in the executor cgroup for proper reporting

with upgraded registry, image pulls are super fast too!

functionality

management

monitoring

Yes, we like our Kaijū on the PaaS team...

- Lets mimic what the server looks like!

- nginx

- java

- python

- yum install ...

- yum install ...

- yum install ...

2GB+ image...

- baragon app

- rodan app

- configuration

Templating start script

distributed deploy  updater

~450MB !

Minimal library installs

- Small python package already used in other places at HubSpot

- Watches the Deploy Service       for new deploys

- Downloads artifacts and starts the app

- Don't need to store app artifacts in the image

- Can update the managing / monitoring apps without 

   unnecessarily interrupting nginx

- ELBs expect nginx to always be on the same port

- Mesos is more dynamic, assigns free port(s) within in a range

- Strategy

- Terminate SSL at the ELB (only 1 port needed for nginx)

- each 'group' is assigned a port outside the mesos range

- nginx also listens on mesos-assigned port for healthcheck

- baragon listens on mesos-assigned port

- ensure no two of same group run on the same slave

- Docker port mapping (EXPOSE / -P) would be consistent

  inside the container, but not outside

- Reliability

- No more getting paged for hardware issues

- Remove (most) human intervention

- Scalability

- Extra LB capacity is one click away

- Fast replacement (10s of seconds vs 10s of minutes)

- Cost

- Run mixed in with existing infrastructure

- Extra resources on big mesos slave machines

- Better utilization of available resources