- Who are we?
- What is HubSpot?
- HubSpot's architecture
- Load balancing at HubSpot
- An All-In-One marketing (and sales) platform
Stephen Salinas (firstname.lastname@example.org)
- Infrastructure Engineer on the HubSpot PaaS team
PaaS (Platform as a Service)
- Manage build and deploy
- Developer productivity
PaaS At HubSpot
- Build and deploy support for...
- 1484 Deployable artifacts
- 150+ Production deploys every day
- ~8900 Req/sec accross our load balancers
- 100+ developers
- 250+ web services / 200+ static apps
The HubSpot Deploy Infrastructure
- HubSpot's open source mesos scheduler
- Stores build and deploy data
- Deploy UI/API, talks to Singularity + Baragon
- Open source API for manipulating load balancers
The HubSpot Infrastructure
How Can We Improve?
- Decrease number of servers
(almost 200 at HubSpot)
- Speed up replacement
1) condense a new instance
2) configure instance
3) wait for puppet to run...
- Mesos is great for app reliability, why not load balancers?
1) Start new mesos task
- Efficient resource usage
(individual machines were using 50% or less of available memory)
How Do We Get There?
Main Problems to Solve
- How do I docker?
- Can we smush it all into an image?
- Restarts must be seamless
- Run nginx on a consistent port for the ELB
- Baragon knows about...
- Agent starting up
- Agent shutting down
- How long since agent was last seen as 'active'
- Keep instances with inactive Baragon agents out of the ELB, add new ones the moment they are ready
- The current 'state of the world'
Singularity and Docker
- Singularity custom mesos executor
- Embedded artifacts
- S3 download service
- thread count monitoring
- S3 upload service
**Can't get all this and use mesos docker containerizer
- Use mesos' docker containerizer?
- Containerizer vs Executor
- custom executor writes to files outside of the mesos
sandbox for the s3 uploader to use
Docker Support for Singularity Executor
- Container lifecycle
- Pull create start attach stop remove
- Map ENV, volumes, and ports
- Resource Monitoring
- Thread count
docker inspect to find the root pid
- CPU / memory usage
- mesos looks at cgroup of the executor
- manually moving processes around cgroups...
Docker 1.6 FTW!
use --cgroup-parent to run as a child of the executor cgroup
enable hierarchy in the executor cgroup for proper reporting
with upgraded registry, image pulls are super fast too!
So What's In This Image???
Yes, we like our Kaijū on the PaaS team...
- Lets mimic what the server looks like!
- yum install ...
- yum install ...
- yum install ...
- baragon app
- rodan app
Put That Image on a Diet...
Distributed (Inverse) Deploys
- Small python package already used in other places at HubSpot
- Watches the Deploy Service for new deploys
- Downloads artifacts and starts the app
- Don't need to store app artifacts in the image
- Can update the managing / monitoring apps without
unnecessarily interrupting nginx
- ELBs expect nginx to always be on the same port
- Mesos is more dynamic, assigns free port(s) within in a range
- Terminate SSL at the ELB (only 1 port needed for nginx)
- each 'group' is assigned a port outside the mesos range
- nginx also listens on mesos-assigned port for healthcheck
- baragon listens on mesos-assigned port
- ensure no two of same group run on the same slave
- Docker port mapping (EXPOSE / -P) would be consistent
inside the container, but not outside
Time To Profit!
- No more getting paged for hardware issues
- Remove (most) human intervention
- Extra LB capacity is one click away
- Fast replacement (10s of seconds vs 10s of minutes)
- Run mixed in with existing infrastructure
- Extra resources on big mesos slave machines
- Better utilization of available resources
Check It Out!
Baragon - github.com/HubSpot/Baragon
Working example of...
- mesos cluster with Singularity
- BaragonService + BaragonAgent/Nginx
- SingularityExecutor docker support is not officially released... yet...
...Or come talk to us!