Scaling a MeteorJS SaaS app on AWS

Brett McLain

Application Systems Architect

PotashCorp

What should you get out of this talk?

• A better understanding of MeteorJS (architecture, deployment tools, clustering).
• A better understanding of Amazon Web Services (EC2, ELB, OpsWorks, CloudFormation, Route 53, Auto Scaling).
• How to deploy, cluster, and scale a MeteorJS web app.

Application Overview

• SaaS application that uses Twilio to send SMS and phone calls to devices that connect to the cell phone network.

• MeteorJS (runs on Node.js)

• MongoDB

• Flot Charts

• Twitter Bootstrap

• Blah blah blah.

What is MeteorJS?

• "Meteor is a full-stack JavaScript platform for developing modern web and mobile applications."
• Out of the box tight integration with MongoDB + full data reactivity.
• Install MeteorJS and have an app written in minutes with well established and easily understood patterns.
• Hot code push.
• Full data reactivity.
• Easily build for Android and iPhone (hot code push).

Meteor Architecture

• Production deployments are NodeJS applications.

• Meteor has it’s own package manager called isobuild. Extremely similar to NPM.

• Meteor can also use NPM packages now on the client or server (as of version 1.3).

• Supports ECMAScript 6 (ES 2015).

• Meteor Up (mup) - Meteor deployment plugin.

• Meteor Cluster - Meteor clustering plugin.

Meteor Deployments

• I use Meteor Up (mup) for deployments.

• Single command to build and deploy code to an unlimited number of servers in a rolling fashion.

• Installs all necessary software automatically (NodeJS, npm, MeteorJS, docker, nginx, mongo).

• Setups up an nginx reverse proxy in front of a docker instance for meteor and a docker instance for mongo.

• 10 seconds of downtime per instance!

{
  // Server authentication info
  "servers": [
    {
      "host": "localhost",
      "username": "ubuntu",
      "pem": "~/RelaySupply.pem"
    }
  ],
  // Install MongoDB in the server, does not destroy local MongoDB on future setup
  "setupMongo": false,

  // WARNING: Node.js is required! Only skip if you already have Node.js installed on server.
  "setupNode": true,

  // WARNING: If nodeVersion omitted will setup 0.10.36 by default. Do not use v, only version number.
  "nodeVersion": "0.10.40",

  // Application name (No spaces)
  "appName": "RelaySupplyControlPanel",

  // Configure environment
  "env": {
    "PORT": 80,
    "ROOT_URL": "https://controlpanel.relaysupply.com",
    "MONGO_URL": "mongodb://10.0.2.151:27017,10.0.3.138:27017,10.0.4.44:27017/relaysupply-prod",
    "MONGO_OPLOG_URL": "mongodb://10.0.2.151:27017,10.0.3.138:27017,10.0.4.44:27017/local",
    "DISABLE_WEBSOCKETS": "0",
    "CLUSTER_BALANCER_URL": "REPLACE_ME",
    "CLUSTER_DISCOVERY_URL": "mongodb://10.0.2.151:27017/relaysupply-prod",
    "CLUSTER_SERVICE": "web",
  },

  // Meteor Up checks if the app comes online just after the deployment
  // before mup checks that, it will wait for no. of seconds configured below
  "deployCheckWaitTime": 60,

  "ssl": {
    "certificate": "./.prod/fullchain.pem",
    "key": "./.prod/privkey.pem",
    "port": 443
  }
}

Meteor Cluster

• Provides clustering capability within Meteor JS apps.

• Meteor Cluster is a regular meteor package.

• Turns each meteor server into a load balancer or a DDP instance (backend server).
• New nodes register with MongoDB backend and are auto-discovered. Traffic begins routing to them immediately.

Round Robin DNS Load Balancing

1 Load Balancer, 2 DDP Servers

2 Load Balancers, 2 DDP Servers

Meteor Clustering

• Load balancer servers are pointed to by DNS records.

• Once the client connects to a load balancer server, it *might* receive the address of a DDP server to share out the load.

• Supports clustering of microservices too!

So...now what?

• Meteor has some great deployment and clustering tools, but how do we deploy it in a scalable way?
• We need to pick a host that fits well with our deployment tools and utilize the clustering capabilities of Meteor.

Initial Objectives

• Low Cost!

• Availability

• Performance

• Low Maintenance

Where to start?

• Originally hosted everything (prod, dev, mongo, reverse proxy) on a single Virtual Private Server (VPS).

• I have past experience with Amazon, it’s well documented, tons of tools.

• Google *appears* pricier, less tools.

• Digital Ocean…?

So what does Amazon offer?

• EC2 (Elastic Compute Cloud)

• Elastic Bean Stalk

• Elastic Load Balancers

• Route 53 (DNS)

• OpsWorks

• CloudFormation

• Elastic Block Store

• The list goes on!

Elastic BeanStalk

• Elastic Beanstalk is a Platform as a Service (PaaS) that handles the complexities of your underlying infrastructure.

• Manages load balancing (via ELBs), instance scaling, instance health monitoring, and deployments.

• Supports Docker, Go, Java, .NET, Node, PHP, Python, Ruby.

Let's see how this goes...

• Elastic Beanstalk seemed like a great solution. Handles scaling and load balancing (with ELB).
• Ran into problems very quickly...
• MeteorJS settings are a JSON file exported as an environment variable. Beanstalk strips environment variables of a number of special characters!
• Quickly realized that it is not possible to do websockets, sticky sessions, and HTTPS with Elastic Load Balancers. 

How about OpsWorks?

• OpsWorks is a configuration management service that helps you configure and operate applications of all shapes and sizes using Chef.
• You can define the application’s architecture and the specification of each component including package installation, software configuration and resources such as storage.

Nope.

• Stood up a MongoDB cluster and two separate Meteor clusters.
• Autoscaling and application stack health-checks are nice!
• Didn't bother using Chef as I have my own deployment tools.
• Ran into problems very quickly...
• "running_setup" never ended. Servers would hang. No support from Amazon!

Ok, uh, what else is left? CloudFormation?

• CloudFormation allows you to define infrastructure and services as a configuration template.
• Provision servers, storage, networks, NATs, relationships, run scripts, etc. all from a JSON config.
• Rapid deployment of an entire stack, extremely powerful and _extremely_ complicated.

CloudFormation

• I used the MongoDB CloudFormation template provided by Amazon (only on github, not via the CloudFormation interface).
• Over time I abandoned it as the scripts they used for deployment contained a race condition that would cause deployment to fail about 50% of the time.
• For my purposes, this was overkill.

So, EC2 then?

• Setup 3 layers: (1) Load Balancers, (1) DDP Servers, (3) Mongo Replica Sets (no shards).
• Created 5 DNS records for the load balancers to do round robin DNS.
• Used Route 53 latency weighted routing to failover load balancers that are offline to load balancers that are online.
• 2 auto scaling groups, one for load balancers, one for DDP servers.
• Lets Encrypt for SSL/TLS.

Route 53

• Route 53 is a highly available and scalable DNS.
• Routing policies are:
  • Simple Routing
  • Weighted Routing
  • Latency Routing
  • Failover Routing

Down the rabbit hole...

• Auto scaling occurs when average CPU > 70% for 2 minutes.

• When auto scaling occurs in either auto scaling group, it creates a new instance using the same image.

• That image runs a script on boot...

do_all_the_things.sh

• Step 1: git pull to get latest code.
• Step 2: Determine if an elastic IP address is already assigned (if not, then this is a fresh instance).
• Step 3a: If fresh instance, and DDP server, use auto-assigned public IP.
• Step 3b: If fresh instance, and load balancer, check to see which DNS elastic IP's are unused and assign one.
• Deploy to self (build code and deploy to localhost).
• Step 4: Done!

Testing Methodology

• Used MeteorDown for load testing. It connects to an endpoint, opens up a DDP connect and completes some action (in this case a 200kb document transfer) before closing the connection.
• Scale up connections incrementally and monitor when and how fast instances are created.
• Determine total capability of 5 load balancers and 10 DDP servers (each instance is t2.micro, 1 vCPU, 512mb ram).

Testing Results

• First auto scaling event: provisioning of instance, boot, git pull, build, and deploy is 2 minutes 30 seconds. 
• Second+ auto scaling event: 1 minute 10 seconds.
• Concurrent subscriptions achieved: 38,000
• Maximum subscriptions per instance is reduced by approximately 1% per added instance.

What I learned...

• Meteor has very powerful tools out of the box, however these tools do not lend themselves well to solutions such as Elastic Beanstalk or Google App Engine.
• DNS is a finicky mistress.
• Scaling load balancers is hard; scaling a web app using sticky websockets is even harder!
• Diving in head first has some serious drawbacks - it's difficult to navigate this space without someone to guide you.

What did I end up with?

• One line command to do rolling deployments to all my servers.
• 10 second downtime when deploying.
• Highly available (across AZ's), scalable architecture (auto-scaling + MongoDB sharding/replicas).
• 38,000 concurrent connections constantly re-establishing websockets and sending data.
• Great support for microservices.
• Reactive data and UI, hot code push, mobile deployment.

What was the point of this?

• Hopefully you understand that it IS possible to scale MeteorJS applications, and that it isn't that hard (I just made it hard for myself).
• High level understanding of a few of Amazon's infrastructure and deployment services.