Scaling Facebook Analytics Infrastructure with MongoDB & node.js

• Page-level data published once a day at 2 AM PT, and varies between 2-7 days behind current date
• Post-level data published every 10-15 minutes
• Polling is the only option
• API limits are roughly 600 calls per 600 seconds per API token

• Monolithic Rails App
• MongoDB 1.8 on MongoHQ cloud
• queue_classic for jobs
• TONS of workers to pull down Page data
• 1 worker per page at a time
• ~1,000 Pages per hour

• Little visibility into errors or problems (our fault)
• Just ~1000 Pages per hour (Ruby's fault)
• Data structure: KISS principal
• Dots in keys (our fault)

 {
    fb_page_id: '...',
	metric: 'page_fans',
	period: 'lifetime',
	data: {
		'2013-01-01': 0,
		'2013-01-02': 3,
		...
	}
}

But it got the job done.

• Collect, store and update data in "real-time", despite the lack of a true real-time API
• Lots of data & lots of calls (posts, likes, comments, shares)
• Run custom logic for derived metrics

• New sync system in node.js, complete rewrite
• New durable queuing system (home grown)
• Queue introspection
• Error Visibility
• Operation Statistics
• Retry to deal with 10-15% error rate from Facebook
• Heroku auto-scaling (homegrown)
• Write / Upsert-only methodology

• Loads of concurrency
• 1,000 Pages/hour => 1,000 Pages/minute
  
• Less servers, less $$
• 4k+ writes/sec to Mongo

• Time-series data laid out as single documents
• Moved to bare-metal boxes
• Sharded 8 DBs on 2 boxes (8x write throughput)
• 4 master/4 slave per box
• Moved to SSDs (10x write throughput)

• I/O Concurrency is different than concurrency
• CPU intensive tasks are still blocking
• RAM limitations on your hosting environment
• Just because you have 10MB of JSON, doesn't mean it's not 100MB parsed
• Log and catch errors everywhere
• Beware of DB server connection issues

• Make small jobs that mainly do I/O
  
• Limit work done per worker to fit in RAM
• Auto-scale to do more work when necessary
• Child processes catching 'uncaughtException' and logging
• Monitor DB connection errors and restart automatically

• Customer base is expanding
• Data collected is expanding
• So many social networks, so little time