PlayON is a sports entertainment company, with customers from over 100 countries.
Official partner of Formula 1 and NBA.
What is PlayON?
The challenge
Build the Official Formula 1 Fantasy game
Start building before contract was signed
2 months deadline, before the start of the season
150k users expected on the first season
Leverage existing daily game structure
Needs to be scalable to other partners
Ruby on Rails on the backend
Ember.js on the frontend
Formula 1 to be the first partner
We call it White Label
Planning
How we've planned the development
Eight chunks of work
Divided in eight weeks
Design, Frontend and Backend in parallel
Workshop with Formula 1 team to define the game mechanics
Architecture
High-level blueprint of the architecture used
Daily Backend
F1 Backend
New frontend app
Database
Building the backend
Namespace on the API for White Label operations
Feed processing engine is the same as the Daily game
API built leveraging Daily game structure
We decided to go with a Rails Engine on the Daily game
Ruby 2.3.7 + Rails 4.1.6
Users, teams, pricing, game periods
Built leagues and boosters on top of existing structure
Building the Frontend
Why Ember.js ?
- Daily game is Ember.js
- Team already familiar
Fresh Ember.js application started
Functionality built in parallel with the API
New design from scratch
Responsive from the ground up
Manual deployment - SCP
Authentication using shared cookie + JSON Web Token
Simple Live data polling
DevOps
One load balancer for the API instances
Auto scaling for the API instances
4 baseline instances for the API
Same database as the Daily game
20 on auto-scale
Postgres tuning was performed
Single Nginx server for the frontend
MVP built, time to load test
Ready on the last day, before the first race of the season
Contract still not signed, launch postponed
Load testing started and...
EXPECTATION
REALITY
All API endpoints optimized
Server configuration tuning - Nginx and Passenger
New frontend features and fine-tuning also in parallel
After a week of load testing, we were ready to support 6K concurrent users
Load testing - Flood.io
Ready to support 6K users before performance degrades
Serving around 25K RPM
Ready to launch!
Game launched!
Game launched on 25th of April
36K signups over the first 24 hours
Great repercussion on media
and social media
Trust me, I'm an engineer
Comes the first race weekend and I'm away in Brazil
Race time approaching, traffic building up
first holiday weekend
database stops responding
With not even 3K concurrent users
Restarting the database server brings it up for a few minutes, goes down again
Unavoidable conclusion:
Our postgres setup won't be able to handle it
Second unavoidable conclusion:
Our load testing strategy is not precise enough
Database migration - Saga of a weekend
Saturday 8:00 AM - Database offline
both Daily and Formula 1 games are down
We didn't have the know-how to tune our postgres setup
We were already planning a migration to AWS RDS so...
decided to migrate to RDS
First approach: Use the database migration service to avoid even more downtime
Comes the problems:
postgres 9.3 to 9.4
data type issues
dozens of other issues
Did the upgrade process on a testing database
only to be stopped by more issues
24 hours later, no success
Gave up on DMS
and started a dump and restore
Worked on the first go
data integrity checked and staging site running
Finished testing and ensuring all servers had the DB details updated
Sunday 9:30 PM - All systems up and running on AWS
Post-apocalypse
Healthy load balancers
Healthy API under stress
Database far faster than our self-managed instance
Remaining migration
Full migration was done afterwards, with practically no downtime and with time to do it
Load balancers
Auto scaling
Redis server
Sidekiq server
Production and staging environments
Ember deploy migrated to a CDN structure
Today
182K+ users
3K to 6K concurrent users at peak times
right before and right after the races
100K+ unique monthly visitors
82MI+ requests served monthly across Daily and White Label
Questions?
Thank you!
Let's play: bit.ly/dublinjs-f1
DublinJS - PlayON - Formula1 Fantasy
By Rafael Dalprá
DublinJS - PlayON - Formula1 Fantasy
The story of how F1 fantasy was built
