Doing the things you should never do

Dr. Ólafur Helgason

VP of Engineering @ OZ

UTMessan 2015

6 feb 2015

What is OZ?

Video channel platform for creators

Upload and manage videos

Create live schedule

VOD and geo-awareness

Subscription and billing

Support major clients

First back-end incarnation

• Micro-services
  • Many (very) small services
  • Well defined interfaces - REST/MQ
• node.js + a little Erlang
• NoSQL - CouchDB
• RabbitMQ backbone
• IaaS hosting

Frustration grows

• Continuous deployment fragile
• Lack of automation and staging env
• IaaS means DYI
• Chef a headache
• Major dev, minor ops

Lesson: Automate everything from beginning

Lesson: Use Service Discovery

Lesson: Don't use IaaS unless you have OPS

Separating relational data

• Data split between services
• How is your data queried?
• Multiple service queries per req
• Solution:
  • Precompute in-memory indexes
  • Lot's of caching
  • Premature sharding

Lesson: Select service boundaries carefully

Lesson: Cache consistency is hard!

CouchDB

• Document/json store
• Quick and easy - no ORM
• Limited querying
  • Views index documents by json properties
  • Expensive to update - downtime!
• De-normalizing data
• Lost momentum
• Limited support & toolset

Lesson: Use relational DBs for relational data

node.js

• Early adopters
• Javascript runtime using Google V8
• Asynchronous non-blocking I/O
  • Excellent performance
• Active community and good package system
• Asynchronous programming challenging
  • Callback hell
  • Promises make life better

Lesson: node.js kicks ass

Sense of ownership

• Original architects left
• Lack of team ownership
• Important for startup

Lesson: team must have sense of ownership

Major requirements change

• Re-focused business model
• Simplified many things
• Dropping features
• This is the chance!

Second incarnation

• SOA but not micro
• node.js, PostgreSQL, RabbitMQ
• All-in on PaaS
  • Heroku
  • Significantly simplifies operations
  • Optimize for speed
• SaaS fanatics
• Focus on OZ!

Lesson: Use PaaS if you can afford it

(and if it makes sense for you business)

Conclusions

Did it take more time than planned?

Is it released?

Is rewriting the biggest mistake a software company can do?

Thanks!