Citadel

Why

  • Indexing is very slow
  • Small features take up a lot of time to build
  • Spaghetti Code

But why was it too slow ?

  • ORM Abuse
  • Serially written code
  • Resources not being used efficiently
  • Incremental development without refactoring and deprecating old functionalities

What to do now ?

YASH TO THE RESCUE

Lets fix everything !

  • Lets write efficient queries
  • Lets use all our resources to their full potential
  • Lets deprecate redundant business logic
  • Lets use a language which helps us achieve this

Hello Go

  • Static Typing
  • Statically Compiled
  • Built with concurrency in mind
  • Very Fast!

Optimizing Everything

  • Controlled Concurrency
  • Configurable number of extraction processes
  • Configurable number of loading processes
  • Non blocking I/O everywhere

Architecture

Result

NIGHTLY Old New
Time Taken 1h 30m 10m
Machine Size r3.8xlarge t2.2xlarge
MERCHANT JOB Old New
Time Taken 25m 45s
Machine Size r3.xlarge t2.large

Thank You

@yashm95

/yashmehrotra

Citadel

By Yash Mehrotra

Citadel

  • 74
Loading comments...

More from Yash Mehrotra