Citadel

Scenario

  • App uses Elasticsearch for displaying products
  • PostgreSQL contains product information
  • Need to get that data from PostgreSQL to Elasticsearch on a nightly basis

Why

  • Indexing in the current system is very slow
  • Small features take up a lot of time to build
  • Spaghetti Code

But why was it too slow ?

  • ORM Abuse (Inefficient Queries)
  • Serial execution of code
  • Resources not being used efficiently (PostgreSQL is free when Elasticsearch is being utilized and vice-versa)
  • Incremental development without refactoring and deprecating old functionalities

What to do now ?

GO TO THE RESCUE

Lets fix everything !

  • Lets write efficient queries
  • Lets use all our resources to their full potential
  • Lets deprecate redundant business logic
  • Lets use a language which helps us achieve this

Hello Go

  • Static Typing
  • Statically Compiled
  • Built with concurrency in mind
  • Very Fast!

Optimizing Everything

  • Controlled Concurrency
  • Configurable number of extraction processes
  • Configurable number of loading processes
  • Non blocking I/O everywhere

Architecture

Result

NIGHTLY Old New
Time Taken 1h 30m 9m
Machine Size r3.8xlarge
(256 GB RAM)
t2.2xlarge
(32 GB RAM)
MERCHANT JOB Old New
Time Taken 25m 45s
Machine Size r3.xlarge t2.large

Thank You

@yashm95

/yashmehrotra

Citadel Meetup

By Yash Mehrotra

Citadel Meetup

  • 68
Loading comments...

More from Yash Mehrotra