Citadel

Scenario

• App uses Elasticsearch for displaying products
• PostgreSQL contains product information
• Need to get that data from PostgreSQL to Elasticsearch on a nightly basis

Why

• Indexing in the current system is very slow
• Small features take up a lot of time to build
• Spaghetti Code

But why was it too slow ?

• ORM Abuse (Inefficient Queries)
• Serial execution of code
• Resources not being used efficiently (PostgreSQL is free when Elasticsearch is being utilized and vice-versa)
• Incremental development without refactoring and deprecating old functionalities

What to do now ?

GO TO THE RESCUE

Lets fix everything !

• Lets write efficient queries
• Lets use all our resources to their full potential
• Lets deprecate redundant business logic
• Lets use a language which helps us achieve this

Hello Go

• Static Typing
• Statically Compiled
• Built with concurrency in mind
• Very Fast!

Optimizing Everything

• Controlled Concurrency
• Configurable number of extraction processes
• Configurable number of loading processes
• Non blocking I/O everywhere

Architecture

Result

Thank You

@yashm95

/yashmehrotra