Citadel
Scenario
- App uses Elasticsearch for displaying products
- PostgreSQL contains product information
- Need to get that data from PostgreSQL to Elasticsearch on a nightly basis
Why
- Indexing in the current system is very slow
- Small features take up a lot of time to build
- Spaghetti Code
But why was it too slow ?
- ORM Abuse (Inefficient Queries)
- Serial execution of code
- Resources not being used efficiently (PostgreSQL is free when Elasticsearch is being utilized and vice-versa)
- Incremental development without refactoring and deprecating old functionalities
What to do now ?
GO TO THE RESCUE
Lets fix everything !
- Lets write efficient queries
- Lets use all our resources to their full potential
- Lets deprecate redundant business logic
- Lets use a language which helps us achieve this
Hello Go
- Static Typing
- Statically Compiled
- Built with concurrency in mind
- Very Fast!
Optimizing Everything
- Controlled Concurrency
- Configurable number of extraction processes
- Configurable number of loading processes
- Non blocking I/O everywhere
Architecture
Result
| NIGHTLY | Old | New |
|---|---|---|
| Time Taken | 1h 30m | 9m |
| Machine Size | r3.8xlarge (256 GB RAM) |
t2.2xlarge (32 GB RAM) |
| MERCHANT JOB | Old | New |
|---|---|---|
| Time Taken | 25m | 45s |
| Machine Size | r3.xlarge | t2.large |
Thank You
@yashm95
/yashmehrotra
