- App uses Elasticsearch for displaying products
- PostgreSQL contains product information
- Need to get that data from PostgreSQL to Elasticsearch on a nightly basis
- Indexing in the current system is very slow
- Small features take up a lot of time to build
- Spaghetti Code
But why was it too slow ?
- ORM Abuse (Inefficient Queries)
- Serial execution of code
- Resources not being used efficiently (PostgreSQL is free when Elasticsearch is being utilized and vice-versa)
- Incremental development without refactoring and deprecating old functionalities
What to do now ?
GO TO THE RESCUE
Lets fix everything !
- Lets write efficient queries
- Lets use all our resources to their full potential
- Lets deprecate redundant business logic
- Lets use a language which helps us achieve this
- Static Typing
- Statically Compiled
- Built with concurrency in mind
- Very Fast!
- Controlled Concurrency
- Configurable number of extraction processes
- Configurable number of loading processes
- Non blocking I/O everywhere
|Time Taken||1h 30m||9m|
(256 GB RAM)
(32 GB RAM)
By Yash Mehrotra