ShapeShift Pub/Sub
Rob Hilgefort
Problem
I worked at a Crypto company a few years ago and they were in the business of doing asset to asset transactions (tx).
As we scaled during the 2017 bull run, our tx times were extremely slow.
Their tx processing architecture was a monolith, run loop, and polling a mongo table/document for various statuses.
They would then call various functions and those functions would update the tx document.
Symptoms
- Monolith, deployed with pm2
- Non-existent TX schema
- Mostly crypto people that got into coding
- No "V2", needed to replace inline, and sooner than later
- Callbacks, not even promises
- Write collisions- missing / inconsistent updates
- Failed functions that never retried
- Run loop limited by slowest functions
Constraints
Roadmap / Solution
- Get the codebase off extremely nested callbacks (control flow optimization)
- Proposed pub/sub and microservices of "functions"
- RabbitMQ vs Redis
- Kafka wasn't considered, don't recall why 😬
- Data integrity & visibility > speed (rabbit fast enough)
- POC'd RabbitMQ & Redis Pub/Sub
- Infra upgrades to Docker / Compose / Swarm while microservicing
- Project breakdown, epic, tickets, parallelization
Outcomes
- TX Speed: Slow functions can be give more resources
- Data Integrity: RabbitMQ allowed us to know when we had problematic processing and capture errors tied to TXs.
- Visibility: RabbitMQ admin interface allowed for visibility into slow functions (message backups)
- Metrics: Dashboard allowed for function average runtime calculations for free.
- In-place Rewrite: We were able to break out individual functions into microservices one by one
- [Bonus] Golang: We (api team) could more easily communicate with the Crypto team that wrote in Go
Human Solutions
- I managed up through my engineering manager to show the benefits of my proposal.
- Held meetings before project approval to pitch the plan and demo the POC to API team members
- Rough estimates of project effort and recommended a team size based on parallel tasks I saw
- Lead and managed a project team, while also being a core contributor.