ShapeShift Pub/Sub

Rob Hilgefort

Problem

I worked at a Crypto company a few years ago and they were in the business of doing asset to asset transactions (tx).

As we scaled during the 2017 bull run, our tx times were extremely slow.

Their tx processing architecture was a monolith, run loop, and polling a mongo table/document for various statuses.

They would then call various functions and those functions would update the tx document.

Get the codebase off extremely nested callbacks (control flow optimization)
Proposed pub/sub and microservices of "functions"
RabbitMQ vs Redis
- Kafka wasn't considered, don't recall why 😬
- Data integrity & visibility > speed (rabbit fast enough)
POC'd RabbitMQ & Redis Pub/Sub
Infra upgrades to Docker / Compose / Swarm while microservicing
Project breakdown, epic, tickets, parallelization

TX Speed: Slow functions can be give more resources
Data Integrity: RabbitMQ allowed us to know when we had problematic processing and capture errors tied to TXs.
Visibility: RabbitMQ admin interface allowed for visibility into slow functions (message backups)
Metrics: Dashboard allowed for function average runtime calculations for free.
In-place Rewrite: We were able to break out individual functions into microservices one by one
[Bonus] Golang: We (api team) could more easily communicate with the Crypto team that wrote in Go

I managed up through my engineering manager to show the benefits of my proposal.
Held meetings before project approval to pitch the plan and demo the POC to API team members
Rough estimates of project effort and recommended a team size based on parallel tasks I saw
Lead and managed a project team, while also being a core contributor.