DEV

QA

PROD

And so on ....

US

EUROPE

• HA Mongo Cluster

 

• HA Rabbitmq cluster

 

• HA Redis cluster

 

• Percona cluster

 

• ETCD 

 

• Kubernetes master controller

 

• Pilosa cluster (Bitmap indexing)

• Fluentd cluster / Log aggregator daemon

• HA Reverse proxy / Gateway

• Exporter agent (Metrics/Logs/Traces)

• Ingress Controller
• External-dns

• User service

 

• Subscription service

 

• Uploader service

 

• Events service

 

• Notify service 

 

......

 

• Accounts service 

 

• Interpretation processor 

 

• Dataset service

 

• Tokenzier, Lemmatizer service

 

• Web hook service

 

• Deterministic

 

• Fault tolerant

 

• Highly available

 

• Resilient

 

• Observable 

 

A service with no side effects.

 

A service which does not rely on cache to serve request.

 

A service which does not rely on another service to serve request.

 

Service A requires that service B to exists and function. 

 

Example:

• Your service talks to database.
• Your service does cache invalidation.
• Your service deployment requires defined sequence of structure.

Delicate intents by events.

 

Your service intent is to make database call for persistence, emit an event.

 

Your service intent is to make email, emit an event.

 

Provision resource creation as well as deletion.

 

Allow your infrastructure to be function of time.

 

Allow your infrastructure to be reproducable.

 

Service/resource deployment requires DEVops.

Service deployment requires reaching to aws console/cli.

Dependency management is a manual process. Eg: You cannot delete vpc because ec2 is attached to it.

Resource creation and teardown should be your religion.

 

• Tools like terraform enforces immutability and provides dependency management 

 

You know how to fail and come back up and not the other way around.

You remove human's error prone brain function during dependency management.

It is the best documentation of your infrastructure that you will ever have.

Reverts are super easy. Time travel.

We spin up entirely new set of infrastructure in 10 minutes.

Kubernetes

Cloudfront 

Ec2 worker nodes

ECR

Security groups, NACL,

ALB

VPC,

3 AZs,

Private and public subnets,

Bastion hosts,

 

and 35 Load Balancers and 60+ services.

HA mongo cluster

HA redis cluster

HA Pilosa cluster

Exporters

Agents 

Fluentd

.....

We spin up dev cluster during weekdays and destroy during weekends.

A service with entry and exit path.

 

A distributed system with shortest time to root cause analysis.

A distributed systems with trace continuity across process boundaries.

Service does have context of outbound and inbound request.

Service does not propagate context across process boundaries.

Your trace is limited to function call stack within the process.

Instrument and profile every execution path

• Request Response lifecycle
• Http request
• Event from rabbitmq
• Event to rabbitmq
• Database transaction
• ....

Use instrumented http client with custom transport layer.

Use monitor api for database to trace transactions.

Use transport headers for messaging systems to trace events.

Database deployment with automatic backup and recovery.

 

Unified API to manage all the above irrespective of your database stack.

Automatic promotion and demotion of master/slave db cluster.

 

You have to hire DBAs.

Your backups and restores are manual.

You require manual work during master/slave replication failure.

Your data storage classes are not well defined. For eg: /data, /journal, /logs

• Wrap your persistence layer with operators. Eg: Percona.
• Provision backup/recovery and co-ordinator agent along side your database of choice.
• Prefer sentinel client over default database client.
• Provision sentinel cluster for automatic master/slave promotion.
• Wrap the underlying details with configuration. Eg: Kubernetes operator, Custom Resource defintions

 

 

README says "Blazingly fast reverse proxy"

README says "Compiles in < 1 sec"

Official website says "Community Edition"

• Validate if  piece of software follows industry standards for  instrumentation and profilings.Eg: Open telemetry, Open census, Open tracing.
• Validate if it comes with sentinel and replication controller built in (DBS)
• Validate if they provide operators for backups and recovery.
• Validate if they have quick release cycles.

 

 

•  $50,000 per year for mongo operator.

•  $25,000 for opencensus exporter for load balancer.

• Incomplete set of terraform modules for kubernetes provisioning of External DNS,  Ingress Controller,  Storage Class ...

• Design for request propagation and cancelation across process boundaries.

• Abstract storage provisioning with operators and sentinel clusters.

• Allow systems to fail as soon as possible.

• Profile everything to better understand the nature of your systems.

• Use circuit breaker for dependent services.