Dive into Service and Data Mesh

?

Gartner Hype Cycle

API gateway AND service mesh

API gateway pattern

  • Authentication and authorization
  • Service discovery integration
  • Response caching
  • Retry policies, circuit breaker, and QoS
  • Rate limiting and throttling
  • Load balancing
  • Logging, tracing, correlation
  • Headers, query strings, and claims transformation
  • IP whitelisting
  • Centralized Logging (transaction ID across the servers, error logging)

How it Works?

How it Works?

API gateway AND service mesh

OSI model

Implementations

Implementations

Why?

  • Secure service-to-service communication in a cluster with TLS encryption, strong identity-based authentication and authorization

 

  • Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic

 

  • Fine-grained control of traffic behaviour with rich routing rules, retries, failovers, and fault injection

 

  • A pluggable policy layer and configuration API supporting access controls, rate limits and quotas

 

  • Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress

The data plane

  • Service discovery: What are all of the upstream/backend service instances that are available?

 

  • Health checking: Are the upstream service instances returned by service discovery healthy and ready to accept network traffic?

 

  • Routing: Given a REST request for /foo from the local service instance, to which upstream service cluster should the request be sent?

The data plane

  • Load balancing: Once an upstream service cluster has been selected during routing, to which upstream service instance should the request be sent? With what timeout? With what circuit breaking settings? If the request fails should it be retried?

 

  • Authentication and authorization: For incoming requests, can the caller be cryptographically attested using mTLS or some other mechanism? If attested, is the caller allowed to invoke the requested endpoint or should an unauthenticated response be returned?

 

  • Observability: For each request, detailed statistics, logging, and distributed tracing data should be generated so that operators can understand distributed traffic flow and debug problems as they occur.

The control plane

 The control plane takes a set of isolated stateless sidecar proxies and turns them into a distributed system.

Data plane vs. control plane summary

Service mesh data plane: Touches every packet/request in the system. Responsible for service discovery, health checking, routing, load balancing, authentication/authorization, and observability.


Service mesh control plane: Provides policy and configuration for all of the running data planes in the mesh. Does not touch any packets/requests in the system. The control plane turns all of the data planes into a distributed system.

Hello world - example 1/2

Bookinfo Application - example 2/2

Questions?

Data mesh

data warehouse

...

data lake

...

data mesh

Domain Ownership

data product
as
architectural quantum

Architectural quantum, as defined by Evolutionary Architecture, is the smallest unit of architecture that can be independently deployed with high functional cohesion, and includes all the structural elements required for its function.

Data as a product

Self-serve data platform

Federated computational governance

Service Mesh

https://www.nginx.com/resources/library/the-enterprise-path-to-service-mesh-architectures/

https://istio.io/latest/docs/

https://docs.nginx.com/nginx-service-mesh/

https://www.amazon.com/Microservices-Patterns-examples-Chris-Richardson/dp/1617294543

https://www.infoq.com/articles/linkerd-v2-production-adoption

https://www.redhat.com/en/topics/microservices/what-is-a-service-mesh

https://www.openshift.com/blog/tag/service-mesh

https://platform9.com/blog/kubernetes-service-mesh-a-comparison-of-istio-linkerd-and-consul/

Data mesh

https://martinfowler.com/articles/data-monolith-to-mesh.html

https://martinfowler.com/articles/data-mesh-principles.html

https://www.jamesserra.com/archive/2021/02/data-mesh/

https://www.dlt.com/blog/2021/01/08/data-mesh-new-kid-data-architecture-block

https://www.infoq.com/news/2020/01/distributed-data-mesh/

https://towardsdatascience.com/what-is-a-data-mesh-and-how-not-to-mesh-it-up-210710bb41e0

https://medium.com/abn-amro-developer/retrospective-my-experience-writing-data-management-at-scale-66f2f315876

https://www.linkedin.com/pulse/data-mesh-design-benefits-hype-reality-saket-saurabh/

https://marcoullasci.com/2021/04/25/will-the-data-mesh-save-organizations-from-the-data-mess/

https://medium.com/abn-amro-developer/retrospective-my-experience-writing-data-management-at-scale-66f2f315876

https://www.jamesserra.com/archive/2021/01/data-lakehouse-defined/