Cross-Domain Events with AsyncAPI and AWS

Luke Hedger

The LEGO Group

AsyncAPI Conf - Nov 2022

Domain A

Domain B

Domain C

Event-driven architecture: Phase 1

Inter-domain events

Domain A

Domain B

Domain C

Event-driven architecture: Phase 2

Cross-domain events

Produce rich payment events in realtime to consumers via transport convenient to them

Baskets

Orders

Domains

Payments

Shopper checkout

Order submission

Payment requests

Asynchronous authorisations

Order fulfilment

Baskets

Orders

Current: pull

Payments

Is payment authorised?

Order submitted

Payment authorised

1

2

3

Baskets

Orders

Future: push

Payments

Order submitted

Payment authorised

1

1

1

2

Cross-domain events

  • Consumable - simple onboarding, familiar delivery
  • Discoverable - well-documented
  • Standard - standards-compliant, schema-based, interoperable
  • Versioned - predictable payloads, backwards-compatible
  • Secure - encrypted, verifiable messages
  • Scalable - handle high volume (100s events p/second)
  • Resilient - fault tolerant, replayable, high-availability

Standards

AsyncAPI

  • The AsyncAPI specification is the industry standard for defining asynchronous APIs
  • Growing ecosystem of tools for building and operating event-driven APIs

Amazon EventBridge

  • A serverless event bus with powerful rule-based filtering
  • Already used in domain - no new tech, subscribe to existing events
  • Target AWS workloads (cross-account events) and non-AWS consumers (HTTP API Destinations)

CloudEvents

  • A specification for describing event data in a common way
  • Predictable, portable event payloads
  • Reduced handling logic

Field mapping

AsyncAPI EventBridge CloudEvents
Channels Rules -
Channel Rule -
Message.name DetailType subject
Message.title DetailType subject
Message.payload.properties Detail data
- Time time

Cross-domain events flow

Discovery

AsyncAPI docs generation

$ ag asyncapi.yml @asyncapi/html-template -o asyncapi.out

Published to Backstage

Event hierarchy

  • Topical hierarchy of event types allows for granular filtering upon consumption
  • {Entity}.{Event}.{Metadata}
  • Metadata could be locale, platform etc
Payment.Authorised.Ecom
Payment.Authorised.Pos
Order.Created.Uk
Basket.Paid.Eu

Event evolution

  • Handle changes to events via additions whenever possible
  • Introduce new event types if breaking changes are required e.g. Payment.Authorised, Payment.AuthorisedV2, Payment.Completed
  • Produce both the old event and the new event for some time/forever to minimise disruption
  • Increment AsyncAPI version accordingly - new events = major, updated events = minor, fixes = patch

Enrichment

JSON Patch

  • Standards-based, schema-driven event payload enrichment
  • Reduces custom logic and risk of regression
  • JSON Patch operations: add, remove, replace, copy, move

Event IDs

  • Use ULID (Universally Unique Lexicographically Sortable Identifier) github.com/ulid/spec
  • Events can be chronologically sorted by ID

01AN4Z07BY79KA1307SR9X4MV3

Idempotency

  • Cannot guarantee exactly-once message production/delivery (3rd party and upstream services)
  • Support idempotent consumption
  • Use content hash (e.g. MD5) for deterministic idempotency key
0800fc577294c34e0b28ad2839435945

Security

Nested JSON Web Tokens

  • Signed - "is this event from trusted source?"
  • Encrypted - sensitive data is encrypted in transit
  • Encoded - compressed payloads

shared HMAC secret

private key (verified with public key)

Testing

Linting

  • Lint AsyncAPI definitions using Spectral
  • Built-in support for AsyncAPI
  • Custom rules can be used to enforce API styleguides

Validation

  • Events can be described as JSON Schema documents
  • Validate event payloads against event schema in unit tests (e.g. with AJV)
  • Producer (and consumers) could also validate at runtime

Testing integration points

  • Build-time integrations - test Infrastructure as Code (workflow is triggered by event, event bus target config)
  • Runtime integrations - test deployed resources (enrichment, event production)
  • Balance with observability

Event Bus

Event Bus

Workflow

Targets

Contract testing

  • Event contract between producer and consumers
  • Broker-less contract testing - replace traditional "broker" with a central registry of schema/contracts e.g. GitHub Packages
  • Schemas need to be versioned and distributed to consumers
  • Test workflow: Assert HTTP response expect.toMatchSchema

Observability

Health

  • Leverage AWS service metrics to monitor health of resources
  • Configure CloudWatch Alarms on key metrics - undeliverable events, workflow failures
  • Use custom metrics to collect data points about the application e.g. frequency of known errors

Performance

  • Trace timelines help identify bottlenecks and inefficient processes
  • Dashboards provide an overall view of system performanc

Recap

Cross-domain events

  • Use AsyncAPI to define, document and distribute event-driven APIs
  • Use CloudEvents to structure event payloads
  • Use Amazon EventBridge to choreograph events at scale
  • Use JSON Patch for event enrichment
  • Secure events with signed and encrypted JWTs
  • Devise comprehensive test strategy with variety of tests/linting
  • Balance testing with observability of application health and performance in production

Thanks!

Cross-Domain Events with AsyncAPI and AWS

By Luke

Cross-Domain Events with AsyncAPI and AWS

In this talk, Luke will describe the design, build and operation of a cross-domain event distribution service at The LEGO Group. The cross-domain event service is part of a production workload powering global e-commerce at massive scale, running on the AWS public cloud. A novel approach to standardising event schema and payloads that utilises AsyncAPI and CloudEvents will be presented. We will also explore the use of web standards in cross-domain event production, such as JSON Patch for data marshaling and JSON Web Tokens for encoding and encryption of payloads. We'll explore how the use of AsyncAPI tools supports the delivery lifecycle of this system, including Modelina for automated generation of type definitions and Generator for documentation. We will also see how AsyncAPI schema can be integrated with Spotify’s Backstage platform to simplify the onboarding of new event consumers through event documentation and discovery.

  • 308