A practical DDD approach to Nameko microservices

Alicante, 5th of October

Julio Trigo

About me

●  Computer Science Engineer

●  University of Alicante Polytechnic School

●  Based in London

●  Certified Scrum Master (CSM)

●  Software developer

juliotrigo

juliotrigo

About me

●  ~17 platform services

●  ~5 user applications

ClearView Flex

●  ~7 external client applications

●  Dev team

–  Cross-functional

–  Works on the whole product

Agenda

  Nameko

➋  A (practical) DDD approach to microservices

➌  Splitting code into components

What is this about❓

✅  My experience working with Nameko, microservices, and DDD across several teams and products that has proven successful and beneficial when working on a product codebase

 

✅  A practical DDD approach

 End up with consistent components that are easy to understand, extend and replace

✅  Just a way of splitting code into components using microservices

  Nameko introduction and overview

  Product codebase that we plan to maintain for a long time

What is this not about

❌  The only way of doing things

❌  The "right" way of doing things: DDD, microservices, Nameko, splitting code, etc.

❌  What microservices are

❌  How to deal with microservices: deployment, logging, etc.

❌  Advantages/disadvantages of using microservices

❌  Not the "best way" for all use cases: POC, research, short-lived components and many other cases

❌  Code performance

❌  Nameko internals

Agenda

  Nameko

➋  A (practical) DDD approach to microservices

➌  Splitting code into components

Nameko

●  Microservices

●  What's Nameko?

●  Services: how to write and run a service

●  Communication protocols: HTTP, RPC, AMQP

●  Extensions (entrypoints, dependencies)

●  Workers

  What they are and how to use them

  Built-in and community

●  Testing

Microservices

"The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services.

While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data."

 Microservices - James Lewis, Martin Fowler

https://martinfowler.com/articles/microservices.html

Microservices

●  Microservices can be a "Replaceable Component Architecture" † 

GOTO 2016 - Software that Fits in Your Head - Dan North

https://www.youtube.com/watch?v=4Y0tOi7QWqM

  if you choose to optimise for replaceability and consistency

Microservices

●  Having small services normally comes as a result

●  Small services Microservices

●  End goal  to have a big number of small services

●  "A framework for building microservices in Python"

●  Japanese mushroom, which grows in clusters

●  Encourages the dependency injection pattern

●  Encourages testability

●  Define your own transport mechanisms and service dependencies: mix & match

●  Is not a web framework

●  "Extensible"

  Compatible with almost any protocol, transport or database

  Built-in extensions, community extensions or "build your own"

●  "Focus on business logic"

●  "Distributed and scalable"

  Write regular Python methods and classes

  Nameko manages connections, transports and concurrency

  Spin up multiple service instances to easily scale out

  Concurrency by yielding workers when they wait for I/O

Service

●  A Nameko service is just a Python class

●  Declares dependencies as attributes

●  Encapsulates the application logic in its methods

●  Methods are exposed with entrypoint decorators

# service.py

from nameko.rpc import rpc


class PaymentService:

    name = "payment"
    
    @rpc
    def healthcheck(self):
        return "OK"
      
$ nameko run service

starting services: payment
Connected to amqp://guest:**@127.0.0.1:5672//

Running services

# config.yaml

AMQP_URI: "amqp://guest:guest@localhost:5672"
WEB_SERVER_ADDRESS: "0.0.0.0:8000"
max_workers: 10
$ nameko run service --config ./config.yaml

starting services: payment
Connected to amqp://guest:**@127.0.0.1:5672//

●  Config

$ nameko shell --broker amqp://guest:guest@localhost:5672/

Nameko Python 3.6.8 (default, Mar  9 2019, 12:20:08)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] shell on darwin
Broker: amqp://guest:guest@localhost:5672/

>>> n.rpc.payment.healthcheck()
'OK'

Interacting with running services​

AMQP broker

$ nameko run service

starting services: payment
Connected to amqp://guest:**@127.0.0.1:5672//

Communication protocols (HTTP)

●  Built on top of werkzeug

●  Supports all the standard HTTP methods

  GET / POST / DELETE / PUT / OPTIONS / ...

Communication protocols (RPC)

●  RPC (Remote procedure call) over AMQP

●  Normal RPC calls block until the remote method completes

●  Asynchronous calling mode to parallelize RPC calls: call_async

●  AMQP messages are ack’d only after the request has been successfully processed

●  Request and response payloads are serialized into JSON for transport over the wire

Communication protocols (Pub-Sub)

●  Publish-Subscriber pattern

●  How event messages are received in a cluster:

  SERVICE_POOL (default): event handlers are pooled by service name and one instance from each pool receives the event (similar to RPC)

  BROADCAST: every listening service instance receives the event

  SINGLETON: exactly one listening service instance receives the event

●  Request and response payloads are serialized into JSON for transport over the wire

Entrypoints

●  Gateways into the service methods they decorate

●  They normally monitor an external entity

  Message queue, Redis event, etc.

●  On a relevant event:

  The entrypoint “fires”

  The decorated method is executed by a service worker

Entrypoints (built-in: HTTP)

●  @http entrypoint

# admin_facade.service.py

from nameko.web.handlers import http


class AdminFacade:

    name = "admin-facade"
    
    @http(
        "POST",
        "/projects/<int:project_id>/orders/<int:order_id>"
    )
    def confirm_order_payment(
      self, request, project_id, order_id
    ):
      	# ...
        return 200, "OK"
      

Entrypoints (built-in: HTTP)

$ curl -X POST -i http://localhost:8000/projects/1/orders/200

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 2
Date: Thu, 03 Oct 2019 21:47:07 GMT
# config.yaml

AMQP_URI: "amqp://guest:guest@localhost:5672"
WEB_SERVER_ADDRESS: "0.0.0.0:8000"
$ nameko run admin_facade.service --config ./config.yaml

starting services: admin-facade

Entrypoints (built-in: RCP)

●  RpcProxy dependency for services to talk to other services

# admin_facade.service.py

from nameko.rpc import RpcProxy
from nameko.web.handlers import http


class AdminFacade:

    name = "admin-facade"
    
    payment_service = RpcProxy("payment")
    
    @http(
        "POST",
        "/projects/<int:project_id>/orders/<int:order_id>"
    )
    def confirm_order_payment(
      	self, request, project_id, order_id
    ):
      	# ...

      	payment = self.payment_service.pay_order(order_id)

        # ...
        
        return 200, "OK"

Entrypoints (built-in: RCP)

# payment.service.py

from nameko.rpc import rpc


class PaymentService:

    name = "payment"
        
    @rpc
    def pay_order(self, order_id):
        """Business logic to pay orders."""
      

●  @rpc entrypoint

Entrypoints (built-in: RCP)

●  ServiceRpcProxy: standalone proxy that non-Nameko clients can use to make RPC calls to a cluster

from nameko.standalone.rpc import ServiceRpcProxy


AMQP_URI = "amqp://guest:guest@localhost:5672"


class NonNamekoService:

    def process_order(self, order_id):
        # ...

        with ServiceRpcProxy("payment", AMQP_URI) as rpc:
            rpc.pay_order(order_id)

        # ...

Entrypoints (built-in: Pub-Sub)

●  EventDispatcher dependency for services to emit messages

●  @event_handler entrypoint

# payment.service.py

from nameko.events import EventDispatcher
from nameko.rpc import rpc


class PaymentService:

    name = "payment"
    
    dispatch = EventDispatcher()
    
    @rpc
    def pay_order(self, order_id):
        # ...

        event_body = {
            "order": {"id": order_id},
            "payment_method": {"id": 100},
            "project": {"id": 200},
            "paid_by": {"id": 300},
        }
        self.dispatch("payment_taken", event_body)

        # ...
      

Entrypoints (built-in: Pub-Sub)

# messaging.service.py

from nameko.events import event_handler


PAYMENT_SERVICE_NAME = "payment"
PAYMENT_TAKEN_EVENT = "payment_taken"


class MessagingService:

    name = "messaging"
    
    @event_handler(PAYMENT_SERVICE_NAME, PAYMENT_TAKEN_EVENT)
    def send_order_receipt_email(self, payload):
        #  handler_type = "service_pool"

        order = payload["order"]
        project = payload["project"]
        payment_method = payload["payment_method"]
        paid_by = payload["paid_by"]
        
        # ...
      

Entrypoints (community)

/nameko/nameko-amqp-retry

/nameko/nameko-salesforce

/iky/nameko-slack

/sohonetlabs/nameko-rediskn

  Nameko extension allowing AMQP entrypoints to retry later

  A Nameko extension for handling Salesforce Streaming API

  Nameko extension for interaction with Slack APIs

  Redis Keyspace Notifications extension for Nameko services

Entrypoints (build your own)

●  Extensions should subclass nameko.extensions.Entrypoint

●  Some methods can be overridden to add functionality

from nameko.extensions import Entrypoint  # Inherits from `Extension`


class ExampleEntrypoint(Entrypoint):
  
    # Extension` methods

    def setup(self):
        """Called before the container starts.

        Do any required initialisation here.
        """

    def start(self):
        """Called when the container has successfully started.

        Start acting upon external events.
        """

    def stop(self):
        """Called when the service container begins to shut down.

        Do any graceful shutdown here.
        """

    def kill(self):
        """Called to stop this extension without grace.
        
        Urgently shut down here.
        """


# Usage: `@entrypoint_name`
entrypoint_name = ExampleEntrypoint.decorator

Dependencies

●  Nameko encourages to declare service dependencies

●  Hide/encapsulate code that isn’t part of the core service logic

●  Simple interface to the service

●  Gateway between service code and everything else

  Other services, external APIs, DBs...

●  The class attribute

●  DependencyProvider

= a DependencyProvider

 the interface that workers can actually use

= a declaration

  Responsible for providing an object that is injected into service workers

  Implement get_dependency(): the result is injected into a newly created worker

  Live for the duration of the service

worker = Service()
worker.other_rpc = worker.other_rpc.get_dependency()
worker.method()
del worker

●  The injected dependency: unique to each worker

Dependency injection

Dependencies (built-in)

●  RpcProxy for services to talk to other services

●  EventDispatcher for services to emit messages

●  Config for read-only access to config values at run time

# service.py

from nameko.dependency_providers import Config
from nameko.rpc import rpc


class PaymentService:

    name = "payment"

    config = Config()

    @rpc
    def healthcheck(self):
        amqp_uri = self.config.get("AMQP_URI")
        return "OK"

Dependencies (community)

/nameko/nameko-sqlalchemy

/nameko/nameko-sentry

/etataurov/nameko-redis

/fraglab/nameko-redis-py

  SQLAlchemy dependency for nameko services

  Captures entrypoint exceptions and sends tracebacks to a Sentry server

  Redis dependency for nameko services

  Redis dependency and utils for Nameko

Dependencies (community)

/nameko/nameko-tracer

/nameko/nameko-salesforce

/sohonetlabs/nameko-statsd

/marcuspen/nameko-stripe

  Logs traces: comprehensive information about entrypoints fired

  Nameko dependency provider for easy communication with the Salesforce REST API

  Statsd Nameko dependency provider

  Stripe dependency for Nameko services

Dependencies (build your own)

●  Subclass nameko.extensions.DependencyProvider

●  Implement a get_dependency() method: returns the object to be injected into service workers

from nameko.extensions import DependencyProvider    # Inherits from `Extension`


class ExampleDependencyProvider(DependencyProvider):
  
    # `Extension` methods

    # `DependencyProvider` methods

    def get_dependency(self, worker_ctx):
        """ Called before worker execution.
        
        Return an object to be injected into the worker instance by the container.
        """
    
    def worker_result(self, worker_ctx, result=None, exc_info=None):
        """ Called with the result of a service worker execution.
        
        Dependencies that need to process the result should do it here.
        This method is called for all `Dependency` instances on completion
        of any worker.
        """

    def worker_setup(self, worker_ctx):
        """ Called before a service worker executes a task.
        
        Dependencies should do any pre-processing here, raising exceptions
        in the event of failure.
        """

    def worker_teardown(self, worker_ctx):
        """ Called after a service worker has executed a task.
        
        Dependencies should do any post-processing here, raising
        exceptions in the event of failure.
        """

   

Workers

Event

Entrypoint fires

Worker is created

Decorated method is executed

Worker is destroyed

Dependencies injected into worker

Instance of the service class

Only lives for the method execution

Result of get_dependency()

Testing

●  Nameko’s conventions: make testing as easy as possible

●  Dependency injection

●  pytest

●  Utilities:

  Makes it simple to replace and isolate pieces of functionality

worker_factory()

entrypoint_hook()

replace_dependencies()

pytest fixtures: web_config, rabbit_config, container_factory

Agenda

  Nameko

➋  A (practical) DDD approach to microservices

➌  Splitting code into components

DDD & Microservices

●  How the design of the solution is driven by the domain of the application

●  Contracts between services

  Service boundaries

  Service APIs

●  Types of services and responsibilities

  Application services (facades)

  Domain services

●  Applications and services

  User applications (internal / external)

  Platform services

  External systems

  Client applications

Web app

Support CP app

Admin CP app

Data analysis app

Applications and services

●  External systems

●  User applications (internal / external)

●  Client applications

●  Platform services

Web Facade

Payment

Service

Network app

Web app

Support CP app

Admin CP app

Data analysis app

Single application

API + Business logic

Web app

Support CP app

Admin CP app

Data analysis app

API

Business logic

Web app

Support CP app

Admin CP app

Data analysis app

These architecture approaches

●  Single application

●  API + business

●  Most of the components are deployed together

●  Difficult to scale individual parts on demand 

●  Refactor: riskier to replace / rewrite components

●  API / business

●  No real separation between components

●  Test / debug: more difficult to isolate single components

Web app

Support CP app

Admin CP app

Data analysis app

Web Facade

Business logic

Mobile Facade

Admin Facade

Support CP Facade

Data Facade

Application services (facades)

●  Provide an interface to the external application that serves

●  Contract between platform services and the user application

  Deploying a facade can only affect one user application

●  It only serves one application

  Simplify smoke tests after deployments

  No risk of introducing bugs into other parts of the system

Support CP app

Support CP Facade

Data (request)

Data (response)

Protocol

(e.g. deploying Support CP Facade can only affect support users)

Web app

Support CP app

Admin CP app

Data analysis app

Web

Facade

Payment

Service

Mobile

Facade

Admin

Facade

Support CP Facade

Data

Facade

User

Service

Auth

Service

Project

Service

Messaging

Service

Domain services

●  Changes within a business domain only affect one service

  Group logic and responsibilities around business domains

Payment

Service

User

Service

Auth

Service

Project

Service

Messaging

Service

●  Contract (namespace) shared with the outside world

Contracts between services

●  Messages that services produce and consume

●  The service API defines the contract with the outside world

●  Service boundaries (API):

  Define the endpoints to be exposed

  Define the "domain language" to be exposed

  Define the domain exceptions to be exposed

  Define the communication protocol to be exposed

●  Encapsulation

A practical DDD approach

●  The design of the solution is driven by the domain of the application

●  A practical approach to DDD

  "Intelligence in the endpoints"

    @rpc
    def confirm_payment_intent(self, payment_intent_id):
        # ...

    @rpc
    def get_viewer_scores(self, streaming_session_id, email):
        # ...

    @rpc
    def get_multiple_transactions_by_id(self, transaction_ids):
        # ...

A practical DDD approach

  Makes refactoring:

  Do not share DBs

  Separation between DBs: Redis, SQL, Elasticsearch...

  Could potentially share the same server/instance (reduce costs)

  Different services can't access the same piece of data

  Easier

  Less time consuming

  Less riskier

  e.g. "refactor internal endpoints logic"

  e.g. "replace external 3rd party systems (Stripe, auth provider...)"

A practical DDD approach

❌  Create services about technologies with CRUD operations

  e.g. Elasticsearch service, Redis service...

  e.g. DB service, API service...

❌  Create services about types of tech with CRUD operations

❌  Couple technologies to services (contract)

  Would need a new service per technology

  Instead: facades, domain services, and intelligent endpoints

  Would affect the whole architecture

  Instead: service dependencies and intelligent endpoints (vs. CRUD)

  Replace technologies without affecting the service contract

Agenda

  Nameko

➋  A (practical) DDD approach to microservices

➌  Splitting code into components

Code components

●  Namespaces

●  Complexity metric

●  Components

  Purpose and boundaries

●  Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

Code components

●  “Simple is better than complex.”

●  Single responsibility principle

●  Simple, intuitive and elegant API

  "A class should have only one reason to change" - Robert C. Martin

PAYMENT_TAKEN_EVENT = "payment_taken"

class TaxCalculationError(Exception):
    pass

def validate_country_format(country):
    # ...

from nameko_sqlalchemy import DatabaseSession

class PaymentMethod(marshmallow.Schema):
    # ...

class PaymentService():

    name = "payment"
    
    db = Database()
    stripe = Stripe()

    @rpc
    def list_payment_methods(self, customer_id):
    	# ...
    	return schemas.dump(
          schemas.PaymentMethod, payment_methods, many=True
        )

Namespaces

●  “Namespaces are one honking great idea -- let's do more of those!”

●  Avoid name collisions

●  "Name" / encapsulate / group components

self.db.transactions.get_by_order_id

self.stripe.list_payment_methods

self.db.NotFound
remote_err.NotFound
http_err.NotFound

schemas.dump
schemas.Payment

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

●  “If the implementation is hard to explain, it's a bad idea.”

●  “Readability counts.”

  More time reading than writing

Admin CP app

Payment Service

Messaging Service

Admin Facade

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

Admin CP app

●  Front-end application

●  Calls the platform services

  Contract with the Admin Facade

●  No back-end logic

●  Serverless? AWS Cloudfront, Lambda@Edge, S3

Payment Service

Messaging Service

/admin_facade

http_errors.py

http_handlers.py

remote_errors.py

schemas.py

service.py

/payment_service

dependencies

db​.py

nameko_stripe.py

exceptions.py

states.py

/messaging_service

service.py

service.py

schemas​.py

models.py

Admin Facade

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# admin_facade.schemas.py

# Responsible of the service messages serializations (in/out)
# Defines the (data) contract with the app that it serves


from marshmallow import fields, Schema


class Payment(Schema):
  
    """Contract with the Admin CP application."""

    id = fields.Str()
    status = fields.Str(required=True)
    # ...


def dump(schema_cls, data, strict=True, **schema_kwargs):
    schema = schema_cls(strict=strict, **schema_kwargs)
    return schema.dump(data).data

Schemas

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# admin_facade.http_handlers.py

import json

from werkzeug.wrappers import Response


JSON_CONTENT_TYPE = "application/json"


def json_response(data, status=200, content_type=JSON_CONTENT_TYPE):
    return Response(
        response=json.dumps(data),
        content_type=content_type,
        status=status
    )

Module

Functions

Constants

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# admin_facade.http_errors.py

class HttpError(Exception):

    status_code = 500

    def __init__(self, description):
        self.payload = description
        super().__init__(self.payload)


class BadRequest(HttpError):

    status_code = 400


class NotFound(HttpError):

    status_code = 404

Exceptions

# admin_facade.remote_errors.py

from nameko.exceptions import registry


def remote_error(exc_path):

    def _wrapper(exc_type):
        registry[exc_path] = exc_type
        return exc_type

    return _wrapper


@remote_error('payment.exceptions.BadRequest')
class BadRequest(Exception):
    pass


@remote_error('payment.exceptions.NotFound')
class NotFound(Exception):
    pass
  

@remote_error('payment.exceptions.TransactionNotFound')
class TransactionNotFound(Exception):
    pass

Exceptions

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Classes

Methods

Functions

Exceptions

Constants

Domain services

# admin_facade.service.py

from nameko.rpc import RpcProxy
from nameko.web.handlers import http
from nameko_sentry import SentryReporter

from . import http_errors as http_err  # Namespaces
from . import remote_errors as remote_err
from . import schemas
from .http_handlers import json_response  # Widely used / not ambiguous


PAYMENT_SERVICE_NAME = "payment"


class AdminFacade:

    name = "admin-facade"
    
    # Dependencies
    payment_service = RpcProxy(PAYMENT_SERVICE_NAME)
    sentry = SentryReporter()
    
    # ...

Service class

Constants

Endpoints

Admin Facade

    # ...
    
    @http(
        "POST",
        "/projects/<int:project_id>/orders/<int:order_id>"
    )
    def confirm_order_payment(self, request, project_id, order_id):
      	# ...
      	
      	try:
      	    payment = self.payment_service.pay_order(order_id)
        # Translate remote "expected exceptions"
        except remote_err.BadRequest as err:
            raise http_err.BadRequest(err.args[0])
        except remote_err.NotFound as err:
            raise http_err.NotFound(err.args[0])
        except remote_err.TransactionNotFound as err:
            # Translate to a different HTTP error
            raise http_err.BadRequest(err.args[0])
        
        # Endpoint does not know about formatting/serializing the response
        response = schemas.dump(schemas.Payment, payment)

        return json_response(response)
      

Payment Service

Messaging Service

/admin_facade

http_errors.py

http_handlers.py

remote_errors.py

schemas.py

service.py

/payment_service

dependencies

db​.py

nameko_stripe.py

exceptions.py

states.py

/messaging_service

service.py

service.py

schemas​.py

models.py

Admin Facade

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# payment.dependencies.db.py

from nameko_sqlalchemy import DatabaseSession

from .. import models  # The dependency uses the SQLAlchemy models


# Logic related to the dependency lives in the dependency module / package:
#   - Constants, exceptions, functions...

# Service endpoints:
#   - Do not know about SQL-related objects/actions
#   - Interact with the DB objects through "collections"
 

# Dependency exceptions live in the dependency namespace
class NotFound(LookupError):
    pass

Service dependencies

Exceptions

Constants

3rd party packages

# ...

class Transaction:
  
    """Collection class to provide an interface for the DB.
    
    Provide an abstraction (contract) between the service and the dependency.
    """

    name = "transactions"
    model = models.Transaction
    
    def __init__(self, session):
        self.session = session

    def get_by_order_id(self, order_id):
        """Logic to interact with the DB to get the transaction.
        
        Raises `NotFound`
        
        "Intelligent endpoints", like services, not just CRUD operations.
        """
# ...

class DatabaseWrapper:

    # Services access the dependency expcetions through the wrapper object
    NotFound = NotFound

    def __init__(self, session, collections):
        for collection in collections:
            setattr(self, collection.name, collection(session))


class Database(DatabaseSession):
  
    """Database dependency provider.
    
    Wrapper around the Nameko SQLAlchemy dependeny provider.
    """

    def __init__(self):
        super().__init__(models.DeclarativeBase)

    def get_dependency(self, worker_ctx):
        # A `DatabaseWrapper` instance will be injected to the service
        # as a dependency.
        return DatabaseWrapper(
            session=super().get_dependency(worker_ctx),
            collections=[Transaction]
        )

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# payment.schemas.py


# Responsible of the service messages serializations (in/out)
# Defines the (data) contract with the rest of the platform services


from marshmallow import fields, Schema


class Payment(Schema):
  
    """Contract with the rest of the platform services."""

    id = fields.Str()
    status = fields.Str(required=True)
    # ...


def dump(schema_cls, data, strict=True, **schema_kwargs):
    schema = schema_cls(strict=strict, **schema_kwargs)
    return schema.dump(data).data

Schemas

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# payment.exceptions.py

# Remote exceptions are as explicit and simple as possible:
#   - Exception name: specific and explicit
#   - Exception argument: error message (string)


class NotFound(Exception):
    pass


class BadRequest(Exception):
    pass


class TransactionNotFound(Exception):
    pass

Exceptions

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# payment.states.py

from enum import auto, Enum


class AutoName(Enum):

    def _generate_next_value_(name, start, count, last_values):
        return name.lower()


class PaymentState(AutoName):

    """Used to define all the states a Payment can be in.
    
    Encapsulate the states in the `PaymentState` class and 
    all the logic around them in the `states` module.
    """

    AWAITING_CONFIRM = auto()
    PAID = auto()
    # ...

Module

Classes

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# payment.service.py

from nameko.events import EventDispatcher
from nameko.rpc import rpc
from nameko_sentry import SentryReporter

from .dependencies.db import Database
from .dependencies.nameko_stripe import Stripe
from . import exceptions as exc  # Namespaces
from . import schemas
from . import states


PAYMENT_TAKEN_EVENT = "payment_taken"


class PaymentService:

    name = "payment"
    
    # Dependencies
    db = Database()
    dispatch = EventDispatcher()
    sentry = SentryReporter()
    stripe = Stripe()

    # ...

Payment Service

Service class

Constants

Endpoints

    # ...  
    
    @rpc
    def pay_order(self, order_id):
        try:
            # Endpoint does not know about SQL-related objects/actions
            # Interacts with the DB objects through "collections"
            transaction = self.db.transactions.get_by_order_id(order_id)
        # Translate dependency "expected exceptions"
        # Services provide their own exception (contract)
        except self.db.NotFound as err:
            message = f"Unable to pay order ID {order_id}."
            raise exc.TransactionNotFound(message)

        try:
            payment_methods = self.stripe.list_payment_methods(customer_id)
        # Translate dependency "expected exceptions"
        # Services provide their own exception (contract)
        except self.stripe.NotFound as err:
            raise exc.NotFound(err.args[0])
        except self.stripe.PaymentMethodError as err:
            raise exc.BadRequest(err.args[0])

        # ...
        # ...
        
        # Compare `transaction` DB collection with payment state
        if transaction.state is states.PaymentState.PAID:
            raise exc.BadRequest(f"Order {order_id} is already paid.")

        try:
            payment = self.stripe.make_payment(
                customer_id, order_id, payment_method_id
            )
        # Translate dependency "expected exceptions"
        # Services provide their own exception (contract)
        except self.stripe.NotFound as err:
            raise exc.NotFound(err.args[0])

        # Dispatch an event (async) with the payment info
        event_body = {
            "order": {"id": order_id},
            "payment_method": {"id": payment_method_id},
            "project": {"id": transaction.project_id},
            "paid_by": {"id": user_id},
        }
        self.dispatch(PAYMENT_TAKEN_EVENT, event_body)
        
        # Endpoint does not know about formatting/serializing
        # the response
        return schemas.dump(schemas.Payment, payment)

Admin Facade

Payment Service

Messaging Service

/admin_facade

http_errors.py

http_handlers.py

remote_errors.py

schemas.py

service.py

/payment_service

dependencies

db​.py

nameko_stripe.py

exceptions.py

states.py

/messaging_service

service.py

service.py

schemas​.py

models.py

Splitting code into components

User applications

Facades

Service class

Endpoints

Serialization schemas

Service dependencies

3rd party packages

Modules

Methods

Functions

Exceptions

Constants

Domain services

Classes

# messaging.service.py

from nameko.events import event_handler
# Replace with the Nameko "email_provider" dependency of your choice
from nameko_email_provider import EmailProvider
from nameko_sentry import SentryReporter


PAYMENT_SERVICE_NAME = "payment"
PAYMENT_TAKEN_EVENT = "payment_taken"


class MessagingService:

    name = "messaging"
    
    # Dependencies
    email_provider = EmailProvider()
    sentry = SentryReporter()
    
    # ...

Service class

Constants

Endpoints

Messaging Service

Methods

    # ...
    
    @event_handler(PAYMENT_SERVICE_NAME, PAYMENT_TAKEN_EVENT)
    def send_order_paid_confirmation_email(self, payload):
        """Create and send order paid confirmation email.
        
        Not just a dumb and generic "send email" endpoint.
        It contains the logic to generate and send order paid 
        confirmation emails. 
        """
        order = payload["order"]
        project = payload["project"]
        payment_method = payload["payment_method"]
        paid_by = payload["paid_by"]
        
        # ...
        
        self._send_email(recipients, subject, content)
      
    def _send_email(self, recipients, subject, content, **kwargs):
        """Use the email provider dependency to send out emails.
        
        Helper method that acts as a wrapper around the email_provider
        dependency. Another approach could be a new "dependency wrapper".
        Reused by all the endpoints in the service that need to
        send emails.
        """
        # ...
        self.email_provider.send(
            recipients, subject, content, sender, bcc=bcc
        )
 

Complexity metric

●  "Code that fits in your head"

GOTO 2016 - Software that Fits in Your Head - Dan North. Pattern described by Dan North, referring to James Lewis for coming up with the expression

https://www.youtube.com/watch?v=4Y0tOi7QWqM

●  Also refers to a physical head (screen) 🙂

●  "You can only reason about something that fits in your head" 

●  Concentrate in one component at a time (ignore other parts)

●  Fully understand what a component does

●  Consistency:

Code formatting, idioms, component design, contracts, technologies (MySQL, PostgreSQL, MariaDB...)

Resources

https://www.nameko.io

nameko

- Clean Code - Robert C. Martin

- Implementing Domain-Driven Design - Vaughn Vernon

- Domain-Driven Design - Eric Evans

>>> import this
The Zen of Python, by Tim Peters

- Microservices - James Lewis, Martin Fowler

https://martinfowler.com/articles/microservices.html

- GOTO 2016 - Software that Fits in Your Head - Dan North

https://www.youtube.com/watch?v=4Y0tOi7QWqM

Thank you!

juliotrigo

juliotrigo

https://slides.com/juliotrigo/pycones2019-a-practical-ddd-approach-to-nameko-microservices

PyConES2019 - A practical DDD approach to Nameko microservices

By juliotrigo

PyConES2019 - A practical DDD approach to Nameko microservices

  • 1,711