Clean architecture in Python REST API’s: Quick start

DISCLAIMERS

Disclaimer #1

This presentation is not a One way fits all set for rules, neither is a theory 'class' about clean arquitecture

is a group of quick/simple approaches to get some benefits of clean architecture

Pragmatism, trade offs

https://www.slideshare.net/mattiabattiston/real-life-clean-architecture-61242830

Disclaimer #2

Oriented to microservices

Tradeoffs: some Clean Architecture parts might represent more cost than benefit

Clean Architecture

Clean architecture chatacteristics

  • Independent of frameworks
  • Testable.
  • Independent of the UI.
  • Independent of the database.
  • Independent of any external agency.

Clean architecture

What the ...?

Let's start simple

Cutest furry little API endpoint

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = db_session().query(Application).get(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)

Does it work?

It does :)

Does it apply clean architecture?

No

Is that a bad thing?

We are fine for now :)

Do we need to apply clean architecture?

Probably not

But...

The API start growing

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = db_session().query(Application).get(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)


@app.route('/application')
def get_user_applications():
    user_id = request.args.get('user_id')
    application = db_session().query(Application).fliter_by(user_id=user_id)
    return jsonify(ApplicationSchema(many=True).dump(application).data)


@app.route('/application/<int:application_id>/process')
def process_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    if application.requested_amount > 50000:
        application.status = 'DECLINE'
        notify_declined_application()
    else
        application.status = 'APPROVE'
        notify_approved_application()
    session.commit()
    return jsonify()


@app.route('/application/<int:application_id>/decline')
def approve_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    application.status = 'DECLINE'
    notify_declined_application()
    session.commit()
    return jsonify()

Lets take another look

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = db_session().query(Application).get(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)


@app.route('/application')
def get_user_applications():
    user_id = request.args.get('user_id')
    application = db_session().query(Application).fliter_by(user_id=user_id)
    return jsonify(ApplicationSchema(many=True).dump(application).data)


@app.route('/application/<int:application_id>/process')
def process_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    if application.requested_amount > 50000:
        application.status = 'DECLINE'
        notify_declined_application()
    else
        application.status = 'APPROVE'
        notify_approved_application()
    session.commit()
    return jsonify()


@app.route('/application/<int:application_id>/decline')
def approve_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    application.status = 'DECLINE'
    notify_declined_application()
    session.commit()
    return jsonify()

Jumm, this smells fishy

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = db_session().query(Application).get(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)


@app.route('/application')
def get_user_applications():
    user_id = request.args.get('user_id')
    application = db_session().query(Application).fliter_by(user_id=user_id)
    return jsonify(ApplicationSchema(many=True).dump(application).data)


@app.route('/application/<int:application_id>/process')
def process_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    if application.requested_amount > 50000:
        application.status = 'DECLINE'
        notify_declined_application()
    else
        application.status = 'APPROVE'
        notify_approved_application()
    session.commit()
    return jsonify()


@app.route('/application/<int:application_id>/decline')
def approve_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    application.status = 'DECLINE'
    notify_declined_application()
    session.commit()
    return jsonify()

*** 1

*** 2

*** 3

*** 4

***1, ***3, ***4

are exactly the same queries

***1, ***2, ***3, ***4 they are all queries to the same table

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = db_session().query(Application).get(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)


@app.route('/application')
def get_user_applications():
    user_id = request.args.get('user_id')
    application = db_session().query(Application).fliter_by(user_id=user_id)
    return jsonify(ApplicationSchema(many=True).dump(application).data)


@app.route('/application/<int:application_id>/process')
def process_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    if application.requested_amount > 50000:
        application.status = 'DECLINE'
        notify_declined_application()
    else
        application.status = 'APPROVE'
        notify_approved_application()
    session.commit()
    return jsonify()


@app.route('/application/<int:application_id>/decline')
def approve_application(application_id):
    session = db_session()
    application = session.query(Application).get(application_id)
    application.status = 'DECLINE'
    notify_declined_application()
    session.commit()
    return jsonify()

*** 1

*** 2

*** 3

*** 4

There must be a better way !!

Raymond Hettinger @raymondh Python core developer.

Freelance programmer/consultant/trainer.

Lets move data access to a common place

# application_manager.py

def __get_application_by_id(session, application_id):
    return session.query(Application).get(application_id)


def get_application(application_id):
    return _get_application_by_id(db_session(), application_id)


def retrieve_user_applications(user_id):
    return db_session().query(Application).fliter_by(user_id=user_id)


def approve_application(application_id):
    session = db_session()
    with session.begin():
        application = __get_application_by_id(session, application_id)
        application.status = 'APPROVE'


def decline_application(application_id):
    session = db_session()
    with session.begin():
        application = __get_application_by_id(session, application_id)
        application.status = 'DECLINE'

Now our API endpoints have nothing to do with database sessions

# api/application.py

@app.route('/application/<int:application_id>')
def get_application(application_id):
    application = get_application(application_id)
    return jsonify(ApplicationSchema(many=False).dump(application).data)


@app.route('/application')
def get_user_applications():
    user_id = request.args.get('user_id')
    application = get_user_applications(user_id)
    return jsonify(ApplicationSchema(many=True).dump(application).data)


@app.route('/application/<int:application_id>/process')
def process_application(application_id):
    application = get_application(application_id)
    if application.requested_amount > 50000:
        decline_application(application_id)
        notify_declined_application()
    else:
        application.status = 'APPROVE'
        notify_approved_application()
    return jsonify()


@app.route('/application/<int:application_id>/decline')
def approve_application(application_id):
    approve_application(application_id)
    notify_declined_application()
    return jsonify()

*** 1

*** 2

*** 3

*** 4

Project structure

Improvements list

Dedicated place to interact with database

Principle involved

Single responsibility principle

The App keeps growing

We need to decline an application via command line

Ideas?

We already have an API endpoint, may be we can call it?

/application/<int:application_id>/decline

NO!!!!

Lets use a  script for it

Hold on

@app.route('/application/<int:application_id>/decline')
def decline_application(application_id):
    decline_application(application_id)
    notify_declined_application()
    return jsonify()

How can my current logic be used by a script?

It can't

@app.route('/application/<int:application_id>/decline')
def declined_application(application_id):
    declined_application(application_id)
    notify_declined_application()
    return jsonify()

The http protocol access is coupled to the business logic

*** 1

*** 2

There must be a better way !!

Raymond Hettinger @raymondh Python core developer.

Freelance programmer/consultant/trainer.

Lets take the business logic appart

# use_cases/application/decline_application_use_case.py

from database.managers.application_manager import decline_application

class DeclineApplicationUseCase:
  
    def __notify_declined_application():
        # add logic to notify declined application
        pass


    def decline(self, application_id):
        decline_application(application_id)
        self.__notify_declined_application()

Adapters

# api/application.py

@app.route('/application/<int:application_id>/decline')
def decline_application(application_id):
    DeclineApplicationUseCase.decline(application_id)
    return jsonify()

The endpoint is a REST API adapter that calls the Decline Application use case

# script/decline_application.py

import argparse

from use_case.application_use_cases import decline_application_use_case

parser = argparse.ArgumentParser(description='Decline given application.')
parser.add_argument('application_id', metavar='a', type=int,
                    help='id of the application to be declined')


def main():
    args = parser.parse_args()
    DeclineApplicationUseCase.decline(args.application_id)

if __name__ == "__main__":
    main()

The script is a command line adapter that calls the Decline Application use case

Adapters

Improvements list

Dedicated place to interact with database

Dedicated place for business logic

Dedicated place for API access

Dedicated place for Scripts

Decoupled business logic from http access

Project structure

Principle involved

Single responsibility principle

Again :)

Jobs

Crossing Boundaries

About parameters and returns....

# database/managers/application_manager.py


session_factory = sessionmaker(bind=engine)
db_session = scoped_session(session_factory)


def __get_application_by_id(session, application_id):
    return session.query(Application).get(application_id)


def get_application(application_id):
    return __get_application_by_id(db_session(), application_id)


def get_user_applications(user_id):
    return db_session().query(Application).fliter_by(user_id=user_id)

*** 2

*** 3

*** 1

*** 2 and ***3 return type is an Application Model

This is not good :(

Why?

Other modules that use data would "know" that an Application Model exist :(

# use_cases/application/application_use_case.py


def process_application_use_case(application_id):
    application = get_application(application_id)
    if application.requested_amount > 50000:
        decline_application(application_id)
        __notify_declined_application()
    else:
        approve_application(application_id)
        __notify_approved_application()

*** 2

This use case knows application is a model instance with requested_amount attribute

The data from/to modules should be simple data structures

  • Don't pass ORM entities or database rows.
  • Simple data structures: Dict, list, string, int, float, boolean etc...

So they are agnostic to the implementation details/technologies of other modules

No Application Models or any Other Model should be returned

# database/managers/application_manager.py


session_factory = sessionmaker(bind=engine)
db_session = scoped_session(session_factory)


def __get_application_by_id(session, application_id):
    return session.query(Application).get(application_id)


def get_application(application_id):
    return __get_application_by_id(db_session(), application_id)


def get_user_applications(user_id):
    return db_session().query(Application).fliter_by(user_id=user_id)

***2

***3

# database/managers/application_manager.py


from database.models import Application, engine
from database.schemas import ApplicationSchema

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker

session_factory = sessionmaker(bind=engine)
db_session = scoped_session(session_factory)


def __get_application_by_id(session, application_id):
    return session.query(Application).get(application_id)


def get_application(application_id):
    session = db_session()
    application = __get_application_by_id(session, application_id)
    return ApplicationSchema(many=False).dump(application).data


def get_user_applications(user_id):
    application = db_session().query(Application).fliter_by(user_id=user_id)
    return ApplicationSchema(many=True).dump(application).data

***2

***3

Schemas to the rescue

from marshmallow_enum import EnumField
from marshmallow_sqlalchemy import ModelSchema


from database.models import Application, ApplicationStatus


class ApplicationSchema(ModelSchema):
    status = EnumField(ApplicationStatus)

    class Meta:
        model = Application

Now the use case would handle a dict instead of an Application model

# use_cases/application/application_use_case.py


def process_application_use_case(application_id):
    application = get_application(application_id)
    if application['requested_amount'] > 50000:
        decline_application(application_id)
        __notify_declined_application()
    else:
        approve_application(application_id)
        __notify_approved_application()

Project structure

Jobs

The App is growing even more

Applications by  companies is needed

@app.route('/application')
def get_applications():
    applications = []
    
    # Filter application by users
    if 'user_id' in request.args:
        user_id = request.args.get('user_id')
        applications = GetUserApplicationsUseCase().get_applications(user_id)
    
    # Filter applications by company
    elif 'company_id' in request.args:
        company_id = request.args.get('company_id')
        applications = GetCompanyApplicationsUseCase().get_applications(company_id)
        
    return jsonify(applications)

This works :)

But

A user usually have 10's or 100's of applications

While a company might have 10k's - 100k's - 1M's of applications

That means some fun with the db:

Indexes
Sharding

Data replication in other tables

Sometimes, that is not enough or too much work

Sometimes, the fastest solution is to handle a different data source for that specific query

For example mongo or redis.

http://hannesdorfmann.com/android/evolution-of-the-repository-pattern

# database/managers/application_sql_manager.py

from database.models import Application, engine
from database.schemas import ApplicationSchema

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker


class ApplicationSQLManager:

    def __init__(self):
        session_factory = sessionmaker(bind=engine)
        db_session = scoped_session(session_factory)
        self.session = db_session()

    ...
    ...

    def get_company_applications(self, company_id):
        application = self.session.query(Application).fliter_by(company_id=company_id)
        return ApplicationSchema(many=True).dump(application).data

    ...
    ...
# database/managers/application_document_manager.py

from pymongo import MongoClient


class ApplicationDocumentManager:

    def __init__(self):
        db_url = 'mongodb://mydbuser:mydbpassword@ds053148.mongolab.com:53148/heroku_app18934798'
        self.collection = MongoClient(db_url)['my_database']['application']

    def get_application(self, application_id):
        raise NotImplementedError

    def get_user_applications(self, user_id):
        raise NotImplementedError

    def get_company_applications(self, company_id):
        fields_object = {'_id': 1, 'company_id': 1, 'created_at': 1, 'user_id': 1}
        return list(self.collection.find({'company_id': company_id}, fields_object))

    def approve_application(self, application_id):
        raise NotImplementedError

    def decline_application(self, application_id):
        raise NotImplementedError
# database/repositories/application_repository.py

from database.managers.application_sql_manager import ApplicationSQLManager


class ApplicationRepository:

	def __init__(self, default_manager=ApplicationSQLManager):
		self.manager = default_manager()

	def get_application_by_company(self, company_id):
		return self.manager.get_company_applications(company_id)

https://8thlight.com/blog/mike-ebert/2013/03/23/the-repository-pattern.html

# database imports
from database.managers.application_sql_manager import ApplicationSQLManager
from database.managers.application_document_manager import ApplicationDocumentManager
from database.repositories.application_repository import ApplicationRepository

## import GetApplicationsUseCase and GetCompanyApplicationsUseCase
# ...
# ...

@app.route('/application')
def get_applications():
    applications = []
    
    # Filter application by users
    if 'user_id' in request.args:
        user_id = request.args.get('user_id')
        applications = GetUserApplicationsUseCase(ApplicationRepository(ApplicationSQLManager()).get_applications(user_id)
    
    # Filter applications by company
    elif 'company_id' in request.args:
        company_id = request.args.get('company_id')
        applications = GetCompanyApplicationsUseCase(ApplicationRepository(ApplicationDocumentManager()).get_applications(company_id)
        
    return jsonify(applications)

Applied technique:

Dependency injection

One object supplies the dependencies of another object

So, instead of a class calling/importing its dependencies...

... the class dependencies are sent as parameters

# database imports
from database.managers.application_sql_manager import ApplicationSQLManager
from database.managers.application_document_manager import ApplicationDocumentManager
from database.repositories.application_repository import ApplicationRepository

## import GetUserApplicationsUseCase and  GetCompanyApplicationsUseCase
# ...
# ...

@app.route('/application')
def get_applications():
    applications = []
    
    # Filter application by users
    if 'user_id' in request.args:
        user_id = request.args.get('user_id')
        applications = GetUserApplicationsUseCase(ApplicationRepository(ApplicationSQLManager()).get_applications(user_id)
    
    # Filter applications by company
    elif 'company_id' in request.args:
        company_id = request.args.get('company_id')
        company_application_use_case = GetCompanyApplicationsUseCase(ApplicationRepository(ApplicationDocumentManager())
        applications = company_application_use_case.get_applications(company_id)                                                      
                                                                     
                                                                     
    return jsonify(applications)

*** 1

*** 2

***1 ApplicationDocumentManager is sent "injected" to ApplicationRepository

***2 ApplicationRepository is sent "injected" to GetCompanyApplicationsUseCase

***3 GetCompanyApplicationsUseCase object is being created

TODO: Inversion of Control for GetCompanyApplicationsUseCase

*** 3

*** 4

***4 Applications are retrieved

TODO: Inversion of Control for GetCompanyApplicationsUseCase?

Ok quick pick :P

# database imports
from database.managers.application_sql_manager import ApplicationSQLManager
from database.managers.application_document_manager import ApplicationDocumentManager
from database.repositories.application_repository import ApplicationRepository

# Use cases imports
# ...
# ...

import dependency_injector.containers as containers
import dependency_injector.providers as providers


class Managers(containers.DeclarativeContainer):
    """IoC container of Managers """

    applicationDocumentManager = providers.Factory(ApplicationDocumentManager)


class Repositories(containers.DeclarativeContainer):
    """IoC container of Repositories """

    applicationDocumentRepository = providers.Factory(ApplicationRepository, default_manager=Managers.applicationDocumentManager)


class UseCases(containers.DeclarativeContainer):
    """IoC container of Adapters """
    companyApplicationUseCase = providers.Factory(GetCompanyApplicationsUseCase, applicatiomn_repository=Repositories.applicationDocumentRepository)


@app.route('/application')
def get_applications():
    applications = []

    ...

    # Filter applications by company
    elif 'company_id' in request.args:
        company_id = request.args.get('company_id')
        company_application_use_case = UseCases.companyApplicationUseCase()
        applications = company_application_use_case.get_list_by_company(company_id)
    return jsonify(applications)

*** 1

***No more those "nested" parameters

application_adapter = CompanyApplicationsAdapter(ApplicationRepository(ApplicationDocumentManager()))

https://pypi.org/project/dependency-injector/

Use Case


class GetCompanyApplicationsUseCase:

    def __init__(self, application_repository):
        self.repository = application_repository

    def get_company_applications(self, company_id):
        return self.repository.get_company_applications(company_id)

It doesn't not import any repository or manager, and yet, is used in execution time.

Dependency rule compliance

 no code in the inner circle can directly reference a piece of code from the outer circle

Adapters don't import endpoints

Use cases don't import Adapters

Project structure

Anything else?

Or course, we are just starting....

What is left?

Presenters

Controllers

Gateways

Interactors: Flow of control

Entities

Boundary Anatomy

Component Cohesion and Coupling

Main

Tests

More...

But with the already explained cases, your new born API could be in a good shape for a while

Final thoughts

Should  you apply all Clean architecture 'techniques' to your project?

Of course not, each project has it's own problems.

Overarchitecture could be very hamful and delay stuff

For start ups: Do not try yo anticipate a scalability problem when you dong even have a single client/user

Once you get users, you would gradually apply some techniques to some parts of the code: do not refactor everything overnight

"The squeaky wheel gets the oil"

A good software architect should fully understand the business, market, project, team, deadlines, etc before working on 'latest', 'trendy' and 'cool' techniques

Technical Debt In Software – The Fine Line Between No Architecture and Over Architecture

Don’t Let Architecture Astronauts Scare You

So, what should I do?

You should understand the techniques and the problems they solve

Then apply them once you see you project is facing or could face the problem *soon.

Resources

THE EVOLUTION OF THE REPOSITORY PATTERN - BE AWARE OF OVER ABSTRACTION

Repository Pattern, Done Right

Clean architectures in Python: a step-by-step example

Hexagonal Architecture

Clean Architecture: Dynamic Parameters in Use Cases

Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series)

Clean Architecture Python (web) apps

Android clean architecture

Real Life Clean Architecture

SOCIAL

Jhon Jairo Roa Acuña

Twitter: @jhonjairoroa87

 

https://www.linkedin.com/in/johnroa27/

Thanks

Made with Slides.com