Projects Webservices
@7digital

A bit of context

Custom projects

  • Managing several projects simultaneously
  • Projects often need custom APIs:
    oauth, editorial contents, social features...
  • Third-party integrations:
    recommendation engines, external metadata
    (Quantone, Music Story...)

Transitionning

Slowly moving projects to use the London
delivery/reporting/user management platform, but:

  • frontend UIs strongly rely on the Lille's
    resources schema,
  • some projects still run on the legacy
    Paris platform (Pulsar, SnowVaultage).

Contractors

  • for most CSS integrations
  • for some projects when Lille is
    at full capacity
  • for some specific functionalities
    (recommendations)

Project oriented

  • it is more about getting the best
    in the given timeframe than
    targetting the state of the art,
  • need to move fast: 2 weeks sprints.

Modularization

One repository per
concern

  • frontends (bo/www)
  • common webservices (delivery london/lille,
    user management strategies, catalogue...)
  • additionnal webservices (third-party metadatas)

A docker image per repository

  • built and published on each commit to master
    and release tags.
  • self contained excluding configuration according
    to the environement/project that are mounted
    via the infrastructure project

Modularization issues

  • hard to maintain/evolve each repository
    simultaneously,
  • configuration/build drift often leading
    to errors and inconsistencies.

An automated update
process

  • using `metapak` and
    `metapak-7digital-projects`,
  • apply macros changes to each repository:
    git hooks, scripts utilities (testing, deployment),
    assets (dockerfile, CI setup ...)

Automatically Customized Documentation

  • a unique swagger definition for
    each product according to its recipe,
  • build frontend API services on its basis :
    generated via webpack, allow non-breaking
    API changes.

Code
Architecture

Dependency Injection
with Inversion of Control

  • using knifecycle,
  • every application states are enclosed
    into services (easy mocking, reusing, testing),
  • allows graceful shutdowns: high availability,
  • only starts services that are used
  • common services are in `webservices-libs`.

Pure functions FTW

  • basically, `lib/` folders functions have
    to be pure
  • makes testing a breeze
  • code readability and reusability increased

Documentation
Driven APIs

  • creating an endpoint is basically
    done by documenting it
  • enforces documented API, and more
    importantly "thought" APIs.
  • leverage the Swagger force: input/output
    validated automagically (contracts pattern)

Workflow oriented
controllers

  • a controller is basically a promises
    chain describing how a HTTP
    transaction flows,
  • more flexible, changing a controller is just
    about adding/updating/removing a stage
    in the chain

Cache ready

  • enforces unique URLs with
    siso/strict-qs
  • way toward legacy URLs change is
    traced thanks to Swagger and our
    API client generator

Still a WIP

  • not RESTful ATM: endpoints may vary
    according to the JWT, POST instead of PUT
  • not all JSON Schemas are done yet
  • not all routes are fully documented yet
    (a temporary mechanism allow to release
    constraints selectively)
  • need an automated way to sync with DB changes

Deployment

One Git repository to
rule them all

  • named `infrastructure-{projectName}`
  • contains webservices/frontends as submodules
  • hold configurations and recipes to build the
    whole infrastructure project locally
  • deploy them on each master commit for UAT,
    on each release tag for prod

Cloud hosting powers

  • high availability thanks to graceful
    shutdowns and Amazon ECS rolling
    releases
  • platform agnostic thanks to Docker
  • one infrastructure repository; multiple deploy
    targets: scripts to generate PM2, Docker Swarm
    or Amazon ECS infrastructure.
  • logging/monitoring with Datadog/Sumologic

Back to the future

  • Deploy frontends to S3 instead of
    using Docker + NGinX,
  • Leverage Amazon Lamba and API Gateway:
    need a set of script to enable this new
    deployment target,
  • Kubernetes as another deployment target
    (Google app engine switch?).

Quality Insurance

  • Unit testing on services / libs
  • Behavior testing for each endpoints
  • Dry runs for each envs (config checks)
  • Ping through the whole architecture
    via a Docker compose on the target env
  • About 80% of code coveraging for all
    webservices

Thanks

@nfroidure on GitHub and Slack