My prod is down, let's escape from Service Oriented Architecture Hell

Clément Bertillon

https://twitter.com/BERTILLONClment

https://github.com/skigun

Based on a real case

One morning on production:

It doesn't work !

What ? What is wrong ?

The product page of the production

So it come from the Backoffice ?

No, it probably come from the MarketPlace

No it come from our provider app who synchronize data in real time

Yes but no, now that we migrate products on ElasticSearch it should come from the new MicroService.. no ? and why logs say nothing?

The dialogue

It doesn't work !

E-commerce

Backoffice

ElasticSearch

MarketPlace

Product MicroService

CRM

Subvention MicroService

Exemple of interaction between applications

Starting points

  • Priority definition
  • Communication inside and betweens teams
  • Create a ticket
  • Load the ticket somewhere, like a sprint
  • Meanwhile pre-analysis

Pre-analysis

Application Log

Raw Log

Difficult

User Interface test

Access log

Access Log - First Tip

Target a server with a header

curl GET 'https://e-commerce.com' 
  --header 'X-Server-number: 1'

ModHeader

Log - 2nd tip

Warning

  • Always communicate 
  • Always rollback

Results

We know a web page crashes
but we cannot find which application is responsible for

Distributed Tracing

Distributed tracing, also called distributed request tracing, is a method used to profile and monitor applications, mainly those built using a distributed architecture.

 

A more suitable tool for our analysis

Xdebug

Xdebug

Blackfire !

Feature: Distributed Profiling

Is Distributed Tracing !

Profile

Profile

Cache

  • Focus on endpoint without cache
  • Try other environments
  • Use Blackfire SDK to change the behaviour

Issue #1

You can't trace everything

Client HTTP involved : HTTP_Request2

Configure the library with curl (standard)

Inject the blackfire header

in the request
Documentation

Issue #2

https://blog.blackfire.io/the-power-of-distributed-profiling.html

$request = new HttpRequest2Iso(RACINE_BO_MEYCLUB.'synchro_ag.php', 'POST');
// vs
$request = new HttpRequest2Iso(RACINE_BO_MEYCLUB.'synchro_ag.php', 'POST', [
    'adapter'=> 'HTTP_Request2_Adapter_Curl'
]);

How to stop our workflow of http requests

// If the workflow start from application A 
// We want to stop the propagation of Http calls
if ($this->httpTracer->getLastOriginFromRequest($masterRequest)) {
    return;
}

How improve our control over tracing ?

services:
    Monolog\Processor\UidProcessor:
        tags: ['monolog.processor']

Monolog Uid

<?php


class HttpTracer
{
    private const HEADER_ORIGIN = 'trace-origin';
    private const HEADER_ID = 'trace-id';


    public function traceOrigin(&$headers): void
    {
    }

    public function traceId(&$headers): void
    {
        $headers[self::HEADER_ID] ?? $headers[self::HEADER_ID] = md5(random_bytes(10));
    }

    public function getLastOriginFromRequest(Request $request): ?string
    {
    }
}

Introducing HttpTracer

Going further

https://github.com/auxmoney/OpentracingBundle-core

 Tracing::injectTracingHeaders($request): RequestInterface

OpenTracing

OpenTelemetry

High-quality, ubiquitous, and portable telemetry to enable effective observability

How to get a free monitoring tools for dev environment ?

# config/packages/dev/web_profiler.yaml
framework:
    profiler:
        only_exceptions: false
        dsn: 'file:/shared_profiler_directory/var/profiler'
# config/packages/dev/web_profiler.yaml
framework:
    profiler:
        only_exceptions: false
        dsn: 'file:/shared_profiler_directory/var/profiler'

Thank you!

https://twitter.com/BERTILLONClment

https://github.com/skigun

My prod is down, let's escape from Service Oriented Architecture Hell

By skigun

Made with Slides.com

My prod is down, let's escape from Service Oriented Architecture Hell

My prod is down, let's escape from Service Oriented Architecture Hell

  • 647

More from skigun