Observability 2.0

feat.

Observability

 

just a fancy word for

 

Monitoring

?

Wikipedia definition

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

Observability vs Monitoring?

 

or rather

 

Observability through Monitoring

Is observability

a new concept?

Old is new again

Originally introduced by Rudolf E. Kálmán in 1960 for linear dynamic systems.

 

Original meaning largely still relevant for modern software.

Signal types

Logs

Metrics

Traces

Signal types

Logs

Metrics

Traces

Logs

Tell me what you are doing.

2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto 
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL:  remaining connection slots are reserved
...

Signal types

Logs

Metrics

Traces

Metrics

Don't need to grep your logs, look at the metrics

Signal types

Logs

Metrics

Traces

Traces

  • correlationID     
    • show me all the calls from this request

A couple of years ago in microservices not that far away...

 

  • requestID
    • show me which service and for how long called which

Traces

  • correlationID trace
    • show me all the calls from this request

A couple of years ago in microservices not that far away...

Today. Here.

  • requestID span
    • show me which service and for how long called which

Traces

Service A

Service B

Database

Client

Traces

A

B

DB

HTTP GET

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0333

Parent Span ID: 0x0222

Name: /order

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0333

Parent Span ID: 0x0222

Name: /order

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0444

Parent Span ID: 0x0333

Name: INSERT INTO TABLE

Status: OK (...)

HTTP POST

DB CLIENT

The same metadata for logs, metrics and traces.

.

Choose your project...

Prometheus

Grafana

Fluentd

Fluent Bit

Jaeger

Zipkin

OpenTracing

Vector

Collectd

Logstash

...

Observability 1.0

Vendor provided

Vendor provided

Vendor provided

Vendor provided

Vendor provided

Vendor provided 

Vendor provided

Vendor provided

Vendor provided

Instrumentation

Collection

Backend

OpenCensus

  • metrics and tracing focused
  • based on the Census concepts

OpenTracing

  • distributed tracing focused
  • based on the Dapper concepts
  • CNCF project since 2016
  • API used by many vendors (Jaeger, DataDog, etc.)

OpenTelemetry

  • OpenCensus + OpenTracing
  • announced May 2019
  • CNCF project - incubating since Aug 2021
  • Backed by all major vendors

Choose your project...

Prometheus

Grafana

Fluentd

Fluent Bit

Jaeger

Zipkin

OpenTracing

Vector

Collectd

Logstash

OpenTelemetry

How is it different this time?

Observability 1.0

Instrumentation

Collection

Backend

Observability 1.0

Observability 2.0

vs

(work in progress)

Observability 1.0

Observability 2.0

Vendor provided

OpenTelemetry provided

OpenTelemetry provided

vs

(work in progress)

Observability 1.0

Observability 2.0

vs

Instrumentation

(work in progress)

Observability 1.0

Observability 2.0

vs

Collection

(work in progress)

Observability 1.0

Observability 2.0

vs

Backend

(work in progress)

Observability 2.0

(work in progress)

Components

Components

Resource semantic conventions

OpenTelemetry Protocol - OTLP

General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.

Specified for both gRPC and HTTP transports.

Components

OpenTelemetry Language SDKs

Components

OpenTelemetry Collector

Receivers

Processors

Exporters

OpenTelemetry Collector

Receivers

Processors

Exporters

Pipeline

OpenTelemetry Collector

aka otelcol

Host Metrics Receiver

Example

receivers:
  hostmetrics:
    scrapers:
      memory:

processors:
  resourcedetection/detect-host-name:
    detectors:
    - system
    system:
      hostname_sources:
      - os

exporters:
  otlp:
    endpoint: otelcol2:4317

service:
  pipelines:
    metrics:
      receivers:
      - hostmetrics
      processors:
      - resourcedetection/detect-host-name
      exporters:
      - otlp

OT Collector distros

Custom distributions

Community

Community - SIGs

Community - repositories

.

Contributing

If you look for high impact project

If you want to contribute to Open Source

If you like to work with supporting community

If you are interested in telemetry

Some more #OpenTelemetry

Thank you!

Marcin Stożek "Perk"

@marcinstozek / perk.pl

Observability 2.0 feat. OpenTelemetry

By Marcin Stożek

Observability 2.0 feat. OpenTelemetry

  • 963