Observability 2.0
feat.
Observability
just a fancy word for
Monitoring
?
Wikipedia definition
Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
Observability vs Monitoring?
or rather
Observability through Monitoring
Is observability
a new concept?
Old is new again
Originally introduced by Rudolf E. Kálmán in 1960 for linear dynamic systems.
Original meaning largely still relevant for modern software.
Signal types
Logs
Metrics
Traces
Signal types
Logs
Metrics
Traces
Logs
Tell me what you are doing.
2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL: remaining connection slots are reserved
...Signal types
Logs
Metrics
Traces
Metrics
Don't need to grep your logs, look at the metrics
Signal types
Logs
Metrics
Traces
Traces
- correlationID
- show me all the calls from this request
A couple of years ago in microservices not that far away...
- requestID
- show me which service and for how long called which
Traces
-
correlationIDtrace- show me all the calls from this request
A couple of years ago in microservices not that far away...
Today. Here.
-
requestIDspan- show me which service and for how long called which
Traces
Service A
Service B
Database
Client
Traces
A
B
DB
HTTP GET
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0444
Parent Span ID: 0x0333
Name: INSERT INTO TABLE
Status: OK (...)
HTTP POST
DB CLIENT
The same metadata for logs, metrics and traces.
.
Choose your project...
Prometheus
Grafana
Fluentd
Fluent Bit
Jaeger
Zipkin
OpenTracing
Vector
Collectd
Logstash
...
Observability 1.0
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Vendor provided
Instrumentation
Collection
Backend
OpenCensus
- metrics and tracing focused
- based on the Census concepts
OpenTracing
- distributed tracing focused
- based on the Dapper concepts
- CNCF project since 2016
- API used by many vendors (Jaeger, DataDog, etc.)
OpenTelemetry
- OpenCensus + OpenTracing
- announced May 2019
- CNCF project - incubating since Aug 2021
- Backed by all major vendors
Choose your project...
Prometheus
Grafana
Fluentd
Fluent Bit
Jaeger
Zipkin
OpenTracing
Vector
Collectd
Logstash
OpenTelemetry
How is it different this time?
Observability 1.0
Instrumentation
Collection
Backend
Observability 1.0
Observability 2.0
vs
(work in progress)
Observability 1.0
Observability 2.0
Vendor provided
OpenTelemetry provided
OpenTelemetry provided
vs
(work in progress)
Observability 1.0
Observability 2.0
vs
Instrumentation
(work in progress)
Observability 1.0
Observability 2.0
vs
Collection
(work in progress)
Observability 1.0
Observability 2.0
vs
Backend
(work in progress)
Observability 2.0
(work in progress)
Components
Components
-
Guidelines, API, SDK, semantic conventions
-
OpenTelemetry Enhancement Proposals (OTEPS)
-
Language independent interface types (proto)
Resource semantic conventions
OpenTelemetry Protocol - OTLP
General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.
Specified for both gRPC and HTTP transports.
Components
Components
OpenTelemetry Collector
Receivers
Processors
Exporters
OpenTelemetry Collector
Receivers
Processors
Exporters
Pipeline
OpenTelemetry Collector
aka otelcol
Host Metrics Receiver
Example
receivers:
hostmetrics:
scrapers:
memory:
processors:
resourcedetection/detect-host-name:
detectors:
- system
system:
hostname_sources:
- os
exporters:
otlp:
endpoint: otelcol2:4317
service:
pipelines:
metrics:
receivers:
- hostmetrics
processors:
- resourcedetection/detect-host-name
exporters:
- otlpOT Collector distros
Core distribution
Contrib distribution
https://github.com/open-telemetry/opentelemetry-collector-contrib
Custom distributions
- Sumo Logic
- AWS
- ...
Community
Community - SIGs
Community - repositories
.
Contributing
If you look for high impact project
If you want to contribute to Open Source
If you like to work with supporting community
If you are interested in telemetry
Some more #OpenTelemetry
- opentelemetry.io
- cncf.io/projects/opentelemetry
- github.com/open-telemetry
-
github/SumoLogic-Labs/opentelemetry-workshop
credits to Przemek Maciołek from Sumo Logic
for his presentation on youtube (in English)
Thank you!
Marcin Stożek "Perk"
@marcinstozek / perk.pl