Distributed Tracing
With OpenTelemetry

 

Kirk Haines

 

 

 

 

@wyhaines everywhere

 

About Me

  • Wrote my first code in 1981
  • DevOps, Systems, and Software Engineering for > 30 Years
  • Fast forward to 2021 -- Learned OpenTelemetry to become an implementor:
    • https://github.com/wyhaines/opentelemetry-api.cr
    • https://github.com/wyhaines/opentelemetry-sdk.cr
    • https://github.com/wyhaines/opentelemetry-instrumentation.cr

      Fully Usable (and in-use in production systems) SDK and auto-instrumentation for Tracing, with Metrics and Logs in-progress

Just in case...

What Is

  • Open source standard for collection and handling of telemetry

What Is

  • Open source standard for collection and handling of telemetry
  • Gives users vendor-neutral control of their data

What Is

  • Open source standard for collection and handling of telemetry
  • Gives users vendor-neutral control of their data
  • Community developed via contributions from industry and individuals

What Is

Applied OpenTelemetry

 

OpenTelemetry focuses on three telemetry signals

(signal == category of telemetry data defined by the specification)

 

  • Traces
  • Metrics
  • Logs

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A

  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A

  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file

Span

==

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B

  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file

child

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B                           METHOD_C

  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file

child

child

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B                           METHOD_C                           METHOD_D

  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file
  • timings
  • attributes
  • metadata such as
    • line number
    • method
    • file

child

child

child

Applied OpenTelemetry - Traces

Applied OpenTelemetry - Traces

Applied OpenTelemetry - Traces

 

 

 

 

What happens in a complex execution path, such as a multiservice architecture involving microservices or serverless execution paths?

Tracing Across Multiple Services

==

Distributed Tracing

Distributed Tracing

Distributed Tracing

 

 

 

 

 

 

 

 

How does that work?

Context Propagation

Context Propagation

 

All Spans in a Trace have Context

 

  • Trace ID
  • Span ID
  • Trace Flags

Context Propagation

 

Additionally, Spans have a parent_span_id

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

Specification describing a standard way to pass context to another service

 

Two HTTP headers

  • traceparent
  • tracestate

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

PARENT_ID == b7ad6b7169203331

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

PARENT_ID == b7ad6b7169203331

TRACE_FLAGS == 01 (* only one bit, for SAMPLED_FLAG, is currently used)

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

tracestate: foo=bar,this=that

OpenTelemetry is Complicated

OpenTelemetry is Complicated

 

But Distributed Tracing is pretty straightforward

OpenTelemetry is

Complicated

 

But Distributed Tracing is pretty straightforward

HTTP (traceparent: context)

HTTP (traceparent: context)

Microservice A

Microservice B

Process

Thank You!

Questions?