Distributed Tracing
With OpenTelemetry

 

Kirk Haines

 

 

 

 

@wyhaines everywhere

 

About Me

• Wrote my first code in 1981
• DevOps, Systems, and Software Engineering for > 30 Years
• Fast forward to 2021 -- Learned OpenTelemetry to become an implementor:
  • https://github.com/wyhaines/opentelemetry-api.cr
  • https://github.com/wyhaines/opentelemetry-sdk.cr
  • https://github.com/wyhaines/opentelemetry-instrumentation.cr
    
    Fully Usable (and in-use in production systems) SDK and auto-instrumentation for Tracing, with Metrics and Logs in-progress

Just in case...

What Is

• Open source standard for collection and handling of telemetry

What Is

• Open source standard for collection and handling of telemetry
• Gives users vendor-neutral control of their data

What Is

• Open source standard for collection and handling of telemetry
• Gives users vendor-neutral control of their data
• Community developed via contributions from industry and individuals

What Is

Applied OpenTelemetry

 

OpenTelemetry focuses on three telemetry signals

(signal == category of telemetry data defined by the specification)

 

• Traces
• Metrics
• Logs

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

Span

==

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

child

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B                           METHOD_C

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

child

child

Applied OpenTelemetry - Traces

 

 

 

Tracing records code execution paths within an application

 

 

METHOD_A                           METHOD_B                           METHOD_C                           METHOD_D

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

• timings
• attributes
• metadata such as
  • line number
  • method
  • file

child

child

child

Applied OpenTelemetry - Traces

Applied OpenTelemetry - Traces

Applied OpenTelemetry - Traces

 

 

 

 

What happens in a complex execution path, such as a multiservice architecture involving microservices or serverless execution paths?

Tracing Across Multiple Services

==

Distributed Tracing

Distributed Tracing

Distributed Tracing

 

 

 

 

 

 

 

 

How does that work?

Context Propagation

Context Propagation

 

All Spans in a Trace have Context

 

• Trace ID
• Span ID
• Trace Flags

Context Propagation

 

Additionally, Spans have a parent_span_id

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

Specification describing a standard way to pass context to another service

 

Two HTTP headers

• traceparent
• tracestate

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

PARENT_ID == b7ad6b7169203331

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

 

VERSION == 00

TRACE_ID == 0af7651916cd43dd8448eb211c80319c

PARENT_ID == b7ad6b7169203331

TRACE_FLAGS == 01 (* only one bit, for SAMPLED_FLAG, is currently used)

Context Propagation

 

W3C Trace Context 

https://www.w3.org/TR/trace-context/

 

tracestate: foo=bar,this=that

OpenTelemetry is Complicated

OpenTelemetry is Complicated

 

But Distributed Tracing is pretty straightforward

OpenTelemetry is

Complicated

 

But Distributed Tracing is pretty straightforward

Microservice A

Microservice B

Process