Java feat.

tl;dr

We need two things:

 

gRPC example code

 

OpenTelemetry Java Agent

~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world

~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
Sep 24, 2022 1:50:09 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
 

Server

Client

Unchanged gRPC example

~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging

Add the OpenTelemetry Java agent

~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:07:51:024 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:07:53 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051

Server

~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
[otel.javaagent 2022-09-24 14:27:21:991 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:27:24 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
[otel.javaagent 2022-09-24 14:27:25:223 +0200] [main] INFO io.opentelemetry.expor
ter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello' : 01aceb203db607f
b1219543673ed742c eea17e747f1cea5b CLIENT [tracer: io.opentelemetry.grpc-1.6:1.18
.0-alpha] AttributesMap{data={net.transport=ip_tcp, net.peer.name=localhost, net.
peer.port=50051, rpc.service=helloworld.Greeter, rpc.method=SayHello, thread.id=1
, rpc.system=grpc, thread.name=main, rpc.grpc.status_code=0}, capacity=128, total
AddedValues=9}
Sep 24, 2022 2:27:25 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world

Client

~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:26:45:737 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:26:51 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
[otel.javaagent 2022-09-24 14:27:25:196 +0200] [grpc-default-executor-0] INFO io.
opentelemetry.exporter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello
' : 01aceb203db607fb1219543673ed742c 8479d2d44df0f580 SERVER [tracer: io.opentele
metry.grpc-1.6:1.18.0-alpha] AttributesMap{data={net.host.name=localhost, net.tra
nsport=ip_tcp, net.host.port=50051, rpc.service=helloworld.Greeter, net.sock.peer
.addr=127.0.0.1, rpc.method=SayHello, thread.id=16, net.sock.peer.port=64271, rpc
.system=grpc, thread.name=grpc-default-executor-0, rpc.grpc.status_code=0}, capac
ity=128, totalAddedValues=11}

Server

tl;dr end

(stay tuned if you want more)

Telemetry 101

Signal types

Logs

Metrics

Traces

Signal types

Logs

Metrics

Traces

Logs

Tell me what you are doing.

2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto 
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL:  remaining connection slots are reserved
...

Signal types

Logs

Metrics

Traces

Metrics

Don't need to grep your logs, look at the metrics

Signal types

Logs

Metrics

Traces

Traces

Service A

Service B

Database

Client

Traces

A

B

DB

HTTP GET

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0333

Parent Span ID: 0x0222

Name: /order

Status: OK (...)

HTTP POST

DB CLIENT

Traces

A

B

DB

HTTP GET

Trace ID: 0x0123

Span ID: 0x0111

Name: HTTP GET (remote)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0222

Parent Span ID: 0x0111

Name: HTTP POST (client)

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0333

Parent Span ID: 0x0222

Name: /order

Status: OK (...)

Trace ID: 0x0123

Span ID: 0x0444

Parent Span ID: 0x0333

Name: INSERT INTO TABLE

Status: OK (...)

HTTP POST

DB CLIENT

The same metadata for logs, metrics and traces.

.

What is

OpenTelemetry

?

Vendor provided

OpenTelemetry provided

OpenTelemetry provided

Components

Components

Guidelines - cross language requirements and expectations for all implementations

 

API, SDK

 

Semantic conventions

 

OTLP

Resource semantic conventions

OpenTelemetry Protocol - OTLP

General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.

Specified for both gRPC and HTTP transports.

Data Model Layers

OTLP Batch

Components

OpenTelemetry Language SDKs

Manual

 

Instrumentation

Manually setup SDK

Resource resource = Resource.getDefault()
  .merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "logical-service-name")));

SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
  .addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
  .setResource(resource)
  .build();

SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
  .registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().build()).build())
  .setResource(resource)
  .build();

OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
  .setTracerProvider(sdkTracerProvider)
  .setMeterProvider(sdkMeterProvider)
  .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
  .buildAndRegisterGlobal();

... or use SDK auto-configuration

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
</dependency>

and

OpenTelemetrySdk sdk = AutoConfiguredOpenTelemetrySdk.initialize()
    .getOpenTelemetrySdk();

Logs

Metrics

Traces

Manual Instrumentation

Logs

Metrics

Traces

Manual Instrumentation

Logs

Status: experimental

 

Embrace existing logging solutions

 

SDK intended for logging libraries only

 

Not intended to be used by the end user directly

Logs

Metrics

Traces

Manual Instrumentation

Metrics

OpenTelemetry openTelemetry = // obtain instance of OpenTelemetry

// Gets or creates a named meter instance
Meter meter = openTelemetry.meterBuilder("instrumentation-library-name")
        .setInstrumentationVersion("1.0.0")
        .build();

// Build counter e.g. LongCounter
LongCounter counter = meter
      .counterBuilder("processed_jobs")
      .setDescription("Processed jobs")
      .setUnit("1")
      .build();

// It is recommended that the API user keep a reference
// to Attributes they will record against
Attributes attributes = Attributes.of(stringKey("Key"), "SomeWork");

// Record data
counter.add(123, attributes);

Logs

Metrics

Traces

Manual Instrumentation

Traces

Tracer tracer =
    openTelemetry.getTracer("instrumentation-library-name", "1.0.0");

Span span = tracer.spanBuilder("my span").startSpan();

// Make the span the current span
try (Scope ss = span.makeCurrent()) {
  // In this scope, the span is the current/active span
} finally {
    span.end();
}

Automatic

 

Instrumentation

Java agent

$ java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar

Java 8+ compatible

 

Dynamically injects bytecode

 

Highly configurable

Libs, frameworks

Supporting lot of libraries, frameworks, app servers and JVMs:

 

Akka, Apache Camel, AWS SDK, Grails, gRPC, Hibernate, RxJava, Spring, Undertow, Vert.x (and more)

 

Jetty, Tomcat, Websphere, Wildfly (and more)

 

OpenJDK, OpenJ9 (Ubuntu, Windows)

Libs, frameworks

Agent config

System properties , env variables , configuration file

 

Will sanitise DB statements by default

 

Can suppress agent instrumentation for specific library only

 

Many other options - full config here

Manual instrumentation
with annotations

$ java \
  -Dotel.instrumentation.common.default-enabled=false \
  -Dotel.instrumentation.opentelemetry-api.enabled=true \
  -Dotel.instrumentation.opentelemetry-annotations.enabled=true \
  ...
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;

public class MyClass {

    @WithSpan
    public void myMethod(@SpanAttribute("parameter1") String parameter1,
        @SpanAttribute("parameter2") long parameter2) {
        <...>
    }
}
import io.opentelemetry.api.trace.Span;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;


public class Controller {

  private static final Logger LOGGER = LogManager.getLogger(Controller.class);
  ...
  
  private void doWork(int sleepTime) throws InterruptedException {
    Span span = tracer.spanBuilder("doWork").startSpan();
    try (Scope ignored = span.makeCurrent()) {
      Thread.sleep(sleepTime);
      LOGGER.info("A sample log message!");
    } finally {
      span.end();
    }
  }
}

Correlate logs and traces - attributes

2022-09-27 19:58:02.931  INFO 6 --- [nio-8080-exec-1] i.o.example.javagent.Controller          : A sample log message!

Correlate logs and traces - attributes

2022-09-27T19:58:03.277Z	info	ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.12.0
Resource labels:
     -> host.arch: STRING(aarch64)
     -> host: STRING(f566af917d1f)
     -> os.description: STRING(Linux 5.10.124-linuxkit)
     -> os.type: STRING(linux)
     -> process.command_line: STRING(/opt/java/openjdk:bin:java -javaagent:/opentelemetry-javaagent.jar)
     -> process.executable.path: STRING(/opt/java/openjdk:bin:java)
     -> process.pid: INT(6)
     -> process.runtime.description: STRING(Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.16.1+1)
     -> process.runtime.name: STRING(OpenJDK Runtime Environment)
     -> process.runtime.version: STRING(11.0.16.1+1)
     -> service: STRING(agent-example-app)
     -> telemetry.auto.version: STRING(1.17.0)
     -> telemetry.sdk.language: STRING(java)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.17.0)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope io.opentelemetry.example.javagent.Controller
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
Severity: INFO
Body: A sample log message!
Trace ID: 6fe5dff61c1c88ad54e488083a24e787
Span ID: 59a214b6de1c6c13

Correlate logs and traces - attributes

Mapped Diagnostic Context

 

"an instrument for distinguishing interleaved log output from different sources" - log4j MDC docs

Correlate logs and traces - MDC

2022-09-27 20:28:58.010 trace_id=67fe345ce6b6271c6ec2b47791c4b1e7 span_id=dfdf3f1559a75bc1 trace_flags=01  INFO 7 --- [nio-8080-exec-1] i.o.example.javagent.Controller          : A sample log message!

Correlate logs and traces - MDC

$ cat src/main/resources/application.properties
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p

Spring Boot app with logback config:

Result:

Agent Extensions

$ java \
  -javaagent:path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.extensions=build/extension-1.0-all.jar \
  -jar myapp.jar

Enhance the agent capabilities, add new features:

 

"I don't want this span"

 

"I want to edit some attributes"

Auto-instrumentation

with
Kubernetes Operator

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
EOF
instrumentation.opentelemetry.io/inject-java: "true"

K8s auto-instrumentation injection

Annotate your Pod or Namespace to enable injection:

Community

Some more #OpenTelemetry

Thank you!

Marcin Stożek "Perk"

@marcinstozek / perk.pl

Java feat. OpenTelemetry

By Marcin Stożek

Java feat. OpenTelemetry

  • 861