Java feat.
tl;dr
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
Sep 24, 2022 1:50:09 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
Client
Unchanged gRPC example
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
Add the OpenTelemetry Java agent
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:07:51:024 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:07:53 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
[otel.javaagent 2022-09-24 14:27:21:991 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:27:24 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
[otel.javaagent 2022-09-24 14:27:25:223 +0200] [main] INFO io.opentelemetry.expor
ter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello' : 01aceb203db607f
b1219543673ed742c eea17e747f1cea5b CLIENT [tracer: io.opentelemetry.grpc-1.6:1.18
.0-alpha] AttributesMap{data={net.transport=ip_tcp, net.peer.name=localhost, net.
peer.port=50051, rpc.service=helloworld.Greeter, rpc.method=SayHello, thread.id=1
, rpc.system=grpc, thread.name=main, rpc.grpc.status_code=0}, capacity=128, total
AddedValues=9}
Sep 24, 2022 2:27:25 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
Client
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:26:45:737 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:26:51 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
[otel.javaagent 2022-09-24 14:27:25:196 +0200] [grpc-default-executor-0] INFO io.
opentelemetry.exporter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello
' : 01aceb203db607fb1219543673ed742c 8479d2d44df0f580 SERVER [tracer: io.opentele
metry.grpc-1.6:1.18.0-alpha] AttributesMap{data={net.host.name=localhost, net.tra
nsport=ip_tcp, net.host.port=50051, rpc.service=helloworld.Greeter, net.sock.peer
.addr=127.0.0.1, rpc.method=SayHello, thread.id=16, net.sock.peer.port=64271, rpc
.system=grpc, thread.name=grpc-default-executor-0, rpc.grpc.status_code=0}, capac
ity=128, totalAddedValues=11}
Server
tl;dr end
(stay tuned if you want more)
Telemetry 101
Signal types
Logs
Metrics
Traces
Signal types
Logs
Metrics
Traces
Logs
Tell me what you are doing.
2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL: remaining connection slots are reserved
...
Signal types
Logs
Metrics
Traces
Metrics
Don't need to grep your logs, look at the metrics
Signal types
Logs
Metrics
Traces
Traces
Service A
Service B
Database
Client
Traces
A
B
DB
HTTP GET
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
HTTP POST
DB CLIENT
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0444
Parent Span ID: 0x0333
Name: INSERT INTO TABLE
Status: OK (...)
HTTP POST
DB CLIENT
The same metadata for logs, metrics and traces.
.
What is
OpenTelemetry
?
Vendor provided
OpenTelemetry provided
OpenTelemetry provided
Components
Components
Guidelines - cross language requirements and expectations for all implementations
API, SDK
Semantic conventions
OTLP
Resource semantic conventions
OpenTelemetry Protocol - OTLP
General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.
Specified for both gRPC and HTTP transports.
Data Model Layers
OTLP Batch
Components
Manual
Instrumentation
Manually setup SDK
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "logical-service-name")));
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
.setResource(resource)
.build();
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
.registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().build()).build())
.setResource(resource)
.build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(sdkTracerProvider)
.setMeterProvider(sdkMeterProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
... or use SDK auto-configuration
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
</dependency>
and
OpenTelemetrySdk sdk = AutoConfiguredOpenTelemetrySdk.initialize()
.getOpenTelemetrySdk();
Logs
Metrics
Traces
Manual Instrumentation
Logs
Metrics
Traces
Manual Instrumentation
Logs
Status: experimental
Embrace existing logging solutions
SDK intended for logging libraries only
Not intended to be used by the end user directly
Logs
Metrics
Traces
Manual Instrumentation
Metrics
OpenTelemetry openTelemetry = // obtain instance of OpenTelemetry
// Gets or creates a named meter instance
Meter meter = openTelemetry.meterBuilder("instrumentation-library-name")
.setInstrumentationVersion("1.0.0")
.build();
// Build counter e.g. LongCounter
LongCounter counter = meter
.counterBuilder("processed_jobs")
.setDescription("Processed jobs")
.setUnit("1")
.build();
// It is recommended that the API user keep a reference
// to Attributes they will record against
Attributes attributes = Attributes.of(stringKey("Key"), "SomeWork");
// Record data
counter.add(123, attributes);
Logs
Metrics
Traces
Manual Instrumentation
Traces
Tracer tracer =
openTelemetry.getTracer("instrumentation-library-name", "1.0.0");
Span span = tracer.spanBuilder("my span").startSpan();
// Make the span the current span
try (Scope ss = span.makeCurrent()) {
// In this scope, the span is the current/active span
} finally {
span.end();
}
Automatic
Instrumentation
Java agent
$ java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
Java 8+ compatible
Dynamically injects bytecode
Highly configurable
Libs, frameworks
Supporting lot of libraries, frameworks, app servers and JVMs:
Akka, Apache Camel, AWS SDK, Grails, gRPC, Hibernate, RxJava, Spring, Undertow, Vert.x (and more)
Jetty, Tomcat, Websphere, Wildfly (and more)
OpenJDK, OpenJ9 (Ubuntu, Windows)
Libs, frameworks
Agent config
System properties ✅, env variables ✅, configuration file ✅
Will sanitise DB statements by default
Can suppress agent instrumentation for specific library only
Many other options - full config here
Manual instrumentation
with annotations
$ java \
-Dotel.instrumentation.common.default-enabled=false \
-Dotel.instrumentation.opentelemetry-api.enabled=true \
-Dotel.instrumentation.opentelemetry-annotations.enabled=true \
...
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;
public class MyClass {
@WithSpan
public void myMethod(@SpanAttribute("parameter1") String parameter1,
@SpanAttribute("parameter2") long parameter2) {
<...>
}
}
import io.opentelemetry.api.trace.Span;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class Controller {
private static final Logger LOGGER = LogManager.getLogger(Controller.class);
...
private void doWork(int sleepTime) throws InterruptedException {
Span span = tracer.spanBuilder("doWork").startSpan();
try (Scope ignored = span.makeCurrent()) {
Thread.sleep(sleepTime);
LOGGER.info("A sample log message!");
} finally {
span.end();
}
}
}
Correlate logs and traces - attributes
2022-09-27 19:58:02.931 INFO 6 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
Correlate logs and traces - attributes
2022-09-27T19:58:03.277Z info ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.12.0
Resource labels:
-> host.arch: STRING(aarch64)
-> host: STRING(f566af917d1f)
-> os.description: STRING(Linux 5.10.124-linuxkit)
-> os.type: STRING(linux)
-> process.command_line: STRING(/opt/java/openjdk:bin:java -javaagent:/opentelemetry-javaagent.jar)
-> process.executable.path: STRING(/opt/java/openjdk:bin:java)
-> process.pid: INT(6)
-> process.runtime.description: STRING(Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.16.1+1)
-> process.runtime.name: STRING(OpenJDK Runtime Environment)
-> process.runtime.version: STRING(11.0.16.1+1)
-> service: STRING(agent-example-app)
-> telemetry.auto.version: STRING(1.17.0)
-> telemetry.sdk.language: STRING(java)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.17.0)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope io.opentelemetry.example.javagent.Controller
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
Severity: INFO
Body: A sample log message!
Trace ID: 6fe5dff61c1c88ad54e488083a24e787
Span ID: 59a214b6de1c6c13
Correlate logs and traces - attributes
Mapped Diagnostic Context
"an instrument for distinguishing interleaved log output from different sources" - log4j MDC docs
Correlate logs and traces - MDC
2022-09-27 20:28:58.010 trace_id=67fe345ce6b6271c6ec2b47791c4b1e7 span_id=dfdf3f1559a75bc1 trace_flags=01 INFO 7 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
Correlate logs and traces - MDC
$ cat src/main/resources/application.properties
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
Spring Boot app with logback config:
Result:
Agent Extensions
$ java \
-javaagent:path/to/opentelemetry-javaagent.jar \
-Dotel.javaagent.extensions=build/extension-1.0-all.jar \
-jar myapp.jar
Enhance the agent capabilities, add new features:
"I don't want this span"
"I want to edit some attributes"
Auto-instrumentation
with
Kubernetes Operator
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
EOF
instrumentation.opentelemetry.io/inject-java: "true"
K8s auto-instrumentation injection
Annotate your Pod or Namespace to enable injection:
Community
Some more #OpenTelemetry
Thank you!
Marcin Stożek "Perk"
@marcinstozek / perk.pl
Java feat. OpenTelemetry
By Marcin Stożek
Java feat. OpenTelemetry
- 861