![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/10035732/pasted-from-clipboard.png)
Java feat.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/10048525/pasted-from-clipboard.png)
tl;dr
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
Sep 24, 2022 1:50:09 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
Client
Unchanged gRPC example
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
Add the OpenTelemetry Java agent
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:07:51:024 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:07:53 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
[otel.javaagent 2022-09-24 14:27:21:991 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:27:24 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
[otel.javaagent 2022-09-24 14:27:25:223 +0200] [main] INFO io.opentelemetry.expor
ter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello' : 01aceb203db607f
b1219543673ed742c eea17e747f1cea5b CLIENT [tracer: io.opentelemetry.grpc-1.6:1.18
.0-alpha] AttributesMap{data={net.transport=ip_tcp, net.peer.name=localhost, net.
peer.port=50051, rpc.service=helloworld.Greeter, rpc.method=SayHello, thread.id=1
, rpc.system=grpc, thread.name=main, rpc.grpc.status_code=0}, capacity=128, total
AddedValues=9}
Sep 24, 2022 2:27:25 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
Client
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:26:45:737 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:26:51 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
[otel.javaagent 2022-09-24 14:27:25:196 +0200] [grpc-default-executor-0] INFO io.
opentelemetry.exporter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello
' : 01aceb203db607fb1219543673ed742c 8479d2d44df0f580 SERVER [tracer: io.opentele
metry.grpc-1.6:1.18.0-alpha] AttributesMap{data={net.host.name=localhost, net.tra
nsport=ip_tcp, net.host.port=50051, rpc.service=helloworld.Greeter, net.sock.peer
.addr=127.0.0.1, rpc.method=SayHello, thread.id=16, net.sock.peer.port=64271, rpc
.system=grpc, thread.name=grpc-default-executor-0, rpc.grpc.status_code=0}, capac
ity=128, totalAddedValues=11}
Server
tl;dr end
(stay tuned if you want more)
Telemetry 101
Signal types
Logs
Metrics
Traces
Signal types
Logs
Metrics
Traces
Logs
Tell me what you are doing.
2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL: remaining connection slots are reserved
...
Signal types
Logs
Metrics
Traces
Metrics
Don't need to grep your logs, look at the metrics
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9450710/pasted-from-clipboard.png)
Signal types
Logs
Metrics
Traces
Traces
Service A
Service B
Database
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Client
Traces
A
B
DB
HTTP GET
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
Traces
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0444
Parent Span ID: 0x0333
Name: INSERT INTO TABLE
Status: OK (...)
HTTP POST
DB CLIENT
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9455002/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459986/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459985/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459993/pasted-from-clipboard.png)
The same metadata for logs, metrics and traces.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9460007/pasted-from-clipboard.png)
.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9460008/pasted-from-clipboard.png)
What is
OpenTelemetry
?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9456306/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458404/unified-collection-short.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9456306/pasted-from-clipboard.png)
Vendor provided
OpenTelemetry provided
OpenTelemetry provided
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458404/unified-collection-short.png)
Components
Components
Guidelines - cross language requirements and expectations for all implementations
API, SDK
Semantic conventions
OTLP
Resource semantic conventions
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458506/pasted-from-clipboard.png)
OpenTelemetry Protocol - OTLP
General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.
Specified for both gRPC and HTTP transports.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9869664/pasted-from-clipboard.png)
Data Model Layers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9869677/pasted-from-clipboard.png)
OTLP Batch
Components
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9456306/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458404/unified-collection-short.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9456306/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458404/unified-collection-short.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458545/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458546/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458547/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458548/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458549/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458550/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458552/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458553/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458554/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458558/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9865115/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9458530/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459414/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459464/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459465/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459473/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459478/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459483/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459488/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459491/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459492/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459498/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9459509/pasted-from-clipboard.png)
Manual
Instrumentation
Manually setup SDK
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "logical-service-name")));
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
.setResource(resource)
.build();
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
.registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().build()).build())
.setResource(resource)
.build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(sdkTracerProvider)
.setMeterProvider(sdkMeterProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
... or use SDK auto-configuration
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
</dependency>
and
OpenTelemetrySdk sdk = AutoConfiguredOpenTelemetrySdk.initialize()
.getOpenTelemetrySdk();
Logs
Metrics
Traces
Manual Instrumentation
Logs
Metrics
Traces
Manual Instrumentation
Logs
Status: experimental
Embrace existing logging solutions
SDK intended for logging libraries only
Not intended to be used by the end user directly
Logs
Metrics
Traces
Manual Instrumentation
Metrics
OpenTelemetry openTelemetry = // obtain instance of OpenTelemetry
// Gets or creates a named meter instance
Meter meter = openTelemetry.meterBuilder("instrumentation-library-name")
.setInstrumentationVersion("1.0.0")
.build();
// Build counter e.g. LongCounter
LongCounter counter = meter
.counterBuilder("processed_jobs")
.setDescription("Processed jobs")
.setUnit("1")
.build();
// It is recommended that the API user keep a reference
// to Attributes they will record against
Attributes attributes = Attributes.of(stringKey("Key"), "SomeWork");
// Record data
counter.add(123, attributes);
Logs
Metrics
Traces
Manual Instrumentation
Traces
Tracer tracer =
openTelemetry.getTracer("instrumentation-library-name", "1.0.0");
Span span = tracer.spanBuilder("my span").startSpan();
// Make the span the current span
try (Scope ss = span.makeCurrent()) {
// In this scope, the span is the current/active span
} finally {
span.end();
}
Automatic
Instrumentation
Java agent
$ java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
Java 8+ compatible
Dynamically injects bytecode
Highly configurable
Libs, frameworks
Supporting lot of libraries, frameworks, app servers and JVMs:
Akka, Apache Camel, AWS SDK, Grails, gRPC, Hibernate, RxJava, Spring, Undertow, Vert.x (and more)
Jetty, Tomcat, Websphere, Wildfly (and more)
OpenJDK, OpenJ9 (Ubuntu, Windows)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9866870/pasted-from-clipboard.png)
Libs, frameworks
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9866873/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/9866875/pasted-from-clipboard.png)
Agent config
System properties ✅, env variables ✅, configuration file ✅
Will sanitise DB statements by default
Can suppress agent instrumentation for specific library only
Many other options - full config here
Manual instrumentation
with annotations
$ java \
-Dotel.instrumentation.common.default-enabled=false \
-Dotel.instrumentation.opentelemetry-api.enabled=true \
-Dotel.instrumentation.opentelemetry-annotations.enabled=true \
...
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;
public class MyClass {
@WithSpan
public void myMethod(@SpanAttribute("parameter1") String parameter1,
@SpanAttribute("parameter2") long parameter2) {
<...>
}
}
import io.opentelemetry.api.trace.Span;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class Controller {
private static final Logger LOGGER = LogManager.getLogger(Controller.class);
...
private void doWork(int sleepTime) throws InterruptedException {
Span span = tracer.spanBuilder("doWork").startSpan();
try (Scope ignored = span.makeCurrent()) {
Thread.sleep(sleepTime);
LOGGER.info("A sample log message!");
} finally {
span.end();
}
}
}
Correlate logs and traces - attributes
2022-09-27 19:58:02.931 INFO 6 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
Correlate logs and traces - attributes
2022-09-27T19:58:03.277Z info ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.12.0
Resource labels:
-> host.arch: STRING(aarch64)
-> host: STRING(f566af917d1f)
-> os.description: STRING(Linux 5.10.124-linuxkit)
-> os.type: STRING(linux)
-> process.command_line: STRING(/opt/java/openjdk:bin:java -javaagent:/opentelemetry-javaagent.jar)
-> process.executable.path: STRING(/opt/java/openjdk:bin:java)
-> process.pid: INT(6)
-> process.runtime.description: STRING(Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.16.1+1)
-> process.runtime.name: STRING(OpenJDK Runtime Environment)
-> process.runtime.version: STRING(11.0.16.1+1)
-> service: STRING(agent-example-app)
-> telemetry.auto.version: STRING(1.17.0)
-> telemetry.sdk.language: STRING(java)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.17.0)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope io.opentelemetry.example.javagent.Controller
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
Severity: INFO
Body: A sample log message!
Trace ID: 6fe5dff61c1c88ad54e488083a24e787
Span ID: 59a214b6de1c6c13
Correlate logs and traces - attributes
Mapped Diagnostic Context
"an instrument for distinguishing interleaved log output from different sources" - log4j MDC docs
Correlate logs and traces - MDC
2022-09-27 20:28:58.010 trace_id=67fe345ce6b6271c6ec2b47791c4b1e7 span_id=dfdf3f1559a75bc1 trace_flags=01 INFO 7 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
Correlate logs and traces - MDC
$ cat src/main/resources/application.properties
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
Spring Boot app with logback config:
Result:
Agent Extensions
$ java \
-javaagent:path/to/opentelemetry-javaagent.jar \
-Dotel.javaagent.extensions=build/extension-1.0-all.jar \
-jar myapp.jar
Enhance the agent capabilities, add new features:
"I don't want this span"
"I want to edit some attributes"
Auto-instrumentation
with
Kubernetes Operator
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
EOF
instrumentation.opentelemetry.io/inject-java: "true"
K8s auto-instrumentation injection
Annotate your Pod or Namespace to enable injection:
Community
Some more #OpenTelemetry
Thank you!
Marcin Stożek "Perk"
@marcinstozek / perk.pl
![](https://s3.amazonaws.com/media-p.slid.es/uploads/65632/images/10035742/pasted-from-clipboard.png)
Java feat. OpenTelemetry
By Marcin Stożek
Java feat. OpenTelemetry
- 716