~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:33 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 24, 2022 1:50:35 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
Sep 24, 2022 1:50:09 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
Client
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:07:51:024 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:07:53 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
Server
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-client
[otel.javaagent 2022-09-24 14:27:21:991 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:27:24 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
[otel.javaagent 2022-09-24 14:27:25:223 +0200] [main] INFO io.opentelemetry.expor
ter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello' : 01aceb203db607f
b1219543673ed742c eea17e747f1cea5b CLIENT [tracer: io.opentelemetry.grpc-1.6:1.18
.0-alpha] AttributesMap{data={net.transport=ip_tcp, net.peer.name=localhost, net.
peer.port=50051, rpc.service=helloworld.Greeter, rpc.method=SayHello, thread.id=1
, rpc.system=grpc, thread.name=main, rpc.grpc.status_code=0}, capacity=128, total
AddedValues=9}
Sep 24, 2022 2:27:25 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world
Client
~/grpc-java/examples $ export JAVA_OPTS="-javaagent:opentelemetry-javaagent.jar"
~/grpc-java/examples $ export OTEL_TRACES_EXPORTER=logging
~/grpc-java/examples $ ./build/install/examples/bin/hello-world-server
[otel.javaagent 2022-09-24 14:26:45:737 +0200] [main] INFO io.opentelemetry.javaa
gent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.18.0
Sep 24, 2022 2:26:51 PM io.grpc.examples.helloworld.HelloWorldServer start
INFO: Server started, listening on 50051
[otel.javaagent 2022-09-24 14:27:25:196 +0200] [grpc-default-executor-0] INFO io.
opentelemetry.exporter.logging.LoggingSpanExporter - 'helloworld.Greeter/SayHello
' : 01aceb203db607fb1219543673ed742c 8479d2d44df0f580 SERVER [tracer: io.opentele
metry.grpc-1.6:1.18.0-alpha] AttributesMap{data={net.host.name=localhost, net.tra
nsport=ip_tcp, net.host.port=50051, rpc.service=helloworld.Greeter, net.sock.peer
.addr=127.0.0.1, rpc.method=SayHello, thread.id=16, net.sock.peer.port=64271, rpc
.system=grpc, thread.name=grpc-default-executor-0, rpc.grpc.status_code=0}, capac
ity=128, totalAddedValues=11}
Server
(stay tuned if you want more)
Tell me what you are doing.
2021-09-21 15:11:44,345 - werkzeug - INFO - \"\u001B[33mPOST /order HTTP/1.1\u001B[0m\" 404
2021-09-21 15:11:45,206 - root - INFO - Preparing espresso coffee
2021-09-21 15:11:46,269 - root - INFO - Get product price: cornetto
2021-09-21 15:11:45,024 - werkzeug - INFO - \"OPTIONS /order HTTP/1.1\" 200
2021-09-21 15:11:45,246 - root - ERROR - Missing some ingredients
2021-09-21 15:11:46,270 - root - INFO - Query DB for price of product: cornetto
2021-09-21 15:11:45,074 - root - INFO - Check if tiramisu is available
2021-09-21 15:11:46,272 - root - ERROR - FATAL: remaining connection slots are reserved
...
Don't need to grep your logs, look at the metrics
Service A
Service B
Database
Client
A
B
DB
HTTP GET
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
HTTP POST
DB CLIENT
A
B
DB
HTTP GET
Trace ID: 0x0123
Span ID: 0x0111
Name: HTTP GET (remote)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0222
Parent Span ID: 0x0111
Name: HTTP POST (client)
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0333
Parent Span ID: 0x0222
Name: /order
Status: OK (...)
Trace ID: 0x0123
Span ID: 0x0444
Parent Span ID: 0x0333
Name: INSERT INTO TABLE
Status: OK (...)
HTTP POST
DB CLIENT
The same metadata for logs, metrics and traces.
.
Vendor provided
OpenTelemetry provided
OpenTelemetry provided
Guidelines - cross language requirements and expectations for all implementations
API, SDK
Semantic conventions
OTLP
General-purpose telemetry data delivery protocol designed in the scope of OpenTelemetry project.
Specified for both gRPC and HTTP transports.
Resource resource = Resource.getDefault()
.merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "logical-service-name")));
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
.setResource(resource)
.build();
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
.registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().build()).build())
.setResource(resource)
.build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(sdkTracerProvider)
.setMeterProvider(sdkMeterProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
</dependency>
and
OpenTelemetrySdk sdk = AutoConfiguredOpenTelemetrySdk.initialize()
.getOpenTelemetrySdk();
Status: experimental
Embrace existing logging solutions
SDK intended for logging libraries only
Not intended to be used by the end user directly
OpenTelemetry openTelemetry = // obtain instance of OpenTelemetry
// Gets or creates a named meter instance
Meter meter = openTelemetry.meterBuilder("instrumentation-library-name")
.setInstrumentationVersion("1.0.0")
.build();
// Build counter e.g. LongCounter
LongCounter counter = meter
.counterBuilder("processed_jobs")
.setDescription("Processed jobs")
.setUnit("1")
.build();
// It is recommended that the API user keep a reference
// to Attributes they will record against
Attributes attributes = Attributes.of(stringKey("Key"), "SomeWork");
// Record data
counter.add(123, attributes);
Tracer tracer =
openTelemetry.getTracer("instrumentation-library-name", "1.0.0");
Span span = tracer.spanBuilder("my span").startSpan();
// Make the span the current span
try (Scope ss = span.makeCurrent()) {
// In this scope, the span is the current/active span
} finally {
span.end();
}
$ java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
Java 8+ compatible
Dynamically injects bytecode
Highly configurable
Supporting lot of libraries, frameworks, app servers and JVMs:
Akka, Apache Camel, AWS SDK, Grails, gRPC, Hibernate, RxJava, Spring, Undertow, Vert.x (and more)
Jetty, Tomcat, Websphere, Wildfly (and more)
OpenJDK, OpenJ9 (Ubuntu, Windows)
System properties ✅, env variables ✅, configuration file ✅
Will sanitise DB statements by default
Can suppress agent instrumentation for specific library only
Many other options - full config here
$ java \
-Dotel.instrumentation.common.default-enabled=false \
-Dotel.instrumentation.opentelemetry-api.enabled=true \
-Dotel.instrumentation.opentelemetry-annotations.enabled=true \
...
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;
public class MyClass {
@WithSpan
public void myMethod(@SpanAttribute("parameter1") String parameter1,
@SpanAttribute("parameter2") long parameter2) {
<...>
}
}
import io.opentelemetry.api.trace.Span;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class Controller {
private static final Logger LOGGER = LogManager.getLogger(Controller.class);
...
private void doWork(int sleepTime) throws InterruptedException {
Span span = tracer.spanBuilder("doWork").startSpan();
try (Scope ignored = span.makeCurrent()) {
Thread.sleep(sleepTime);
LOGGER.info("A sample log message!");
} finally {
span.end();
}
}
}
2022-09-27 19:58:02.931 INFO 6 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
2022-09-27T19:58:03.277Z info ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.12.0
Resource labels:
-> host.arch: STRING(aarch64)
-> host: STRING(f566af917d1f)
-> os.description: STRING(Linux 5.10.124-linuxkit)
-> os.type: STRING(linux)
-> process.command_line: STRING(/opt/java/openjdk:bin:java -javaagent:/opentelemetry-javaagent.jar)
-> process.executable.path: STRING(/opt/java/openjdk:bin:java)
-> process.pid: INT(6)
-> process.runtime.description: STRING(Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.16.1+1)
-> process.runtime.name: STRING(OpenJDK Runtime Environment)
-> process.runtime.version: STRING(11.0.16.1+1)
-> service: STRING(agent-example-app)
-> telemetry.auto.version: STRING(1.17.0)
-> telemetry.sdk.language: STRING(java)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.17.0)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope io.opentelemetry.example.javagent.Controller
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
Severity: INFO
Body: A sample log message!
Trace ID: 6fe5dff61c1c88ad54e488083a24e787
Span ID: 59a214b6de1c6c13
Mapped Diagnostic Context
"an instrument for distinguishing interleaved log output from different sources" - log4j MDC docs
2022-09-27 20:28:58.010 trace_id=67fe345ce6b6271c6ec2b47791c4b1e7 span_id=dfdf3f1559a75bc1 trace_flags=01 INFO 7 --- [nio-8080-exec-1] i.o.example.javagent.Controller : A sample log message!
$ cat src/main/resources/application.properties
logging.pattern.level = trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
Spring Boot app with logback config:
Result:
$ java \
-javaagent:path/to/opentelemetry-javaagent.jar \
-Dotel.javaagent.extensions=build/extension-1.0-all.jar \
-jar myapp.jar
Enhance the agent capabilities, add new features:
"I don't want this span"
"I want to edit some attributes"
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
EOF
instrumentation.opentelemetry.io/inject-java: "true"
Annotate your Pod or Namespace to enable injection:
Marcin Stożek "Perk"
@marcinstozek / perk.pl