OTel Me a Story
Full Stack Observability for your Future Self

Photo by Frederick Marshall on Unsplash



while True:
try:
x = int(input("Enter a number: "))
break
except ValueError:
print("Oops!")


Observability
Observability


Metrics, Logs and Traces




Metrics
Photo by Altered Vision on Unsplash
Metrics

Logs

Photo by Oliver Paaske on Unsplash
Logs
E0419 06:59:50.681167 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: the server has asked for the client to provide credentials (get endpoints)
time="2024-04-17T20:10:41Z" level=info msg="successfully synced configuration to kong." subsystem=proxy-cache-resolver
Microsoft.Extensions.Logging.Debug.DebugLoggerProvider: Critical: Processing Customer A123, Vogel
Logs

Logs
Logs
Traces

Traces
get '/posts' do
Datadog::Tracing.trace('web.request', service: 'my-blog', resource: 'GET /posts') do |span|
# Trace the activerecord call
Datadog::Tracing.trace('posts.fetch') do
@posts = Posts.order(created_at: :desc).limit(10)
end
# Add some APM tags
span.set_tag('http.method', request.request_method)
span.set_tag('posts.count', @posts.length)
# Trace the template rendering
Datadog::Tracing.trace('template.render') do
erb :index
end
end
endTraces


Traces

1.
2.
Traces



Observability 2.0
OpenTelemetry (OTEL)
OTel captures traces, metrics, logs, and other application telemetry data and lets you send it to the backend of your choice.

OTel ≈ Observability api + spec

OTel Deployment

OTel Deployment

OTel Deployment


OTel in the Real World
Real ~= Imaginary


@app.post("/startSession")
async def start_session():
span = trace.get_current_span()
span.set_attribute("creating_session", True)
session = str(uuid.uuid4())
span.add_event("completed uuid generation, sending {} to client".format(session))
return {"sessionId": session}@app.post("/dm")
async def dm(chat: Chat):
bot = DungeonMaster(chat.sessionId)
span = trace.get_current_span()
if span:
span.set_attribute("input_length", len(chat.input))
span.set_attribute("session_id", chat.sessionId)
response = bot.invoke(chat.input)
return {"sessionId": chat.sessionId, "response": response}// http -v POST http://localhost:3000/api/dm/startSession
{
"sessionId": "3dca7890-cb67-47ef-8ca6-195418659ff7"
}
// http -v POST http://localhost:3000/api/dm/dm \
sessionId=3dca7890-cb67-47ef-8ca6-195418659ff7 input="hello, there"
{
"response": "Welcome to Luminos! I'm your Dungeon Master for this adventure in the world of Luminosity. It's great to have you here.\n\nBefore we begin, would you like to introduce your characters? Please provide me with their names, classes, backgrounds, and any relevant information about them. This will help me tailor the story to fit their unique personalities and abilities.\n\nAlso, I'll need to know how many players are joining this adventure today.",
"sessionId": "3dca7890-cb67-47ef-8ca6-195418659ff7"
} @PostMapping("/start")
@ResponseBody
public ResponseEntity<DmStartSessionResponse> startGame() {
Long now = System.currentTimeMillis();
Span startGameSpan = Span.current();
DmStartSessionResponse session = gameService.startGame();
Long duration = System.currentTimeMillis() - now;
startGameSpan.setAttribute("app.proxy_duration", duration);
startGameSpan.end();
return ResponseEntity.ok(session);
}
@PostMapping("/dm")
public ResponseEntity<String> sendChat(@RequestBody DmChatRequest request) {
String response = dmClient.sendChat(request.getMessage(), request.getSessionId());
return ResponseEntity.ok(response);
}// POST http://localhost:3000/api/rpg/game/start
// Content-Type: application/json
{
"sessionId": "68c54d82-08de-4b8b-b022-046781bc61d4"
}
// POST http://localhost:3000/api/rpg/game/dm
// Content-Type: application/json
// {
// "message": "hello, there",
// "sessionId": "68c54d82-08de-4b8b-b022-046781bc61d4"
// }
Welcome to Luminos! I'm your Dungeon Master, here to guide you...


// .env
# Honeycomb API key. Find one at https://ui.honeycomb.io/environments
export HONEYCOMB_API_KEY="your-Honeycomb-api-key-goes-here"
# Service name (opentelemetry standard field). Can be any string
export OTEL_SERVICE_NAME=sequence-of-numbers// ✗ go run main.go
Sending to Honeycomb with API Key <ykds> and service name intro-to-otel-sequence-of-numbers
Initializing the server... look for the app on http://localhost:3001

OTel your future self

Observability

OTel Me a Story: Full Stack Observability for your Future Self (OrlandoCodeCamp)
By virtualandy
OTel Me a Story: Full Stack Observability for your Future Self (OrlandoCodeCamp)
You shipped your code and someone says it’s busted. How can it be?! Observability is a super power to help you diagnose your systems. OpenTelemetry - or OTEL - is a set of APIs, SDKs and specs for observability. We’ll briefly cover Observability and how to get started so you can analyze and improve your software.
- 283



