OTel Me a Story

Full Stack Observability for your Future Self

https://giphy.com/gifs/creative-thinking-thinker-VbEuHLBUPQm55MyqJg
while True:
    try:
        x = int(input("Enter a number: "))
        break
    except ValueError:
        print("Oops!")

Observability

Observability

Metrics, Logs and Traces

Metrics

Metrics

Logs

Photo by Oliver Paaske on Unsplash

Logs

E0419 06:59:50.681167       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.4/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: the server has asked for the client to provide credentials (get endpoints)

time="2024-04-17T20:10:41Z" level=info msg="successfully synced configuration to kong." subsystem=proxy-cache-resolver

Microsoft.Extensions.Logging.Debug.DebugLoggerProvider: Critical: Processing Customer A123, Vogel

Logs

Logs

Logs

Traces

Traces

get '/posts' do
  Datadog::Tracing.trace('web.request', service: 'my-blog', resource: 'GET /posts') do |span|
    # Trace the activerecord call
    Datadog::Tracing.trace('posts.fetch') do
      @posts = Posts.order(created_at: :desc).limit(10)
    end

    # Add some APM tags
    span.set_tag('http.method', request.request_method)
    span.set_tag('posts.count', @posts.length)

    # Trace the template rendering
    Datadog::Tracing.trace('template.render') do
      erb :index
    end
  end
end

Traces

Traces

1.

2.

Traces

Observability 2.0

OpenTelemetry (OTEL)

OTel captures traces, metrics, logs, and other application telemetry data and lets you send it to the backend of your choice.


 

OTel ≈ Observability api + spec

OTel Deployment

OTel Deployment

OTel Deployment

OTel in the Real World

Real ~= Imaginary

@app.post("/startSession")
async def start_session():
    span = trace.get_current_span()
    span.set_attribute("creating_session", True)
    session = str(uuid.uuid4())
    span.add_event("completed uuid generation, sending {} to client".format(session))
    return {"sessionId": session}
@app.post("/dm")
async def dm(chat: Chat):
    bot = DungeonMaster(chat.sessionId)
    span = trace.get_current_span()
    if span:
        span.set_attribute("input_length", len(chat.input))
        span.set_attribute("session_id", chat.sessionId)
    response = bot.invoke(chat.input)
    return {"sessionId": chat.sessionId, "response": response}
// http -v POST http://localhost:3000/api/dm/startSession
{
    "sessionId": "3dca7890-cb67-47ef-8ca6-195418659ff7"
}

// http -v POST http://localhost:3000/api/dm/dm \
sessionId=3dca7890-cb67-47ef-8ca6-195418659ff7 input="hello, there"
{
    "response": "Welcome to Luminos! I'm your Dungeon Master for this adventure in the world of Luminosity. It's great to have you here.\n\nBefore we begin, would you like to introduce your characters? Please provide me with their names, classes, backgrounds, and any relevant information about them. This will help me tailor the story to fit their unique personalities and abilities.\n\nAlso, I'll need to know how many players are joining this adventure today.",
    "sessionId": "3dca7890-cb67-47ef-8ca6-195418659ff7"
}
 @PostMapping("/start")
    @ResponseBody
    public ResponseEntity<DmStartSessionResponse> startGame() {
        Long now = System.currentTimeMillis();
        Span startGameSpan = Span.current();
        DmStartSessionResponse session = gameService.startGame();
        Long duration = System.currentTimeMillis() - now;
        startGameSpan.setAttribute("app.proxy_duration", duration);
        startGameSpan.end();
        return ResponseEntity.ok(session);
    }
@PostMapping("/dm")
    public ResponseEntity<String> sendChat(@RequestBody DmChatRequest request) {
        String response = dmClient.sendChat(request.getMessage(), request.getSessionId());
        return ResponseEntity.ok(response);
    }
// POST http://localhost:3000/api/rpg/game/start
// Content-Type: application/json

{
  "sessionId": "68c54d82-08de-4b8b-b022-046781bc61d4"
}
// POST http://localhost:3000/api/rpg/game/dm
// Content-Type: application/json
// {
//  "message": "hello, there",
//  "sessionId": "68c54d82-08de-4b8b-b022-046781bc61d4"
// }

Welcome to Luminos! I'm your Dungeon Master, here to guide you...
// .env

# Honeycomb API key. Find one at https://ui.honeycomb.io/environments
export HONEYCOMB_API_KEY="your-Honeycomb-api-key-goes-here"

# Service name (opentelemetry standard field). Can be any string
export OTEL_SERVICE_NAME=sequence-of-numbers
// ✗ go run main.go
Sending to Honeycomb with API Key <ykds> and service name intro-to-otel-sequence-of-numbers
Initializing the server... look for the app on http://localhost:3001

OTel your future self

Observability

OTel Me a Story: Full Stack Observability for your Future Self (OrlandoCodeCamp)

By virtualandy

OTel Me a Story: Full Stack Observability for your Future Self (OrlandoCodeCamp)

You shipped your code and someone says it’s busted. How can it be?! Observability is a super power to help you diagnose your systems. OpenTelemetry - or OTEL - is a set of APIs, SDKs and specs for observability. We’ll briefly cover Observability and how to get started so you can analyze and improve your software.

  • 283