Dynamic Observability: Introduction to Dynamic Tracing in Elixir

Thomas Depierre

@DianaO

Diana Olympos

 

Twitter :

Github :

Debugging

  • The art of asking questions

  • The art of not making hypotheses

  • Writing software that works in production is really close to the art of debugging

  • To be able to ask questions, you need tools to answer

http://dtrace.org/blogs/bmc/2018/02/03/talks/#DebuggingTrilogy

Debugging in production

  • If noone touch your system, how long will it still run ?

  • Debugging everyday is what keep your system running

  • Production is the best place where you can get meaningful information about your system

https://www.youtube.com/watch?v=xA5U85LSk0M

Observability

  • Static
    • logs
    • events/tracing
    • metrics
  • Before the fact
  • Alerting/Pulse checking
  • BEAM is great at this too
  • But less out of the box
  • Big out there
  • Dynamic :
    • debugger
    • dynamic tracing
    • ps
    • netstat
  • Reactive
  • Debugging
  • BEAM is great at this ...
  • Rare because it means incident or resilience.

But i has logs

I can just go check my logs ! I am going to grep it with regex !

  • Logs are noisy
  • Logs are heavy
  • Logs are hard to query
  • Logs can contain Personal Data
  • Logs are expensive on your machines

PS: use ripgrep

But i has debugger

I can just run it under a debrgger! I am going to check what is happening !

  • Debuggers are bad at concurrency
  • They stop the world
  • Heavy runtime cost
  • Need to run under the debugger
  • Not in prod!

Dynamic Tracing!

  • Get events from any function
  • Two dimensions to attach to
    • PID spec
    • Trace pattern
  • Works for every single function in the BEAM
  • "For free"*
    • Intrinsinc to the BEAM
    • Meant for use in prod
    • Nothing to add, available right now

How to use ?

:dbg.start
:dbg.tracer(:process, {fn _, 5 -> :dbg.stop_clear()
                        msg, n -> IO.inspect(msg); n+1 end, 0})
:dbg.tpl(Enum, :map, [])
:dbg.p(:all, :c)

# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)

# No more trace
Enum.map([1,2,3,4], & &1 + 1)

Urgh what ?

  • Not super intuitive
  • Can overload the machine
  • Send information in logs
  • Really low level

Let's bring a dependency

Erlang in Anger

https://www.erlang-in-anger.com/

:recon_trace.calls({Enum, :map, :_}, 5)

# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)

# No more trace
Enum.map([1,2,3,4], & &1 + 1)

With Recon

Ok but just input ?

  • Two parts to a Trace Pattern
    • Module, Function, Arity
    • Match Spec
      • guards
      • matching like a function head
      • ActionFunction like return_trace()
  • Hard to write by hand

Let's bring a dependency, Ex2ms

import Ex2Ms

# Ex2Ms define the "fun" macro
match_spec = fun do _ -> return_trace() end

:recon_trace.calls({Enum, :map, match_spec}, 10)

# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)

# No more trace
Enum.map([1,2,3,4], & &1 + 1)

With Recon

Thank You

Questions ?

https://slides.com/di4nao/elixir-tracing

Live Demo :)