Dynamic Observability: Introduction to Dynamic Tracing in Elixir
Thomas Depierre
@DianaO
Diana Olympos
Twitter :
Github :
Debugging
-
The art of asking questions
-
The art of not making hypotheses
-
Writing software that works in production is really close to the art of debugging
-
To be able to ask questions, you need tools to answer
Debugging in production
-
If noone touch your system, how long will it still run ?
-
Debugging everyday is what keep your system running
-
Production is the best place where you can get meaningful information about your system
Observability
-
Static
- logs
- events/tracing
- metrics
- Before the fact
- Alerting/Pulse checking
- BEAM is great at this too
- But less out of the box
- Big out there
- Dynamic :
- debugger
- dynamic tracing
- ps
- netstat
- Reactive
- Debugging
- BEAM is great at this ...
- Rare because it means incident or resilience.
But i has logs
I can just go check my logs ! I am going to grep it with regex !
- Logs are noisy
- Logs are heavy
- Logs are hard to query
- Logs can contain Personal Data
- Logs are expensive on your machines
PS: use ripgrep
But i has debugger
I can just run it under a debrgger! I am going to check what is happening !
- Debuggers are bad at concurrency
- They stop the world
- Heavy runtime cost
- Need to run under the debugger
- Not in prod!
Dynamic Tracing!
- Get events from any function
- Two dimensions to attach to
- PID spec
- Trace pattern
- Works for every single function in the BEAM
- "For free"*
- Intrinsinc to the BEAM
- Meant for use in prod
- Nothing to add, available right now
How to use ?
:dbg.start
:dbg.tracer(:process, {fn _, 5 -> :dbg.stop_clear()
msg, n -> IO.inspect(msg); n+1 end, 0})
:dbg.tpl(Enum, :map, [])
:dbg.p(:all, :c)
# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
# No more trace
Enum.map([1,2,3,4], & &1 + 1)Urgh what ?
- Not super intuitive
- Can overload the machine
- Send information in logs
- Really low level
Let's bring a dependency
Erlang in Anger
:recon_trace.calls({Enum, :map, :_}, 5)
# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
# No more trace
Enum.map([1,2,3,4], & &1 + 1)With Recon
Ok but just input ?
- Two parts to a Trace Pattern
- Module, Function, Arity
- Match Spec
- guards
- matching like a function head
- ActionFunction like return_trace()
- Hard to write by hand
Let's bring a dependency, Ex2ms
import Ex2Ms
# Ex2Ms define the "fun" macro
match_spec = fun do _ -> return_trace() end
:recon_trace.calls({Enum, :map, match_spec}, 10)
# With trace
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
Enum.map([1,2,3,4], & &1 + 1)
# No more trace
Enum.map([1,2,3,4], & &1 + 1)With Recon
Thank You
Questions ?
Live Demo :)
Dynamic Observability. An Intro to Dynamic Tracing in Elixir
By di4nao
