Static Observability: Introduction to metrics in Elixir
Thomas Depierre
@DianaO
Diana Olympos
Twitter :
Github :
Audience time
- Who does "DevOps" at work ?
- Who test "in production" ?
- Who regularly SSH into prod to debug and gather data ?
Debugging
-
The art of asking questions
-
The art of not making hypotheses
-
Writing software that works in production is really close to the art of debugging
-
To be able to ask questions, you need tools to answer
Debugging in production
-
If noone touch your system, how long will it still run ?
-
Debugging everyday is what keep your system running
-
Production is the best place where you can get meaningful information about your system
Observability
- Dynamic :
- dynamic tracing
- ps
- netstat
- Reactive
- Debugging
- BEAM is great at this ...
- Rare because it means incident or resilience.
-
Static
- logs
- (events)
- metrics
- Before the fact
- Alerting/Pulse checking
- BEAM is great at this too
- But not out of the box
- Big out there
But i has logs
I can just go check my logs ! I am going to grep it with regex !
- Logs are noisy
- Logs are heavy
- Logs are hard to query
- Logs can contain Personal Data
- Logs are expensive on your machines
PS: use ripgrep
Metrics
- Small
- Easy to aggregate
- Easy to do maths on
- Edge !
- Alarms !
Logger equivalent for metrics ?
- StatsD (from Etsy/Flickr) https://github.com/etsy/statsd
- UDP socket that accept well formed message
- StatsD agent will aggregate, batch and send where needed (Datadog, Prometheus, TimeSerie DB, etc)
- Protocol
- https://hex.pm/packages/statix
How to use ?
Data Types
- counter
- gauge
- timer
- sets
- histogram