Log Aggregation with Graylog
Why Log-Aggregation?
- Accessibility
- Usability
- Quantitative analysis
- Alerting
Graylog Dashboard
Graylog Features
- Open Source (Open Core)
- Optional Enterprise Features
- Collection & Extraction
- Interactive analysis and searches
- Dashboards & Visualization
- Alerts & Triggers
Architecture
Architecture (Cluster)
Architecture (Kafka + Fluentd)
billions of logs per day, > 100k messages per second
Transport
- Beats
- GELF
- Logstash
- Kafka
- Fluentd
- All kinds of crazy combos!
Application Logging
- Logback driver
- STDOUT
- special driver, i.e. GELF
- Docker
- default (JSON file)
- GELF
- journal
- Sidecar containers possible
Beware Multiline Java-Stacktraces!
Use Logstash or GELF appender
¯\_(ツ)_/¯
multiline {
#type => "all" # no type means for all inputs
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}Exercise
- Checkout the workshop and get Graylog up and running
- https://github.com/kiview/graylog-workshop
- Configure Docker GELF driver in order to let containers send logs to Graylog
- Configure Graylog extractors
- Setup some useful streams and visualizations
Log Aggregation
By Kevin Wittek
