Log Aggregation and Monitoring with Containers

Why Log-Aggregation?

  • Accessibility
  • Usability
  • Quantitative analysis
  • Alerting

Graylog Dashboard

Graylog Features

  • Open Source (Open Core)
    • Optional Enterprise Features
  • Collection & Extraction
  • Interactive analysis and searches
  • Dashboards & Visualization
  • Alerts & Triggers

Architecture

Architecture (Cluster)

Architecture (Kafka + Fluentd)

billions of logs per day, > 100k messages per second 

Transport

  • Beats
  • GELF
  • Logstash
  • Kafka
  • Fluentd
  • All kinds of crazy combos! 

Application Logging

  • Logback driver
    • STDOUT
    • special driver, i.e. GELF
  • Docker
    • default (JSON file)
    • GELF
    • journal
  • Sidecar containers possible

Beware Multiline Java-Stacktraces!

Use Logstash or GELF appender
¯\_(ツ)_/¯ 

multiline {
  #type => "all" # no type means for all inputs
  pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
  what => "previous"
}

Exercise

  • Checkout the workshop and get Graylog up and running
    • https://github.com/kiview/graylog-workshop
  • Configure Docker GELF driver in order to let containers send logs to Graylog
  • Configure Graylog extractors
  • Setup some useful streams and visualizations

Docker Vulnerability Scans

docker scan $imageHash

Prometheus

  • High dimensionality time-series database
  • Used for quantifiable metrics
    • E.g. req/s, CPU usage, etc.
  • HTTP pull model for querying data
  • Query data using PromQL
  • Data is gathered using exporters
  • Alertmanager for alerting
  • Grafana can be used for dashboards

Data Model

https://devconnected.com/the-definitive-guide-to-prometheus-in-2019/

Prometheus Architecture

Exercise

  • Extend the system using Prometheus and Grafana for metrics monitoring
  • Add additional exporters for the system (node exporter, cadvisor)
  • Query data and build dashboards
  • Optional: Add custom metrics to the Spring-Boot application (e.g. Micrometer Timed for profiling)
  • Open ended explorative exercise
Made with Slides.com