Logging mit Graylog

Ein eigener Log-Server mit Graylog

Stefan Moises        [DevOps Camp 2015 - #dvop15]

Logging?!

  • Eigene Logfiles
  • Webserver Logs
  • Applikations Logs
  • System Logs
  • Build Logs
  • Datenbank Logs
  • Security Logs
  • ...

Logging Möglichkeiten

  • Lokale Dateien & grep :)
  • Log4J, Log4PHP, etc.
  • Cloud / SAAS (Loggly, New Relic, ...)
  • Logstash (+ Kibana / Banana / ...)
  • Email, SMS, Push,...
  • Graylog!

Graylog

  • Open Source Logserver
  • Java-Stack (Java 7+)
  • verwendet intern Elastic Search & MongoDB
  • diverse Install-Packages verfügbar:
    VM, Vagrant, Puppet, Chef, Docker,...
    => https://www.graylog.org/download-graylog/
  • Skalierbar: unterstützt ES-Cluster, Load Balancer, ...
  • diverse, flexible Input-Kanäle (Streams), Alerting, Outputs,...

Graylog Architektur

Graylog Einsatzgebiete

  • Applikations Management (Exception Logs, etc.)
  • Infrastruktur / Operations Management (Hosting, Webserver, Build Logs, etc.)
  • Security Management (Access-Log Analyse, Audits, ...)

Graylog Features

  • Flexibler Input (HTTP, UDP, TCP, JSON, ...) für alle Arten von Daten
  • Schnittstellen / Plugins für diverse Anwendungen / Protokolle
  • "Extrator" Schnittstelle zum Parsen beliebiger Datenstrukturen (a la GROK-Patterns, aber in JSON)
  • REST-Schnittstelle und "Output Routing" (Forwarding) 
  • Mächtige, frei definierbare Dashboards mit Filtern / Suchen, Charts aller Art, über alle Inputs/Streams
  • Ausgefeilte Suche für Datenanalyse
  • Alerting (Email, Hipchat, ...) über "Streams" (Message-Filter)
  • GELF - Graylog Extended Log Format, inkl. Compression (https://www.graylog.org/resources/gelf-2/) 

Graylog Plugins (Auswahl)

Graylog Input Formate

  • GELF Bibliothek (Log4J, gelf-php, ...)
  • Extractors (MongoDB, Heroku, syslogs, ...)
"Parser Definitionen" in JSON (ähnlich zu GROK Patterns bei Logstash) [Extractors are applied on message inputs and instruct Graylog2 how to structure unstructured parts of a log message. ]
  • Content Packs (nginx, etc.)
[Content packs are bundles of Graylog2 input, extractor, stream, dashboard and output configurations that can provide full support for a data source. Some content packs are shipped with Graylog2 by default and some are available from this web site. Packs that were downloaded from here can be imported using the Graylog2 web interface.]
https://www.graylog.org/resource/content-pack/547b5021e4b0a06d87eea01e/

Graylog MongoDB Extractor

Graylog gelf-php

https://github.com/bzikarsky/gelf-php/

Diskussion

  • Wer nutzt Graylog?
  • Vergleich zu ELK-Stack?
  • Andere Tools?
  • ...
Made with Slides.com