Logging mit Graylog

Ein eigener Log-Server mit Graylog

Stefan Moises [DevOps Camp 2015 - #dvop15]

Logging?!

Eigene Logfiles
Webserver Logs
Applikations Logs
System Logs
Build Logs
Datenbank Logs
Security Logs
...

Logging Möglichkeiten

Lokale Dateien & grep :)
Log4J, Log4PHP, etc.
Cloud / SAAS (Loggly, New Relic, ...)
Logstash (+ Kibana / Banana / ...)
Email, SMS, Push,...
Graylog!

Graylog

Open Source Logserver
Java-Stack (Java 7+)
verwendet intern Elastic Search & MongoDB
diverse Install-Packages verfügbar:
VM, Vagrant, Puppet, Chef, Docker,...
=> https://www.graylog.org/download-graylog/
Skalierbar: unterstützt ES-Cluster, Load Balancer, ...
diverse, flexible Input-Kanäle (Streams), Alerting, Outputs,...

Graylog Architektur

Graylog Einsatzgebiete

Applikations Management (Exception Logs, etc.)
Infrastruktur / Operations Management (Hosting, Webserver, Build Logs, etc.)
Security Management (Access-Log Analyse, Audits, ...)

Graylog Features

Flexibler Input (HTTP, UDP, TCP, JSON, ...) für alle Arten von Daten
Schnittstellen / Plugins für diverse Anwendungen / Protokolle
"Extrator" Schnittstelle zum Parsen beliebiger Datenstrukturen (a la GROK-Patterns, aber in JSON)
REST-Schnittstelle und "Output Routing" (Forwarding)
Mächtige, frei definierbare Dashboards mit Filtern / Suchen, Charts aller Art, über alle Inputs/Streams
Ausgefeilte Suche für Datenanalyse
Alerting (Email, Hipchat, ...) über "Streams" (Message-Filter)
GELF - Graylog Extended Log Format, inkl. Compression (https://www.graylog.org/resources/gelf-2/)

Graylog Plugins (Auswahl)

komplette Liste: https://www.graylog.org/resources/data-sources/
Plugins für Log4J, Log4PHP, ...
Plugins für Java, PHP, Go, Node.js, Ruby, C++, etc.
Plugins für MongoDB, nginx, Apache, etc.

Graylog Input Formate

GELF Bibliothek (Log4J, gelf-php, ...)

Extractors (MongoDB, Heroku, syslogs, ...)
"Parser Definitionen" in JSON (ähnlich zu GROK Patterns bei Logstash) [Extractors are applied on message inputs and instruct Graylog2 how to structure unstructured parts of a log message. ]

Content Packs (nginx, etc.)
[Content packs are bundles of Graylog2 input, extractor, stream, dashboard and output configurations that can provide full support for a data source. Some content packs are shipped with Graylog2 by default and some are available from this web site. Packs that were downloaded from here can be imported using the Graylog2 web interface.]
https://www.graylog.org/resource/content-pack/547b5021e4b0a06d87eea01e/