Richard Whaling
@RichardWhaling
Spantree Technology Group
(And how do we do better?)
A log is a record of the events occurring within an organization’s systems and networks.
Logs are composed of log entries.
Each entry contains information related to a specific event that has occurred within a system or network.
NIST Special Publication 800-92:
http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf
at 1 events/minute: 100 * 1 * 60 * 24 = 144 KB / Day
at 1 events/minute: 100 * 1 * 60 * 24 = 144 KB / Day
at 1,000 events/minute: 100 * 1000 * 60 * 24 = 144 MB / Day
at 1 events/minute: 100 * 1 * 60 * 24 = 144 KB / Day
at 1,000 events/minute: 100 * 1000 * 60 * 24 = 144 MB / Day
at 1,000,000 events/minute: 100 * 1000000 * 60 *24 = 144 GB / Day
...
For a self-hosted system, capacity is much more likely to be a bottleneck
We fit an unbounded stream of events into a finite system by discarding older events, i.e., log rotation.
Two ways to plan for capacity and retentions:
(Although these are not entirely distinct categories, they provide distinct perspectives and use cases)
How do we build a log analysis system that is efficient enough to meet all these needs, without imposing an enormous operational burden?
A poorly maintained system can be worse than useless.
Real systems always have tradeoffs.
(Let's not create more problems than we solve)
Testing 3 log analysis systems:
Elasticsearch
OkLog
Humio
Chosen to illustrate technical differences
Not a "versus" talk
I use all of them
Versions:
OkLog 0.2.1
Humio 0.0.37
Elasticsearch 5.2.1
Credit: Digital Ocean, "How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04"
Text
Credit: "Search Basics", Apple SearchKit Developer guide
Credit: "What you get by replicating Lucene indexes on the Infinispan Data Grid", Sanne Giroveno, Red Hat
Elasticsearch | OkLog | Humio |
---|---|---|
Ad Hoc Queries | Streaming Queries | Streaming Queries |
Global Aggregations | Bounded Queries | Prepared Aggregation |
Structured Data | Unstructured Data | Structured Data |
Full Text | Grep | Grep |
Global Search | Ingestion | Compression |
Tiers of service:
Credit: @mipsytipsy (Charity Majors):