Elasticsearch, logging, and ninefold

Hi, I'm Ryan

Lead Rails Support Engineer at Ninefold

What we'll cover

Ninefold's stack (ELK, sorta)

Architecting the system

Getting the logs out

Lessons learned

Logging:

Until it's not!

What is elasticsearch?

Highly scalable
Auto-clustering
Auto-sharding
Auto-indexing
Auto-magical!
Apache Lucene under the covers

Why Elasticsearch?

See previous slide
Easy to setup
Easy to scale
Readily available resources on the web
Super simple to understand what's going on
Lots of tools/integrations/plugins

Installing and running

$ curl -L -O http://download.elasticsearch.org/PATH/TO/LATEST/$VERSION.zip
$ unzip elasticsearch-$VERSION.zip

$ cd  elasticsearch-$VERSION

$ bin/elasticsearch #-d to daemonize
            
$ curl 'http://localhost:9200/?pretty'

Our supporting roles:

Beaver (https://github.com/josegonzalez/beaver)
Logstash
RabbitMQ

A little bit about Beaver:

Written in Python

Runs as a daemon on the client

Can listen to and send as many/few logs as you want

Can do some or all log formatting

Formats and transmits logs one line at a time to the queue in JSON

Attaches metadata for later

More on Logstash:

Pulls logs from the queue

Formats logs again and pushes them to Elasticsearch

Grok to easily format messages

Logstash Grok:


...
 {
    # find checkpoint, turn it into a field and add it as a tag
    :grok => {
      :tags => ['cheflog'],
      :pattern => "#{node[:ninefold_logger][:checkpoint_tag]}:%{WORD:checkpoint_type}",
      :tag_on_failure => false,
      :add_tag => "#{node[:ninefold_logger][:checkpoint_tag]}%{checkpoint_type}",
      :singles => true
    },
...