Elasticsearch, logging, and ninefold




Hi, I'm Ryan 

Lead Rails Support Engineer at Ninefold



What we'll cover



Ninefold's stack (ELK, sorta)
Architecting the system
Getting the logs out
Lessons learned


Logging: 


Until it's not!



What is elasticsearch?


Highly scalable
Auto-clustering
Auto-sharding
Auto-indexing
Auto-magical!
Apache Lucene under the covers



Why Elasticsearch?


See previous slide
Easy to setup
Easy to scale
Readily available resources on the web
Super simple to understand what's going on
Lots of tools/integrations/plugins



Installing and running



$ curl -L -O http://download.elasticsearch.org/PATH/TO/LATEST/$VERSION.zip
$ unzip elasticsearch-$VERSION.zip

$ cd  elasticsearch-$VERSION

$ bin/elasticsearch #-d to daemonize
            
$ curl 'http://localhost:9200/?pretty'


Our supporting roles: 


  • Beaver (https://github.com/josegonzalez/beaver)
  • Logstash
  • RabbitMQ





A little bit about Beaver:


Written in Python
Runs as a daemon on the client
Can listen to and send as many/few logs as you want
Can do some or all log formatting
Formats and transmits logs one line at a time to the queue in JSON
Attaches metadata for later


More on Logstash:


Pulls logs from the queue
Formats logs again and pushes them to Elasticsearch
Grok to easily format messages



Logstash Grok:


...
 {
    # find checkpoint, turn it into a field and add it as a tag
    :grok => {
      :tags => ['cheflog'],
      :pattern => "#{node[:ninefold_logger][:checkpoint_tag]}:%{WORD:checkpoint_type}",
      :tag_on_failure => false,
      :add_tag => "#{node[:ninefold_logger][:checkpoint_tag]}%{checkpoint_type}",
      :singles => true
    },
...


About RabbitMQ:


Fast 
AMQP Messaging
High Availability
Ability to add nodes to existing cluster 

The IDEA:


The IMPLEMENTATION:




Uhh..

LogstashQueueBuildup.png

Ok...


Gimme my logs!



Log Entry Model:


Find and sort them:

Demo time!!




What we've learned


  • Multi-line logs are a challenge
  • Multiple instances of Logstash can complicate things
  • Local storage > network storage 
  • Don't use globs - be specific. 
  • Elasticsearch is awesome!


THANK YOU


ryan@ninefold.com

@I_am_Ryo

github: IAMRYO


Elasticsearch, Logging, and Ninefold

By iamryo

Elasticsearch, Logging, and Ninefold

  • 6,334

More from iamryo