ELK Intro

Brief Intro

• Scheme-free, REST & JSON based document store
• Distributed and horizontally scalable
• Search, analyze in real-time.
• based on Lucene
• Open Source: Apache License 2.0/ Java
• Zero Configuration
• Plugins

see more: http://www.elasticsearch.org/overview/elasticsearch/

Brief Intro

• Collect various data
• Parse and enrich data
• Output to a variety of endpoint
• Plugins: Inputs, Filters, Outputs
• Open Source: Apache License 2.0/ (J)Ruby

see more: http://logstash.net/

Brief Intro

• Visualize data
• Flexible and powerful
• Dashboards
• Panels
• Execute queries and show results
• Filter data

see more: http://www.elasticsearch.org/guide/en/kibana/current/index.html

Come to Work Together

Architecture

Distributed Log Analysis

Case 1

Indexer

Shipper

Case 2

Indexer

Buffer

Case 3

Case 4

Case 5

Case 6

Scale ELK is Easy !!

Getting Started is Simple

Install ES 1.3.2

1. wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz -O elasticsearch-1.3.2.tar.gz

2. tar zxvf elasticsearch-1.3.2.tar.gz

3. cd elasticsearch-1.3.2 && bin/elasticsearch

4. curl -X GET http://localhost:9200/

Install Kibana 3.1.0

1. cd to your web server's( e.g. nginx) www root 

2. wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz -O kibana-3.1.0.tar.gz

3. tar zxvf kibana-3.1.0.tar.gz

4.configure elasticsearch host in  config.js

5. open your web browser with url http://your_server/kibana-3.1.0

Install Logstash 1.4.2

1. wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz -O logstash-1.4.2.tar.gz

2. tar zxvf logstash-1.4.2.tar.gz

3. cd logstash-1.4.2 && bin/logstash agent -f logstash_test.conf

# logstash_test.conf : a simple configuration example of logstash
input {
    stdin {}
}

filter {
    grok {
        match => [ "message", "%{NUMBER:http_status} %{WORD:method}" ]
    }
    if [ http_status ] =~ /^[4-5]\d\d/ {
        drop {}
    }
}

output {
    stdout { codec => rubydebug }
}

References

1.http://www.elasticsearch.org

2.http://logstash.net

3.https://speakerdeck.com/elasticsearch/scale-12x-introduction-to-elasticsearch-logstash-and-kibana

4.http://garyelephant.me/article/details/23750563