ELK Intro

Brief Intro

  • Scheme-free, REST & JSON based document store
  • Distributed and horizontally scalable
  • Search, analyze in real-time.
  • based on Lucene
  • Open Source: Apache License 2.0/ Java
  • Zero Configuration
  • Plugins

Brief Intro

  • Collect various data
  • Parse and enrich data
  • Output to a variety of endpoint
  • Plugins: Inputs, Filters, Outputs
  • Open Source: Apache License 2.0/ (J)Ruby

Brief Intro

  • Visualize data
  • Flexible and powerful
  • Dashboards
  • Panels
  • Execute queries and show results
  • Filter data

Come to Work Together

Architecture

Distributed Log Analysis

Case 1

Indexer

Shipper

Case 2

Indexer

Buffer

Case 3

Case 4

Case 5

Case 6

Scale ELK is Easy !!

Getting Started is Simple

Install ES 1.3.2

1. wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz -O elasticsearch-1.3.2.tar.gz

2. tar zxvf elasticsearch-1.3.2.tar.gz

3. cd elasticsearch-1.3.2 && bin/elasticsearch

4. curl -X GET http://localhost:9200/

Install Kibana 3.1.0

1. cd to your web server's( e.g. nginx) www root 

2. wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz -O kibana-3.1.0.tar.gz

3. tar zxvf kibana-3.1.0.tar.gz

4.configure elasticsearch host in  config.js

5. open your web browser with url http://your_server/kibana-3.1.0

Install Logstash 1.4.2

1. wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz -O logstash-1.4.2.tar.gz

2. tar zxvf logstash-1.4.2.tar.gz

3. cd logstash-1.4.2 && bin/logstash agent -f logstash_test.conf

# logstash_test.conf : a simple configuration example of logstash
input {
    stdin {}
}

filter {
    grok {
        match => [ "message", "%{NUMBER:http_status} %{WORD:method}" ]
    }
    if [ http_status ] =~ /^[4-5]\d\d/ {
        drop {}
    }
}

output {
    stdout { codec => rubydebug }
}

References

1.http://www.elasticsearch.org

2.http://logstash.net

3.https://speakerdeck.com/elasticsearch/scale-12x-introduction-to-elasticsearch-logstash-and-kibana

4.http://garyelephant.me/article/details/23750563

ELK Intro

By Gary Gao

ELK Intro

Introduction of distributed Log Collect & Analysis System built by Elasticsearch + Logstash + Kibana

  • 4,751