filebeat for ELK

Ming-der Wang

ming@log4analytics.com

遠端資料導入與 filebeat 介紹

TurboTeam 集先鋒科技

架構圖

filebeat

redis

logstash

elasticsearch

kibana

filebeat -> redis -> logstash -> Elasticsearch -> kibana (view)

Data flow

  安裝 filebeat

A Lightweight Shipper for Log Data

<- 點選黃色箭頭下載 Linux, Mac, Windows

Config 檔在 /etc/filebeat/filebeat.yml 

$ sudo service filebeat start

安裝 redis

in-memory data structure storage

<- 點選黃色箭頭下載 Linux, Mac, (no Windows)

$ redis-server

 

$ redis-cli
redis> set foo bar
OK
redis> get foo
"bar"

啟動 server                                     測試 client

或用 docker 執行 redis

https://docs.docker.com/engine/examples/running_redis_service/

點擊它 ->

實作 filebeat -> redis -> logstash -> Elasticsearch -> kibana (view)

filebat (192.168.1.5),   redis + ELK (192.168.1.6) (假設 IP)

filebeat.yml

output:

  redis:

    host: "192.168.1.6"

    port: 6379

    index: "filebeat_test”

logstash.conf

input {

  redis {

    data_type => "list"

    key => "filebeat_test

    batch_count => 100

  }

}

on host: 192.168.1.5                    on host: 192.168.1.6

check data is coming in redis server on 192.168.1.6

[192.168.1.6]$ redis-cli

127.0.0.1:6379> keys *

1) "filebeat_test"

127.0.0.1:6379>

filebeat.yml

filebeat:

  prospectors:

    -

      paths:

        - "/var/log/authd.log"

        - "/tmp/test.log*"

      document_type: authd_test

 

logstash.conf

filter {

  if [type] == “authd_test” { 

  ...

  }

}

 

(if) The Logs are Multi-lines

E Mon May 24 01:00:49 2016  AXA-83  am:19941  am_utils.c(148):6355 2:AXA-83:atp_cgh_loader:1991:10690864:63:74744563:2:root.0.0.0.1:::
    AM AM->SM input flist: opcode=PDM_OP_PUBLISH_GEN_PAYLOAD, flags=0x80, errno=PIN_ERR_AM_CONNECT_FAILED:2628

   0 PIN_FLD_NAME            STR [0] "Account logout"

E Mon May 24 01:00:59 2016  AXA-83  am:19931  am_utils.c(149):6355 2:AXA-83

E Mon May 24 01:01:19 2016  AXA-83  am:19921  am_utils.c(110):6349 2:AXA-83

(if) The Logs are Multi-lines

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/authd.log"
      document_type: authd_test

    multiline:

        pattern: "^[MWDE]"

        negate: true

        match: after

filebeat.yml

logstash.conf

input   {
    file {
         ...
      codec => multiline {
      pattern => "^[MWDE]"
      negate => true
      what => previous
    }
}

    multiline:

        pattern: "^[MWDE]"

        negate: true

        match: after

fields:

    host: 172.16.4.125

    level: debug

    review: 1

 

shipper:

  name: staging

  tags: ["staging"]

More About filebeat.yml

logging:

  level: warning

  to_files: true

 

files:

    path: /var/log/filebeat

    name: authd_filebeat.log

    keepfiles: 7

開啟 logstash

看資料有沒有進來

// 假設你的 logstash.conf 已經做好

$ sudo service logstash restart

output { stdout { codec => rubydebug } }

Tips

  • 先打開 standard output, 確定有 log 到資料
  • 沒有資料, 也有可能 filter 錯誤, 所以可以把 filter 先拿空
  • filebeat 也能直接丟到 elasticsearch, or logstash
  • 直接丟 elasticsearch 就只有 indexing, 沒有 parsing.
  • 還有很多跟 filebeat 一樣的東西, such as topbeat ...
  • https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

Q & A

TurboTeam 集先鋒科技

第二天: 遠端資料導入與 filebeat 介紹

By Ming-der Wang

第二天: 遠端資料導入與 filebeat 介紹

  • 1,346