filebeat for ELK
Ming-der Wang
ming@log4analytics.com
遠端資料導入與 filebeat 介紹
TurboTeam 集先鋒科技
架構圖
filebeat
redis
logstash
elasticsearch
kibana
filebeat -> redis -> logstash -> Elasticsearch -> kibana (view)
Data flow
安裝 filebeat
A Lightweight Shipper for Log Data
<- 點選黃色箭頭下載 Linux, Mac, Windows
Config 檔在 /etc/filebeat/filebeat.yml
$ sudo service filebeat start
安裝 redis
in-memory data structure storage
<- 點選黃色箭頭下載 Linux, Mac, (no Windows)
$ redis-server
$ redis-cli redis> set foo bar OK redis> get foo "bar"
啟動 server 測試 client
或用 docker 執行 redis
https://docs.docker.com/engine/examples/running_redis_service/
點擊它 ->
實作 filebeat -> redis -> logstash -> Elasticsearch -> kibana (view)
filebat (192.168.1.5), redis + ELK (192.168.1.6) (假設 IP)
filebeat.yml
output:
redis:
host: "192.168.1.6"
port: 6379
index: "filebeat_test”
logstash.conf
input {
redis {
data_type => "list"
key => "filebeat_test”
batch_count => 100
}
}
on host: 192.168.1.5 on host: 192.168.1.6
check data is coming in redis server on 192.168.1.6
[192.168.1.6]$ redis-cli
127.0.0.1:6379> keys *
1) "filebeat_test"
127.0.0.1:6379>
filebeat.yml
filebeat:
prospectors:
-
paths:
- "/var/log/authd.log"
- "/tmp/test.log*"
document_type: authd_test
logstash.conf
filter {
if [type] == “authd_test” {
...
}
}
(if) The Logs are Multi-lines
E Mon May 24 01:00:49 2016 AXA-83 am:19941 am_utils.c(148):6355 2:AXA-83:atp_cgh_loader:1991:10690864:63:74744563:2:root.0.0.0.1:::
AM AM->SM input flist: opcode=PDM_OP_PUBLISH_GEN_PAYLOAD, flags=0x80, errno=PIN_ERR_AM_CONNECT_FAILED:2628
0 PIN_FLD_NAME STR [0] "Account logout"
E Mon May 24 01:00:59 2016 AXA-83 am:19931 am_utils.c(149):6355 2:AXA-83
E Mon May 24 01:01:19 2016 AXA-83 am:19921 am_utils.c(110):6349 2:AXA-83
(if) The Logs are Multi-lines
filebeat:
prospectors:
-
paths:
- "/var/log/authd.log"
document_type: authd_test
multiline:
pattern: "^[MWDE]"
negate: true
match: after
filebeat.yml
logstash.conf
input {
file {
...
codec => multiline {
pattern => "^[MWDE]"
negate => true
what => previous
}
}
multiline:
pattern: "^[MWDE]"
negate: true
match: after
fields:
host: 172.16.4.125
level: debug
review: 1
shipper:
name: staging
tags: ["staging"]
More About filebeat.yml
logging:
level: warning
to_files: true
files:
path: /var/log/filebeat
name: authd_filebeat.log
keepfiles: 7
開啟 logstash
看資料有沒有進來
// 假設你的 logstash.conf 已經做好
$ sudo service logstash restart
output { stdout { codec => rubydebug } }
Tips
- 先打開 standard output, 確定有 log 到資料
- 沒有資料, 也有可能 filter 錯誤, 所以可以把 filter 先拿空
- filebeat 也能直接丟到 elasticsearch, or logstash
- 直接丟 elasticsearch 就只有 indexing, 沒有 parsing.
- 還有很多跟 filebeat 一樣的東西, such as topbeat ...
- https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html
Q & A
TurboTeam 集先鋒科技
第二天: 遠端資料導入與 filebeat 介紹
By Ming-der Wang
第二天: 遠端資料導入與 filebeat 介紹
- 1,346