ELK deployment with docker-compose

Ming-der Wang

ming@log4analytics.com

最基本的 DevOps 開發環境

 ELK 也可以這麼用 (我覺得更好)

我們可以依不同需求和價格來部署

開發人員

QA 測試人員

業務

客服

Operator, MIS

(jenkins)

production ->

staging ->

testing ->

development ->

docker machine 開啟和選用

docker-machine ls

docker-machine create --driver google --google-project  crested-studio-139023 vm01

docker-machine create dev -d "virtualbox"

以 virtualbox 為例

Microsoft Azure 

$ docker-machine create -d azure --azure-image canonical:UbuntuServer:16.04.0-LTS:16.04.201608150   --azure-ssh-user mwang  --azure-subscription-id $AZURE_SUBSCRIPTION_ID  --azure-open-port 80  az01

 

$ docker-machine env az01

$ eval $(docker-machine env az01)

Digital Ocean

$ docker-machine create \

    --digitalocean-image ubuntu-16-04-x64 \
    --driver digitalocean \
    --digitalocean-access-token $DOTOKEN staging

 

$ docker-machine env staging

$ eval $(docker-machine env stagin)

Google Cloud

// 先 gcloud auth

$ docker-machine create --driver google --google-project  crested-studio-139023 vm03

 

$ docker-machine env vm03

$ eval $(docker-machine env vm03)

Scaleway

$ docker-machine create -d scaleway --scaleway-name="cloud-scaleway-1" --scaleway-token=$TOKEN --scaleway-organization=$ACCESS_KEY csw01

 

$ docker-machine env csw01

$ eval $(docker-machine env csw01)

Scaleway CLI - scw

$ GO15VENDOREXPERIMENT=1 go get -u github.com/scaleway/scaleway-cli/cmd/scw

 

$ scw version

Client version: v1.11.1
Go version (client): go1.7.3
Git commit (client): homebrew
OS/Arch (client): darwin/amd64

 

$ scw login

// 適用來開 k8s

用 docker machine 來切換

eval $(docker-machine env)

eval $(docker-machine env vm01)

註: vm01 是開在 Google Compute Engine

vm03 開在 Google Cloud 為例

vm03 在這裡

docker-compose up

$ git clone -b v1.0-google-cloud-ok https://github.com/mingderwang/docker-elk

$ cd docker-elk

$ docker-compose up

docker ps

注意: kibana 的 5601 已經被我們轉到 80 port 了

http://104.197.107.96:9200/_cat

貓在哪裡?

http://104.197.107.96

echo "ming test" | nc 104.197.107.96 5000

docker version Logstash

docker run -it -v /vagrant:/data --rm docker.elastic.co/logstash/logstash-oss:6.1.1 -V

docker version E + K

wget https://gist.githubusercontent.com/mingderwang/15430cd896066f650008950807d001c1/raw/b5dcea79b7254e457f06fdb94d2389d9d831f760/stack.yml -O docker-compose.yml

 

docker-compose up

or

https://labs.play-with-docker.com/?stack=https://gist.githubusercontent.com/mingderwang/15430cd896066f650008950807d001c1/raw/b5dcea79b7254e457f06fdb94d2389d9d831f760/stack.yml 

live demo

Q & A

ming@log4analytics.com

devopstaiwan.slack.com

TurboTeam 急先鋒

https://github.com/mingderwang/docker-elk

如何改成可以在 Google Cloud 執行

Google 防火牆要打開 5601

*port 9200 理論上不可以開, for testing only

但同樣的 deployment, Google Cloud 還是有問題

Elasticsearch OK

Kibana ERROR

 

 

Logstash ERROR

 

 

docker-compose.yml

經過修改, 就可以在 Google 跑

docker-compose.yml

kibana/Dockerfile

logstash/Dockerfile

成功 http://130.211.124.148:5601/

作業 1: 部署兩台 Elasticsearch

作業 2: 改用 kubernetes 部署

改成 kubernetes 格式,

在 minikube 測試

demo ELK deploy with docker-compose

By Ming-der Wang

demo ELK deploy with docker-compose

  • 1,820