Centralized Logging Solution using FireLens and Fluent Bit for ECS

David Chou @ Golang Taipei

@ Umbo Computer Vision

@ Golang Taipei

 

david74.chou @ facebook

david74.chou @ medium

david7482 @ github

Golang Taipei Telegram

Golang Taipei Facebook

Why do we need centralized logging?

Why do we need centralized logging?

  • Usability

  • Availability

  • Analyzability

Centralized logging architecture

AWS Container Logging Deep Dive

  • Implemented in Ruby/C @ 2011

  • Ecosystem of Plugins (900+)

  • Extend with Ruby

  • Memory Usage (40MB)

  • Higher CPU Usage

  • Better for Aggregation

  • Support Forward Protocol

  • Implemented in C @ 2015

  • Plugins are Included (56)

  • Extend with C/Go/Lua

  • Reduced Memory Usage (500KB)

  • Lower CPU Usage

  • Better as a Sidecar

  • Support Forward Protocol

KubeCon 2019: Fluent Bit Extending Your Logging Pipeline with Go

AWS Blog: Centralized Container Logging with Fluent Bit

Input

Parser

Filter

Buffer

Output 1

Output 2

Output N

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Information gathering,
e.g. forward, tail, syslog, memory

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Unstructured logs => Structured binary format,
e.g. json, regx (nginx, syslog, etc)

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Input data manipulation,
e.g. grep, record_modifier

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Message buffering & retry

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Tag routing

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Send data to different destinations,
e.g. forward, es, Go plugins

Buffer

Routing

How to write a Fluent Bit Go plugin?

Fluent Bit Go Plugin

fluent-bit

plugin.so

dynamic loading

//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
  // Gets called only once when the .so is loaded.
  return output.FLBPluginRegister(def, "gstdout", "Stdout GO!")
}

//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
  // Gets called once for each instance you have configured.
  param := output.FLBPluginConfigKey(plugin, "param")
  return output.FLB_OK
}

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
  // Gets called once for each message to be written to an instance.
}

//export FLBPluginExit
func FLBPluginExit() int {
  // Gets called on teardown.
  return output.FLB_OK
}

func main() {
}
# Build fluent bit go plugin 
$ go build -o plugin.so -buildmode c-shared plugin.go

# Run Fluent Bit with the new plugin
$ fluent-bit -e plugin.so -i cpu -o gstdout
# Build fluent bit go plugin 
$ go build -o plugin.so -buildmode c-shared plugin.go

# Run Fluent Bit with the new plugin
$ fluent-bit -e plugin.so -i cpu -o gstdout

Go build modes

  • default

  • archive

  • exe

  • pie

  • shared

  • plugin

  • c-archive

  • c-shared

AWS FireLens

AWS FireLens

  • Support a wide-array of AWS Services as log destinations

  • Require no additional configuration beyond ECS Task Definition

  • Use Open Source for extensibility: Fluent Bit, Fluentd

  • Facilitate partner integration

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

AWS for Fluent Bit

AWS for Fluent Bit - Log destination

  • Kinesis Data Firehose
    • S3
    • Amazon ElasticSearch Service
  • Kinesis Data Streams
  • CloudWatch Logs
     
  • Kafka
  • Self-hosted ElasticSearch (#698)
  • DataDog
  • Forward to a Fluentd aggregator

AWS for Fluent Bit - Log destination

$ cat entrypoint.sh 
exec /fluent-bit/bin/fluent-bit \
    -e /fluent-bit/firehose.so \
    -e /fluent-bit/cloudwatch.so \
    -e /fluent-bit/kinesis.so \
    -c /fluent-bit/etc/fluent-bit.conf

github: aws-for-fluent-bit

Use AWS FireLens in ECS

aws-samples/amazon-ecs-firelens-examples

{
    "essential": true,
    "image": "amazon/aws-for-fluent-bit:latest",
    "name": "log_router",
    "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
            "enable-ecs-log-metadata": "true",
            "config-file-type": "file",
            "config-file-value": "/json.conf"
        }
    }
},
{
    "essential": true,
    "image": "<your-app-image>",
    "name": "app",
    "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
            "Name": "cloudwatch",
            "region": "us-west-2",
            "log_group_name": "firelens-fluent-bit",
            "log_stream_prefix": "from-fluent-bit",
            "include-pattern": "[Ee]rror"
        }
   }
}
[INPUT]
    Name forward
    unix_path /var/run/fluent.sock

[INPUT]
    Name forward
    Listen 0.0.0.0
    Port 24224

[FILTER]
    Name   grep
    Match app-firelens*
    Regex  log [Ee]rror

[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster firelens-example
    Record ecs_task_arn arn:aws:ecs:us-west-2:0123456789:task/1111
    Record ecs_task_definition firelens-example-session:5

@INCLUDE /json.conf

[OUTPUT]
    Name cloudwatch
    Match app-firelens*
    region us-west-2
    log_group_name firelens-fluent-bit
    log_stream_prefix from-fluent-bit

ECS task defition

fluent-bit.conf

FireLens internal

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

FireLens internal

  • ECS generates the Fluent Bit config file in Firelens container
    • /fluent-bit/etc/fluent-bit.conf
  • ​Fluent TCP interface
    • FLUENT_HOST (127.0.0.1), FLUENT_PORT (24224)
    • Use SDK to send logs into Fluent Bit instead of stdout

FireLens internal

  • FireLens tag
    • ​<container name>-firelens-<task ID>
      • app-firelens-dcef9dee-d960-4af8-a206-46c31a7f1e67
    • Currently, cannot customized this tag
    • Workaround
      • Fluent Bit stream processor​
      • CREATE STREAM myTag WITH (tag='myTag') AS SELECT * from TAG:'app-firelens-*'

FireLens log loss test

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

FireLens resource usage

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

Future improvement

Future improvement

  • Enhance the reliability of FireLens on Fargate (#700)
    • ​Logs could be lost if Fluent Bit container goes down unexpectedly
    • Firelens container needs to be essential
    • Fargate containers are ephemeral
  • Better Support for Multiple Go Plugins​
$ ls -lh bin/fluent-bit *.so
root root 18M Dec 12 23:14 bin/fluent-bit
root root 24M Jan  08 20:50 cloudwatch.so
root root 26M Dec 12 23:04 firehose.so
root root 26M Dec 12 23:03 kinesis.so

Future improvement

  • Better Support for Multiple Go Plugins​

Future improvement

  • Better Support for Multiple Go Plugins​

Future improvement

Any Question?

Golang Taipei Telegram

Golang Taipei Facebook

問卷往這邊！！

Centralized Logging Solution using FireLens and Fluent Bit for ECS

By Ting-Li Chou

Made with Slides.com

Centralized Logging Solution using FireLens and Fluent Bit for ECS

  • 571

More from Ting-Li Chou