Centralized Logging Solution using FireLens and Fluent Bit for ECS

David Chou @ Golang Taipei

@ Umbo Computer Vision

@ Golang Taipei

david74.chou @ facebook

david74.chou @ medium

david7482 @ github

Golang Taipei Telegram

Golang Taipei Facebook

Why do we need centralized logging?

Usability
Availability
Analyzability

Centralized logging architecture

AWS Container Logging Deep Dive

Implemented in Ruby/C @ 2011
Ecosystem of Plugins (900+)
Extend with Ruby
Memory Usage (40MB)
Higher CPU Usage
Better for Aggregation
Support Forward Protocol

Implemented in C @ 2015
Plugins are Included (56)
Extend with C/Go/Lua
Reduced Memory Usage (500KB)
Lower CPU Usage
Better as a Sidecar
Support Forward Protocol

KubeCon 2019: Fluent Bit Extending Your Logging Pipeline with Go

AWS Blog: Centralized Container Logging with Fluent Bit

Input

Parser

Filter

Buffer

Output 1

Output 2

Output N

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Information gathering,
e.g. forward, tail, syslog, memory

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Unstructured logs => Structured binary format,
e.g. json, regx (nginx, syslog, etc)

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Input data manipulation,
e.g. grep, record_modifier

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Message buffering & retry

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Tag routing

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Send data to different destinations,
e.g. forward, es, Go plugins

Buffer

Routing

How to write a Fluent Bit Go plugin?

Fluent Bit Go Plugin

fluent-bit

plugin.so

dynamic loading

//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
  // Gets called only once when the .so is loaded.
  return output.FLBPluginRegister(def, "gstdout", "Stdout GO!")
}

//export FLBPluginInit
func FLBPluginInit(plugin unsafe.Pointer) int {
  // Gets called once for each instance you have configured.
  param := output.FLBPluginConfigKey(plugin, "param")
  return output.FLB_OK
}

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
  // Gets called once for each message to be written to an instance.
}

//export FLBPluginExit
func FLBPluginExit() int {
  // Gets called on teardown.
  return output.FLB_OK
}

func main() {
}

# Build fluent bit go plugin 
$ go build -o plugin.so -buildmode c-shared plugin.go

# Run Fluent Bit with the new plugin
$ fluent-bit -e plugin.so -i cpu -o gstdout

# Build fluent bit go plugin 
$ go build -o plugin.so -buildmode c-shared plugin.go

# Run Fluent Bit with the new plugin
$ fluent-bit -e plugin.so -i cpu -o gstdout

Go build modes

default
archive
exe
pie

shared
plugin
c-archive
c-shared

AWS FireLens

Support a wide-array of AWS Services as log destinations
Require no additional configuration beyond ECS Task Definition
Use Open Source for extensibility: Fluent Bit, Fluentd
Facilitate partner integration

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

AWS for Fluent Bit

A Fluent Bit image with AWS Go plugins
amazon/aws-for-fluent-bit on Docker Hub
Regional public images hosted on ECR

AWS for Fluent Bit - Log destination

Kinesis Data Firehose
- S3
- Amazon ElasticSearch Service
Kinesis Data Streams
CloudWatch Logs
Kafka
Self-hosted ElasticSearch (#698)
DataDog
Forward to a Fluentd aggregator

AWS for Fluent Bit - Log destination

$ cat entrypoint.sh 
exec /fluent-bit/bin/fluent-bit \
    -e /fluent-bit/firehose.so \
    -e /fluent-bit/cloudwatch.so \
    -e /fluent-bit/kinesis.so \
    -c /fluent-bit/etc/fluent-bit.conf

github: aws-for-fluent-bit

Use AWS FireLens in ECS

aws-samples/amazon-ecs-firelens-examples

{
    "essential": true,
    "image": "amazon/aws-for-fluent-bit:latest",
    "name": "log_router",
    "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
            "enable-ecs-log-metadata": "true",
            "config-file-type": "file",
            "config-file-value": "/json.conf"
        }
    }
},
{
    "essential": true,
    "image": "<your-app-image>",
    "name": "app",
    "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
            "Name": "cloudwatch",
            "region": "us-west-2",
            "log_group_name": "firelens-fluent-bit",
            "log_stream_prefix": "from-fluent-bit",
            "include-pattern": "[Ee]rror"
        }
   }
}

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock

[INPUT]
    Name forward
    Listen 0.0.0.0
    Port 24224

[FILTER]
    Name   grep
    Match app-firelens*
    Regex  log [Ee]rror

[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster firelens-example
    Record ecs_task_arn arn:aws:ecs:us-west-2:0123456789:task/1111
    Record ecs_task_definition firelens-example-session:5

@INCLUDE /json.conf

[OUTPUT]
    Name cloudwatch
    Match app-firelens*
    region us-west-2
    log_group_name firelens-fluent-bit
    log_stream_prefix from-fluent-bit

ECS task defition

fluent-bit.conf

FireLens internal

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

FireLens internal

ECS generates the Fluent Bit config file in Firelens container
- /fluent-bit/etc/fluent-bit.conf
Fluent TCP interface
- FLUENT_HOST (127.0.0.1), FLUENT_PORT (24224)
- Use SDK to send logs into Fluent Bit instead of stdout

FireLens internal

FireLens tag
- <container name>-firelens-<task ID>
  - app-firelens-dcef9dee-d960-4af8-a206-46c31a7f1e67
- Currently, cannot customized this tag
- Workaround
  - Fluent Bit stream processor
  - CREATE STREAM myTag WITH (tag='myTag') AS SELECT * from TAG:'app-firelens-*'

FireLens log loss test

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

FireLens resource usage

AWS Blog: Under the hood: FireLens for Amazon ECS Tasks

Future improvement

Enhance the reliability of FireLens on Fargate (#700)
- Logs could be lost if Fluent Bit container goes down unexpectedly
- Firelens container needs to be essential
- Fargate containers are ephemeral

Better Support for Multiple Go Plugins

$ ls -lh bin/fluent-bit *.so
root root 18M Dec 12 23:14 bin/fluent-bit
root root 24M Jan  08 20:50 cloudwatch.so
root root 26M Dec 12 23:04 firehose.so
root root 26M Dec 12 23:03 kinesis.so

Future improvement

Better Support for Multiple Go Plugins

Future improvement

Better Support for Multiple Go Plugins

Future improvement

Any Question?

Centralized Logging Solution using FireLens and Fluent Bit for ECS

@ Umbo Computer Vision

@ Golang Taipei

Golang Taipei Telegram

Golang Taipei Facebook

Why do we need centralized logging?

Why do we need centralized logging?

Usability

Availability

Analyzability

Centralized logging architecture

Implemented in Ruby/C @ 2011

Ecosystem of Plugins (900+)

Extend with Ruby

Memory Usage (40MB)

Higher CPU Usage

Better for Aggregation

Support Forward Protocol

Implemented in C @ 2015

Plugins are Included (56)

Extend with C/Go/Lua

Reduced Memory Usage (500KB)

Lower CPU Usage

Better as a Sidecar

Support Forward Protocol

Input

Parser

Filter

Buffer

Output 1

Output 2

Output N

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Information gathering,e.g. forward, tail, syslog, memory

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Unstructured logs => Structured binary format, e.g. json, regx (nginx, syslog, etc)

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Input data manipulation, e.g. grep, record_modifier

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Message buffering & retry

Buffer

Routing

Input

Parser

Filter

Output 1

Output 2

Output N

Tag routing

Buffer

Routing

Input

Parser

Information gathering,
e.g. forward, tail, syslog, memory

Unstructured logs => Structured binary format,
e.g. json, regx (nginx, syslog, etc)

Input data manipulation,
e.g. grep, record_modifier

Send data to different destinations,
e.g. forward, es, Go plugins