Collector

Marcin "Perk" Stożek

Journey from Fluent Bit, Fluentd and Prometheus to

Collectors Zoo

Collector

Marcin Stożek "Perk"

Journey from Fluent Bit, Fluentd and Prometheus to

Collector

Marcin Stożek "Perk"

Journey from Fluent Bit, Fluentd and Prometheus to

Collectors Zoo

where it all started?

de facto k8s

telemetry data collection standards
in early 2023

logs

metrics

traces

?

logs

a lot of plugins

community, documentation

written in Ruby

logs

a lot of plugins

community, documentation

written in Ruby

Exhibit 1 - kubeclient gem

used by more than 1k projects

1 (one) active maintainer

logs

Exhibit 2 - multithreading vs HPA

1 worker = 1 thread

resources.requests.cpu = 2000

autoscaling.targetCPUUtilizationPercentage = 50

logs

metrics

traces

?

crazy fast

great memory usage

written in C

logs

metrics

traces

?

metrics

community, documentation

memory usage

database, not a forwarder *

logs

metrics

traces

?

traces

?

traces

why?

OpenTelemetry

101

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md

Vendor provided

OpenTelemetry provided

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md

No OpenTelemetry backend

Vendor provided

Vendor support

www.cncf.io/reports/opentelemetry-project-journey-report/

OpenTelemetry

Specification

Language SDKs

Collector

OpenTelemetry

Specification

Language SDKs

Collector

Specification

Guidelines - cross language requirements and expectations for all implementations

Semantic conventions

API, SDK

OTLP

https://github.com/open-telemetry/opentelemetry-specification

Specification

Guidelines - cross language requirements and expectations for all implementations

Semantic conventions

API, SDK

OTLP

https://github.com/open-telemetry/opentelemetry-specification

Community driven
industry standard

OpenTelemetry

Specification

Language SDKs

Collector

OpenTelemetry

Specification

Language SDKs

Collector

Language SDKs

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md

Instrumentation

https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md

Language SDKs

OpenTelemetry

Specification

Language SDKs

Collector

OpenTelemetry

Specification

Language SDKs

Collector

OpenTelemetry

Collector

101

https://github.com/open-telemetry/opentelemetry-specification/tree/v1.21.0/specification/logs

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/overview.md

Collection

OpenTelemetry Collector

https://opentelemetry.io/docs/collector/

Receivers

Processors

Exporters

OpenTelemetry Collector

https://opentelemetry.io/docs/collector/

Receivers

Processors

Exporters

Pipelines

OpenTelemetry Collector

aka otelcol

https://opentelemetry.io/docs/collector/

Host Metrics Receiver

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.78.0/receiver/hostmetricsreceiver

Resource Detection Processor

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.78.0/processor/resourcedetectionprocessor

OTLP Exporter

https://github.com/open-telemetry/opentelemetry-collector/tree/v0.78.0/exporter/otlpexporter

Put it all together

receivers:
  hostmetrics:
    scrapers:
      memory:

processors:
  resourcedetection/detect-host-name:
    detectors:
    - system
    system:
      hostname_sources:
      - os

exporters:
  otlp:
    endpoint: otelcol2:4317

service:
  pipelines:
    metrics:
      receivers:
      - hostmetrics
      processors:
      - resourcedetection/detect-host-name
      exporters:
      - otlp

receivers:
  hostmetrics:
    scrapers:
      memory:

processors:
  resourcedetection/detect-host-name:
    detectors:
    - system
    system:
      hostname_sources:
      - os

exporters:
  otlp:
    endpoint: otelcol2:4317

service:
  pipelines:
    metrics:
      receivers:
      - hostmetrics
      processors:
      - resourcedetection/detect-host-name
      exporters:
      - otlp

Data Pipeline

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection/detect-host-name]
      exporters: [otlp]
      
    metrics/kafka:
      receivers: [kafka]
      exporters: [otlp/kafka]
      
    logs:
      receivers: [filelog]
      exporters: [otlp]
    
    logs/kafka:
      receivers: [kafka]
      exporters: [otlp, otlp/kafka]
      
    traces:
      receivers: [otlp, kafka]
      processors: [resourcedetection/detect-host-name]
      exporters: [otlp, otlp/kafka]