Fluentd and AWS

Gea-Suan Lin

gslin@kkbox.com

(2014/11/25)

Overview

  • What is AWS ?
  • What is Fluentd ?
  • Why to use it ?
  • Setup & Demo
  • Further Reading

What is AWS ?

  • Amazon Web Services
  • Introduce in 2006
  • EC2 & S3 at beginning
  • Beginning of "Cloud Computing"
  • More and more services

Simple Storage Service (S3)

  • Dynamo: Amazon's High Available Key-value Store
    • CAP theorem
    • Cassandra (Facebook)
    • Voldemort (Linkedin)
  • Today we will use S3 to store logs

Amazon S3 Bucket

  • s3://bucketname/key
  • http://s3.amazonaws.com/bucket/key
  • http://bucket.s3.amazonaws.com/key
  • http://bucket/key

Manage Amazon S3

What is Fluentd ?

Why to use it ?

  • High Performance
    • Ruby and C
  • Flexible (Pluggable Architecture)
    • Data Source (Input)
    • Data Output (Output)

Setup & Demo

Further Reading

  • High Availability
  • Input Plugins:
    • MySQL binlog
    • MySQL slow query log
    • syslog
  • Output Plugins:
    • Elasticsearch
    • HDFS

Questions ?