Fluentd and AWS

Gea-Suan Lin

gslin@kkbox.com

(2014/11/25)

Overview

• What is AWS ?
• What is Fluentd ?
• Why to use it ?
• Setup & Demo
• Further Reading

What is AWS ?

• Amazon Web Services
• Introduce in 2006
• EC2 & S3 at beginning
• Beginning of "Cloud Computing"
• More and more services

Simple Storage Service (S3)

• Dynamo: Amazon's High Available Key-value Store
  • CAP theorem
  • Cassandra (Facebook)
  • Voldemort (Linkedin)
• Today we will use S3 to store logs

Amazon S3 Bucket

• s3://bucketname/key

• http://s3.amazonaws.com/bucket/key

• http://bucket.s3.amazonaws.com/key

• http://bucket/key

Manage Amazon S3

• Amazon Web Console
  • Web based interface
• s3cmd
  • Command line interface

What is Fluentd ?

Why to use it ?

• High Performance
  • Ruby and C
• Flexible (Pluggable Architecture)
  • Data Source (Input)
  • Data Output (Output)

Setup & Demo

Further Reading

• High Availability
• Input Plugins:
  • MySQL binlog
  • MySQL slow query log
  • syslog
• Output Plugins:
  • Elasticsearch
  • HDFS

Questions ?