OOYALA

HACKATHON

2014

Project

The Puppet Master

(application/system logging, indexing and searching)

WHY?

- Large amount of application logs (S3) (~30GB/day/app)

- Splunk stores data raw

- Logs are unstructured

- Expensive and slow to store and search

1. 30 minutes to get trending data
2. 5 minutes between searches
3. Sometimes no results return

Use cases

• Trending of errors code during/after deployment
• Troubleshooting
• Real-time log tracking (required full solution)

Interim solution

• Ingest finite set of data (days or weeks)
• Load data onto local machine
• Index and search
• All in 1 command (kitchen/chef/ELK)

Lession learned: Daily log data from all Regions is over 30GB compressed.

Unicorn logs before

{
       "message" => "{\"timestamp\":1408676404,\"report\":{\"ServerRoot: request_url\":\"https://sas.ooyala.com/secure_key?embed_code=sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY&key_token=qHFrDooTAlU4cUUVYqssFhrrNAi3NA34fgRhZGaDco8%3D&signature=8be9aec26cbba1cdc8a194832dbc7f80499f60e2&secure_ios_token=WndEQzE0L2RmRFA1czQyQUNNeEtyUFBueklmdlFyZ1VhcGp5WTNRUVZBcEhzNVJEV3ovZGlVa29obS94Cnd5bU1jZjRWa3dBZ3NGdlhoR0lRbUwrQ21BPT0K\",\"Datastore: Cache Hit on key: \":\"sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY\",\"request_id\":\"168328f01b7df1cf4f5e2d8439afb701\"}}",
      "@version" => "1",
    "@timestamp" => "2014-08-24T23:24:22.534Z",
          "host" => "default-ubuntu-1204",
     "timestamp" => 1408676404,
           "pid" => "364",
        "report" => {
              "ServerRoot: request_url" => "https://sas.ooyala.com/secure_key?embed_code=sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY&key_token=qHFrDooTAlU4cUUVYqssFhrrNAi3NA34fgRhZGaDco8%3D&signature=8be9aec26cbba1cdc8a194832dbc7f80499f60e2&secure_ios_token=WndEQzE0L2RmRFA1czQyQUNNeEtyUFBueklmdlFyZ1VhcGp5WTNRUVZBcEhzNVJEV3ovZGlVa29obS94Cnd5bU1jZjRWa3dBZ3NGdlhoR0lRbUwrQ21BPT0K",
        "Datastore: Cache Hit on key: " => "sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY",
                           "request_id" => "168328f01b7df1cf4f5e2d8439afb701"
    }
}

A, [2014-08-22T03:00:04.648441 #364]   ANY -- : {"timestamp":1408676404,"report":
{"ServerRoot: request_url":"https://sas.ooyala.com/secure_key?embed_code=sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY&key_token=qHFrDooTAlU4cUUVYq
ssFhrrNAi3NA34fgRhZGaDco8%3D&signature=8be9aec26cbba1cdc8a194832dbc7f80499f60e2&secure_ios_token=
WndEQzE0L2RmRFA1czQyQUNNeEtyUFBueklmdlFyZ1VhcGp5WTNRUVZBcEhzNVJEV3ovZGlVa29obS94Cnd5bU1jZj
RWa3dBZ3NGdlhoR0lRbUwrQ21BPT0K","Datastore: Cache Hit on key: ":"sxcHRybzqy4UuyhFVkgtII3YPTxGT2CY",
"request_id":"168328f01b7df1cf4f5e2d8439afb701"}}

after...

Helios log before

  (0.04) SELECT * FROM `users` WHERE (`api_key` = 'dvamMxOk36q9rMNQNbZ6IEvlBc0F.9jzPP') LIMIT 1
  (0.00) SELECT * FROM `providers` WHERE (`providers`.`id` = 58264) LIMIT 1
  (0.00) SELECT * FROM `movies` WHERE ((`movies`.`provider_id` = 58264) AND (movies.status = 'uploading') AND (movies.updated_at > '2014-07-24')) ORDER BY `movies`.`id` DESC LIMIT 101 OFFSET 0
  (0.00) SELECT `id`, `ad_set_code` FROM `ad_sets` WHERE (`id` != `id`)
  (0.00) SELECT `movies`.`id`, `primary_preview_images`.`url` FROM `movies` INNER JOIN `primary_preview_images` ON (`primary_preview_images`.`movie_id` = `movies`.`id`) WHERE ((`movie_id` != `movie_id`) AND (`content_type` NOT IN ('Channel', 'MultiChannel')))
3a7d5748187d11e49e - 127.0.0.1 - - - dvamMxOk36q9rMNQNbZ6IEvlBc0F.9jzPP - - - - - [31/Jul/2014 06:38:06] 
"GET /assets?api_key=dvamMxOk36q9rMNQNbZ6IEvlBc0F.9jzPP&where=status%3D%27uploading%27+
and+updated_at%3E%272014-07-24%27&user_permission=manage HTTP/1.1" 200 12 0.1172 119M +0k

after...

"_index": "logstash-2014.08.22",
  "_type": "logs",
  "_id": "PtMb7D83Q7ij0LnA7agNzA",
  "_score": null,
  "_source": {
    "@timestamp": "2014-08-22T06:43:01.000Z",
    "@version": "1",
    "tags": [
      "multiline"
    ],
    "host": "default-ubuntu-1204",
    "path": "/opt/instance_data/helios/extracted/server.0.log-20140822",
    "queries": [
      "(0.00) SELECT * FROM `users` WHERE (`api_key` = '1pYW86dh8wN5U2uZleq3KBViWU61.nhpjg') LIMIT 1",
      "(0.00) SELECT * FROM `providers` WHERE (`providers`.`id` = 29768) LIMIT 1",
      "(0.00) SELECT * FROM `movies` WHERE ((`movies`.`provider_id` = 29768) AND (movies.embed_code = 'tmM2ZkaTpDTq4jwcuG3oHDMngtB7FN9t')) GROUP BY `id` ORDER BY `movies`.`id` DESC LIMIT 101 OFFSET 0",
    ],
    "http_x_guid": "8f73617429c711e48f",
    "remote_addr": "127.0.0.1",
    "remote_user": "-",
    "http_x_api_key": "1pYW86dh8wN5U2uZleq3KBViWU61.nhpjg",
    "http_x_pcode": "-",
    "http_x_provider_id": "-",
    "timestamp": "22/Aug/2014 06:43:01",
    "request_method": "GET",
    "path_info": "/assets?api_key=1pYW86dh8wN5U2uZleq3KBViWU61.nhpjg&include=labels%2Cmetadata&limit=100&where=embed_code%3D%27tmM2ZkaTpDTq4jwcuG3oHDMngtB7FN9t%27&orderby=&user_permission=manage",
    "http_version": "HTTP/1.1",
    "response_status": "200",
    "length": "977",
    "duration": "0.2379",
    "memory_used": "115M",
    "memory_diff": "+0k"
  },
  "sort": [
    1408689781000,
    1408689781000
  ]

Kibana results dashboard

Full scale proposal

Quick Demo

Questions?

Code at Ooyala's Github

https://github.services.ooyala.net/infra/hackathon-pm