Portal Webserver ELK Logging Architecture

ELK Stack

• Stands for:
  • Elasticsearch
  • Logstash
  • Kibana

Currently:

• We have the E and L implemented. K is a goal of ours
• Logstash and Elasticsearch organize and store the data for retrieval
• Kibana is just a user agent, a frontend to access Elasticsearch with

In summary

• Our software (e.g. the Phoenix API) creates some log in a plaintext file, readable on disk by Filebeat
• Filebeat monitors these logs and sends updates to Logstash
• Logstash transforms into an Elasticsearch-friendly format and sends logs to Elasticsearch
• Logs are then accessible via a REST API on the Elasticsearch endpoint

More info

• Many portal webservers
  • web01-web08.[cluster].[datacenter].synacor.com
  • iterate through each number and cluster
• Each of them running Filebeat locally
• Shipping logs to one central Logstash location
• Which then sends logs to one central Elasticsearch location

In the future:

• Log other portal-based output
• Log other services adjacent to the portal
• Improve logging format
• Actually stand up an instance of Kibana so we don't need to use command line queries anymore

Elasticsearch API Credentials

• The DBA team wants us to keep them secret and secure
• Do not transmit them over email or any on-the-record chat service (like HipChat)
• A few people have the username/password for PS' ES user
• This is why I'd rather get Kibana stood up sooner rather than later -- more secure that way

Limitation:

• Hard disk space
  • Currently we're storing 14 days' worth of logs and deleting any older than that. A cron job runs nightly to delete anything older than that.
  • If we add more services, need to be mindful of how much space usage each service consumes
  • Might just be able to ask for more storage as we need it, though