Centralized App Logging

Goal

Provide a single interface for viewing, searching and analyzing log events created by Matterhorn and related applications.

"requirements"

  • minimal configuration
  • no local agent install required*
  • free text and structured search
  • configurable notifications/alerts
  • multiline exceptions as single event
  • replay existing logs**

* Event forwarding via local agent as an option is OK, but should not be the only method available.

** from services where direct forwarding is not possible, e.g., Epiphan capture agents

Candidates reviewed

  • Loggly
  • Papertrail
  • Logentries
  • Splunk Cloud*
  • ELK (Elasticsearch/Logstash/Kibana)

* Local agent required; methodology differs. Thanks, Miguel!

Methodology

  • rsyslog drains for each candidate on dev05 cluster
  • fabric script for
    • start/stop AWS instances
    • deploy/rollback configuration changes
    • start/stop MH & rsyslog
  • confirm log events sent/recieved
  • execute test MH media upload
  • Review UIs
  • estimate daily log volume based on MH prdAWS cluster and example Epiphan logs

MH log4j config

log4j.rootLogger=ERROR,stdout,file,SYSLOG
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost
log4j.appender.SYSLOG.Facility=Local3
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=matterhorn %p %t %c{1}.%M - %m%n

/opt/matterhorn/etc/services/org.ops4j.pax.logging.properties

rsyslog config

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

/etc/rsyslog.conf

$template Logentries,"38e7cc87-5765-4672-bc16-c54e1cd2190c %HOSTNAME% %syslogtag%%msg%\n"
$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [cfd4ede0-dc56-42ef-8c87-f1504258f4bd@41058] %msg%\n"

:app-name, isequal, "matterhorn" @logs2.papertrailapp.com:24350
& @@logs-01.loggly.com:514;LogglyFormat
& @@api.logentries.com:10000;Logentries
& @@52.0.118.179:1514

/etc/rsyslog.d/22-dce-app-logging.conf

[UI Demos]

Log volume estimates

MH averages based on prdAWS cluster for 01/12/2015 - 03/01/2015

CA averages based on Epiphan instance for 01/22/2015 - 02/12/2015

Cost estimates

Recommendation

Loggly!

 Next steps

  • Explore options for dealing with multiline events
  • Review service TOS for any red flags
  • Discuss configuration & provisioning w/ Naomi & Dan
Made with Slides.com