Centralized App Logging

Goal

Provide a single interface for viewing, searching and analyzing log events created by Matterhorn and related applications.

"requirements"

• minimal configuration
• no local agent install required*
• free text and structured search
• configurable notifications/alerts
• multiline exceptions as single event
• replay existing logs**

* Event forwarding via local agent as an option is OK, but should not be the only method available.

** from services where direct forwarding is not possible, e.g., Epiphan capture agents

Candidates reviewed

• Loggly
• Papertrail
• Logentries
• Splunk Cloud*
• ELK (Elasticsearch/Logstash/Kibana)

* Local agent required; methodology differs. Thanks, Miguel!

Methodology

• rsyslog drains for each candidate on dev05 cluster
• fabric script for
  • start/stop AWS instances
  • deploy/rollback configuration changes
  • start/stop MH & rsyslog
• confirm log events sent/recieved
• execute test MH media upload
• Review UIs
• estimate daily log volume based on MH prdAWS cluster and example Epiphan logs

MH log4j config

log4j.rootLogger=ERROR,stdout,file,SYSLOG
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost
log4j.appender.SYSLOG.Facility=Local3
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=matterhorn %p %t %c{1}.%M - %m%n

/opt/matterhorn/etc/services/org.ops4j.pax.logging.properties

rsyslog config

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

/etc/rsyslog.conf

$template Logentries,"38e7cc87-5765-4672-bc16-c54e1cd2190c %HOSTNAME% %syslogtag%%msg%\n"
$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [cfd4ede0-dc56-42ef-8c87-f1504258f4bd@41058] %msg%\n"

:app-name, isequal, "matterhorn" @logs2.papertrailapp.com:24350
& @@logs-01.loggly.com:514;LogglyFormat
& @@api.logentries.com:10000;Logentries
& @@52.0.118.179:1514

/etc/rsyslog.d/22-dce-app-logging.conf

[UI Demos]

Log volume estimates

MH averages based on prdAWS cluster for 01/12/2015 - 03/01/2015

CA averages based on Epiphan instance for 01/22/2015 - 02/12/2015

Cost estimates

Recommendation

Loggly!

 Next steps

• Explore options for dealing with multiline events
• Review service TOS for any red flags
• Discuss configuration & provisioning w/ Naomi & Dan