Monitoring Docker environments with syslog-ng

Monitoring Myths

• "logging daemons are for syslog"
• "monitoring and logging are different things"
• Log Management Maturity Scale

Let's talk about Monitoring

• Log is everything, everything is log
• Log = Timestamp + Data
• Events, state changes, metrics
• Realtime data + historic data
• Querying, visualization

Monitoring Tools

• Logging: syslog daemons, home-grown scripts, logstash, splunk
• Metrics collection: munin, graphite
• State checking: nagios
• DevOps tools are coming!

Monitoring Methods

• logging: push, monitoring: pull
• old ways: do the full collecting pipeline
• now: integrate everything with everything
• see: http://devops.com/features/guide-modern-monitoring-alerting/

Let's see an example!

But first, meet syslog-ng and Docker!

Meet syslog-ng

• More than a syslog daemon
• Collects, parses, normalizes, correlates, transforms, stores log events
• Structured data format.
• Swiss army knife of logging

 Meet Docker

• Containerized applications
• Automated deployment
• Lightweight virtualization
• Vivid ecosystem
• PaaS solutions.

Docker monitoring

• Docker uses cgroups and several storage backends
• Depending on your distro & Docker version, you should vary how you collect metrics.
• Docker only containerizes, does not give you a platform for collecting logs & metrics.

Collecting Docker logs

• Several ways, no official way
• Daemon logs: no logfile for daemon, depending on the distro (ubuntu: /var/log/upstart/docker.log)

Collecting Docker logs #2

• stderr and stdout -> json file (problem: no logrotate)
• Mapping a "log volume"
• Logger daemon inside the container
• Map /dev/log into the container. Socket should be up and listening before the container starts.
• logspout

Collecting Docker metrics

• CPU, Memory: with cgroups, varying by kernel version/distro/docker version.
• Storage: dependent on storage driver
  • dm: possible
  • aufs: well, not so possible (diffs, etc.)
  • btrfs, other backends: (?)

The syslog-ng way

• Not revolutionary, but combines the possibilities
• PoC phase, lot of ways to improve
• Still needs coding

The config

@version:3.6
source s_monitor {
   monitor(
     monitor-script("/etc/syslog-ng/docker.lua")
     monitor-func("docker")
     monitor-freq(5)
   );
};

source s_containers {
   log {
     source {
        file("/var/lib/docker/containers/<container_id>/<container_id>-json.log"
         flags(no-parse)
        );
     };
     rewrite { set("<container_id>" value("docker.container_id") ); };
   };
};

parser p_json {
    json-parser( prefix("docker.")  template("$MESSAGE") );
};

destination d_elastic {
  elasticsearch(
    host("es_cluster.mydomain.gtld") port("9200")
    index("docker")
  );
};

destination d_graphite {
   tcp( "172.16.177.139"
     port(2003)
     template("$(graphite-output --key docker.* )")
   );
};

log {
   source(s_monitor);
   destination(d_graphite);
};

log {
  source(s_containers);
  parser(p_json);
  destination(d_elastic);
};

Monitoring script

function _get_image_list()
  local popen = io.popen
  local result = {}
  for image in popen('docker ps -q --no-trunc'):lines() do
    table.insert(result, image)
  end
  return result
end

function docker()
   result = {}
   local images = _get_image_list()
   for i,image in ipairs(images) do
      local f = io.open("/sys/fs/cgroup/memory/docker/"..image.."/memory.usage_in_bytes")
      bytes = f:read("*l")
      result["docker."..image..".memory"] = bytes
   end
   return result
end

Limits

• Manually maintain container list in config -> wildcard file source, SCL
• Lua -> Python, Java bindings for syslog-ng

Thanks for watching!

Tusa Viktor

tusavik@gmail.com

www.syslog-ng.org