Monitoring Docker environments with syslog-ng

Monitoring Myths

"logging daemons are for syslog"
"monitoring and logging are different things"
Log Management Maturity Scale

Let's talk about Monitoring

Log is everything, everything is log
Log = Timestamp + Data
Events, state changes, metrics
Realtime data + historic data
Querying, visualization

Monitoring Tools

Logging: syslog daemons, home-grown scripts, logstash, splunk
Metrics collection: munin, graphite
State checking: nagios
DevOps tools are coming!

Monitoring Methods

logging: push, monitoring: pull
old ways: do the full collecting pipeline
now: integrate everything with everything
see: http://devops.com/features/guide-modern-monitoring-alerting/

Let's see an example!

But first, meet syslog-ng and Docker!

Meet syslog-ng

More than a syslog daemon
Collects, parses, normalizes, correlates, transforms, stores log events
Structured data format.
Swiss army knife of logging

Meet Docker

Containerized applications
Automated deployment
Lightweight virtualization
Vivid ecosystem
PaaS solutions.

Docker monitoring

Docker uses cgroups and several storage backends
Depending on your distro & Docker version, you should vary how you collect metrics.
Docker only containerizes, does not give you a platform for collecting logs & metrics.

Collecting Docker logs

Several ways, no official way
Daemon logs: no logfile for daemon, depending on the distro (ubuntu: /var/log/upstart/docker.log)

Collecting Docker logs #2

stderr and stdout -> json file (problem: no logrotate)
Mapping a "log volume"
Logger daemon inside the container
Map /dev/log into the container. Socket should be up and listening before the container starts.
logspout

Collecting Docker metrics

CPU, Memory: with cgroups, varying by kernel version/distro/docker version.
Storage: dependent on storage driver
- dm: possible
- aufs: well, not so possible (diffs, etc.)
- btrfs, other backends: (?)

The syslog-ng way

Not revolutionary, but combines the possibilities
PoC phase, lot of ways to improve
Still needs coding

The config

@version:3.6
source s_monitor {
   monitor(
     monitor-script("/etc/syslog-ng/docker.lua")
     monitor-func("docker")
     monitor-freq(5)
   );
};

source s_containers {
   log {
     source {
        file("/var/lib/docker/containers/<container_id>/<container_id>-json.log"
         flags(no-parse)
        );
     };
     rewrite { set("<container_id>" value("docker.container_id") ); };
   };
};

parser p_json {
    json-parser( prefix("docker.")  template("$MESSAGE") );
};

destination d_elastic {
  elasticsearch(
    host("es_cluster.mydomain.gtld") port("9200")
    index("docker")
  );
};

destination d_graphite {
   tcp( "172.16.177.139"
     port(2003)
     template("$(graphite-output --key docker.* )")
   );
};

log {
   source(s_monitor);
   destination(d_graphite);
};

log {
  source(s_containers);
  parser(p_json);
  destination(d_elastic);
};

Monitoring script

function _get_image_list()
  local popen = io.popen
  local result = {}
  for image in popen('docker ps -q --no-trunc'):lines() do
    table.insert(result, image)
  end
  return result
end

function docker()
   result = {}
   local images = _get_image_list()
   for i,image in ipairs(images) do
      local f = io.open("/sys/fs/cgroup/memory/docker/"..image.."/memory.usage_in_bytes")
      bytes = f:read("*l")
      result["docker."..image..".memory"] = bytes
   end
   return result
end

Limits

Manually maintain container list in config -> wildcard file source, SCL
Lua -> Python, Java bindings for syslog-ng

Thanks for watching!

Tusa Viktor

tusavik@gmail.com