Logging with Docker

faisal@druva.com

The Problem

1. Logging is a pain
2. Docker: single process/container
3. App already logs to syslog
4. Consolidating for all containers
5. Network logging for archival, analysis etc

Application stack

• Single logger daemon running on host
• Each app logs to /dev/log
• syslog configured to log locally and/or send to network logger

running without Docker

Application Stack

• One or more apps now run in a container
• Container filesystem is constrained
• What about logging?

running under Docker

Logging with Docker

• Depends on your container architecture
• Potential solutions:

1. Run log daemon on docker host
2. Run log daemon inside each container
3. Run single log daemon in a separate container for all other containers
4. For non-syslogging apps, use syslog driver when launching container

Logger on docker host

Logger on docker host..

• Set up logger daemon on host as usual
• Map /dev/log on host to /dev/log on all containers when launching
• Log post-processing/analysis on host machine

The Good

1. Setup straightforward
2. Easy to test
3. Handle logging for all containers including docker daemon
4. Containers need no change

The Bad

1. Tied to host
2. Deploying at scale
3. Managing at scale

Logger in each container

Logger in each container..

• Logger launched before actual process in each container
• Optionally use fat containers like phusion/baseimage
• /dev/log created by each logger instance prior to actual process startup
• Unless logging to n/w, logger target folder needs to be mapped into each container

The Good

1. Completely self-contained
2. Easy to setup/test
3. Logger failure limited to container
4. Great for standalone containers

The Bad

1. Container custom-built
2. Extra process overhead
3. Cannot log docker daemon logs
4. Upgrading logger requires container rebuild
5. Logging to single file painful

Single Logger Container

Single Logging Container

• Logging container started first
• /some/host/path mapped to /dev for this container
• /some/host/log/output mapped to /var/log
• Each app container launched with /some/host/path/log mapped to /dev/log

The Good

1. Deploy at scale
2. Upgrading logger easy
3. Just another container to run

The Bad

1. Cannot log docker daemon logs

Docker logging driver

• Great for non-syslog compliant apps
• Supports:

1. json-file: default
2. syslog
3. journald (systemd)
4. gelf (Graylog Extended Log Format)
5. fluentd
6. awslogs (AWS cloudwatch logs)

Docker syslog driver

• Great feature for non-syslog capable apps
• Leverage existing logger
• Usable with logger running on host or in a single container:

$docker run --log-opt syslog-address==unix://path ...

• Or on a remote host:

$docker run --log-opt syslog-address=[tcp|udp]://host:port  ...