Logging with Docker

faisal@druva.com

What's Docker?

• Containers for app packaging

• Like “chroot” on steroids

• Image shipped as FS layers + metadata

• Container runs with COW FS layer on top of image

• FS layers repository for distribution

Why Docker?

• Change in paradigm
• Transition from monolithic design to micro-services
• Designed for scale
• Easy to add new micro-services
• Dev-ops model - predictable/repeatable services as containers

The downside

• Application is collection of micro-services
• Potentially different logging mechanism used by each service
• Application logs all over the place
• Log files need special handling to get out of the container
• Monitoring applications/containers is challenging

Log/Monitor Strategy

• Application spread across machines
• Different logging mechanisms in use
• Strive to consolidate all logs and metrics centrally
• Analyse logs to make decisions about application health and scale
• At scale, powerful log analysis absolutely necessary 

Log Targets

• Log to file(s)
• Log to stdout
• Log to syslog
• Log to custom logger
• etc.

Log to file(s) on Docker

• Map common volume(s) for logs into each container
• Volume(s) can be docker-only or host-mapped
• Run a collector on the docker host or in container to send collected logs to central logging

Log to stdout

• Recommended by Docker
• Easy to develop for
• May not work for existing apps
• Offload to file/data store
• logspout
• Use log driver

Log to stdout

Log to syslog

• Good support under Linux
• Multiple ways of running under docker

Syslog on host

• Set up logger daemon on host as usual
• Map /dev/log on host to /dev/log on all containers when launching
• Log post-processing/analysis on host machine

Syslog on host

Syslog in container

• Logging container started first
• /some/host/path mapped to /dev for this container
• /some/host/log/output mapped to /var/log
• Each app container launched with /some/host/path/log mapped to /dev/log

Syslog in container

Docker logging driver

• Application logs to stdout
• Log backend can be changed without changing app
• Supports:

1. json-file: default
2. syslog
3. journald (systemd)
4. gelf (Graylog Extended Log Format)
5. fluentd
6. awslogs (AWS cloudwatch logs)

Docker logging driver

Example: syslog driver

• Great feature for non-syslog capable apps
• Leverage existing logger
• Usable with logger running on host or in a single container:

$docker run --log-opt syslog-address==unix://path ...

• Or on a remote host:

$docker run --log-opt syslog-address=[tcp|udp]://host:port  ...