Abdoul N'Diaye

@AbdoulNDiaye

http://abdoul-ndiaye.info

Symfony Application's Monitoring with Heka

About Me

  • Symfony Lead Developer
  • Mostly PHP
  • Started SF since 1.4

Let's Get Started

Why collecting data is important?

Because :

  • Keep a visibility in complex applications
  • Detect potential errors
  • Improve your reactivity
  • tail is not a solution for everything :)

Logs are full of information BUT they are also full of useless

things when you have a problem to solve !

And now, what do you use ?

http://slides.seld.be/?file=2013-12-13+Application+monitoring+with+Heka+and+statsd.html#5

Nothing ? :(

System monitoring ?

collectd, cacti, nagios, ganglia, munin, ...

Performance / Metrics ?

 

Statsd, sentry, librato, ...

Error reporting ?

New relic, ...

Log aggregation and routing?

graylog, loggly, splunk, logstash, syslog, ..

+ Kibana, graphite, graphana, etc ...

Consequences....

To many tools

Not enough coffee

All tools have one pattern in common 

  • Get data
  • Transform data
  • Deliver data

One Solution

Heka

Made by Mozilla

Written in Go and Lua

Inspired by Logstash

What is already integrated in Heka?

System monitoring !

Performance / Metrics !

Log aggregation and routing !

Usefull dashboard

Basic Pattern

Inputs

Decoders

Router

Outputs

In details...

  • Simple to use/debug
  • Very fast
  • Heka can be extended by plugins in Lua

Heka is also :

Dashboard

Simple Aggregator

Getting started with Heka

v0.8.0

An example

sf2 app

heka

elastic search

Kibana

Main configuration

  • Choose your number of CPU’s Core available for Heka 
  • Activate the dashboard

Our goal

Centralise Symfony's logs

2 differents ways

First method

Keep the original file format of our logs

Advantages

  • Keep your original file format in the server
  • So, You have several ways to see your logs

Inconvenients

  • You have to configure each log file in heka
  • Files are not recommended with high traffic websites

Files

 

Log file configuration example

Configuration in Symfony

Second method

Send our logs via UDP ^^

Advantages

  • Performance
  • Less configuration in heka
  • Better management of logger channels

Inconvenients

  • No log if heka daemon is down

Explication Here

UDP

 

UDP file configuration example

Monolog.lua

Udp configuration

UDP : Symfony configuration

  • Less configurations in symfony
  • One input in heka
  • Less file opens by the system
  • File aggregation can make cry the server

And finally, add an ElasticSearch output

Conclusion

  • The most important is the strategy that you will choose after the installation of a log interface
  • Heka is not adapted for all case (little website)
  • You have to choose your best tools (they are all different)

Download heka in github : Here

Sample of init.d file : Here

Ansible role  : Here

Usefull Links

TomL language : Here 

Thank you

Questions ?

@AbdoulNdiaye

https://github.com/AbdoulNdiaye

abdoul.nd@gmail.com

Symfony2 with heka

By Abdoul N'Diaye

Made with Slides.com

Symfony2 with heka

Web applications have more and more complexity that needs lots of data about our environment. Monitoring, metrics, aggregation, stats... We use a lot of things to monitor application. Heka is an open source stream processing software system developed by Mozilla who centralise all our logs. We will see how we use it in a Symfony application and how heka can centralise the management of your data in one tool.

  • 4,892