Abdoul N'Diaye
@AbdoulNDiaye
http://abdoul-ndiaye.info
Symfony Application's Monitoring with Heka
About Me
- Symfony Lead Developer
- Mostly PHP
- Started SF since 1.4
Let's Get Started
Why collecting data is important?
Because :
- Keep a visibility in complex applications
- Detect potential errors
- Improve your reactivity
- tail is not a solution for everything :)
Logs are full of information BUT they are also full of useless
things when you have a problem to solve !
And now, what do you use ?
http://slides.seld.be/?file=2013-12-13+Application+monitoring+with+Heka+and+statsd.html#5
Nothing ? :(
System monitoring ?
collectd, cacti, nagios, ganglia, munin, ...
Performance / Metrics ?
Statsd, sentry, librato, ...
Error reporting ?
New relic, ...
Log aggregation and routing?
graylog, loggly, splunk, logstash, syslog, ..
+ Kibana, graphite, graphana, etc ...
Consequences....
To many tools
Not enough coffee
All tools have one pattern in common
- Get data
- Transform data
- Deliver data
One Solution
Heka
Made by Mozilla
Written in Go and Lua
Inspired by Logstash
What is already integrated in Heka?
System monitoring !
Performance / Metrics !
Log aggregation and routing !
Usefull dashboard
Basic Pattern
Inputs
Decoders
Router
Outputs
In details...
- Simple to use/debug
- Very fast
- Heka can be extended by plugins in Lua
Heka is also :
Dashboard
Simple Aggregator
Getting started with Heka
v0.8.0
An example
sf2 app
heka
elastic search
Kibana
Main configuration
- Choose your number of CPU’s Core available for Heka
- Activate the dashboard
Our goal
Centralise Symfony's logs
2 differents ways
First method
Keep the original file format of our logs
Advantages
- Keep your original file format in the server
- So, You have several ways to see your logs
Inconvenients
- You have to configure each log file in heka
- Files are not recommended with high traffic websites
Files
Log file configuration example
Configuration in Symfony
Second method
Send our logs via UDP ^^
Advantages
- Performance
- Less configuration in heka
- Better management of logger channels
Inconvenients
- No log if heka daemon is down
Explication Here
UDP
UDP file configuration example
Udp configuration
UDP : Symfony configuration
- Less configurations in symfony
- One input in heka
- Less file opens by the system
- File aggregation can make cry the server
And finally, add an ElasticSearch output
Conclusion
- The most important is the strategy that you will choose after the installation of a log interface
- Heka is not adapted for all case (little website)
- You have to choose your best tools (they are all different)
Download heka in github : Here
Sample of init.d file : Here
Ansible role : Here
Usefull Links
TomL language : Here
Thank you
Questions ?
https://github.com/AbdoulNdiaye
abdoul.nd@gmail.com