Real Time Analytics
Using the ELK stack
Felipe Montoya
www.edunext.co
{
"username": "FelipeMontoya",
"event_source": "server",
"name": "edx.user.settings.changed",
"accept_language": "en,en-US;q=0.8,de-DE;q=0.6,de;q=0.4,es;q=0.2",
"time": "2015-10-12T02:16:52.652428+00:00",
"agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
"page": null,
"host": "conf2015.edunext.io",
"session": "74e31a596c853f7bcf2d2ca790d7fa5e",
"referer": "http://conf2015.edunext.io/dashboard",
"context": {
"course_user_tags": {},
"user_id": 1,
"org_id": "eduNEXT",
"course_id": "course-v1:eduNEXT+open-edx-con+2015",
"path": "/courses/course-v1:eduNEXT+open-edx-con+2015/info"
},
"ip": "72.74.28.37",
"event": {
"user_id": 1,
"truncated": [],
"old": null,
"setting": "dark-lang",
"table": "user_api_userpreference",
"new": "es-419"
},
"event_type": "edx.user.settings.changed"
}
A log event
A lot of logs
- A medium instance receives between 1 and 3 million events of the tracking logs per week
- An edX size instance receives about 90 million tracking log events per week
Real time?
Real-Time allows you to monitor activity as it happens on your site or app. The reports are updated continuously and each hit is reported seconds after it occurs.
--Google analytics
Insights
- analytics api
- analytics dashboard
- hadoop
Enter ELK stack
See: michael.bouvy.net
Application servers running edxapp
Receiver
Indexer
Collect
Logstash forwarder
Lumberjack
- Lightweight
- Secure
- Configurable
- Fast
Application servers running edxapp
Receiver
Indexer
Index
Logstash indexer
Collect, Enrich, and Store
- Sanitizing
- Formatting
- Analysis via custom filter
Query and visualize
Kibana
Dashboards
Custom queries
Percolator
Store queries and be notified when they match
Advantages
- Speed -> RT
- Scaling up is "easy"
- Interactive
- Query possibilities
ELK is super fun
Disadvantages
- Yet another stack
- Requires maintenance
Not everything is shiny
Lessons learned
- Plugins can be unstable
- Keeping indexes open requires CPU and Memory
- Retention is costly
Next steps
- Query builder
- Alerts using the percolator feature
- Connect to MySql and Mongo
- More Kibana panels
In Action
Thanks
Felipe Montoya
felipe.montoya@edunext.co
real-time-event-tracking-openedx
By Felipe Montoya
real-time-event-tracking-openedx
- 454