Real Time Analytics

Using the ELK stack

Felipe Montoya

www.edunext.co

{
  "username": "FelipeMontoya",
  "event_source": "server",
  "name": "edx.user.settings.changed",
  "accept_language": "en,en-US;q=0.8,de-DE;q=0.6,de;q=0.4,es;q=0.2",
  "time": "2015-10-12T02:16:52.652428+00:00",
  "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
  "page": null,
  "host": "conf2015.edunext.io",
  "session": "74e31a596c853f7bcf2d2ca790d7fa5e",
  "referer": "http://conf2015.edunext.io/dashboard",
  "context": {
    "course_user_tags": {},
    "user_id": 1,
    "org_id": "eduNEXT",
    "course_id": "course-v1:eduNEXT+open-edx-con+2015",
    "path": "/courses/course-v1:eduNEXT+open-edx-con+2015/info"
  },
  "ip": "72.74.28.37",
  "event": {
    "user_id": 1,
    "truncated": [],
    "old": null,
    "setting": "dark-lang",
    "table": "user_api_userpreference",
    "new": "es-419"
  },
  "event_type": "edx.user.settings.changed"
}

A log event

A lot of logs

  • A medium instance receives between 1 and 3 million events of the tracking logs per week
  • An edX size instance receives about 90 million tracking log events per week

Real time?

Real-Time allows you to monitor activity as it happens on your site or app. The reports are updated continuously and each hit is reported seconds after it occurs.

 

--Google analytics

Insights

  • analytics api
  • analytics dashboard
  • hadoop

Enter ELK stack

Application servers running edxapp

Receiver

Indexer

Collect

Logstash forwarder

Lumberjack

  • Lightweight
  • Secure
  • Configurable
  • Fast

Application servers running edxapp

Receiver

Indexer

Index

Logstash indexer

Collect, Enrich, and Store

  • Sanitizing
  • Formatting
  • Analysis via custom filter

Query and visualize

Kibana

Dashboards

Custom queries

Percolator

Store queries and be notified when they match

Advantages

  • Speed -> RT
  • Scaling up is "easy"
  • Interactive
  • Query possibilities 

ELK is super fun

Disadvantages

  • Yet another stack
  • Requires maintenance

Not everything is shiny

Lessons learned

  • Plugins can be unstable
  • Keeping indexes open requires CPU and Memory
  • Retention is costly

Next steps

  • Query builder
  • Alerts using the percolator feature
  • Connect to MySql and Mongo
  • More Kibana panels

In Action

Thanks

Felipe Montoya

felipe.montoya@edunext.co

Made with Slides.com