Data Exploration

with the ELK stack 

@mikaelkaron
EF Labs (Shanghai)

SHOW Clip

Criminal minds 24:05

Sure, I could have gone with any of the CSI flavors, alas I wanted something remotely plausible

So what can we learn about data exploration from watching TV?

  • Who knows about this show?
  • How does the story usually go?
  • Who are the key players?

Introducing your very own

  • DevOps: Collects and normalizes data
  • DevInt: Finds bottlenecks and problems in our applications
  • Developers: Lives code - kills bugs
  • BizInt: Feeds metrics about the busines to owners
  • BizOwn: Drinks coffe and watches YouTube ... ( just kidding )

What kind of questions do we have?

  • How many users signed up between x and y?
  • Check the logs
  • How many signups did this campaign result in?
  • Check the logs
  • When should we schedule downtime?
  • Check the logs
  • Why is the DB slow today?
  • Check the logs

It's all about the logs, so What's the problem?

  • Logs are everywhere, but not all in one place
  • Not everyone have access to logs
  • All devices have their own log formats
2013-11-23 00:00:00 192.168.64.84 GET /Home - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 46
"11/18/2014","10:06:37","Cleared Cache-Run_1^http://ef.com","http://ef.com","3316","1501","0","30484","885741","16","20","46","39","5","0","0","2","0","2290","524","0","0.000","5908","Launch","-1","-1","0","0","http://ef.com","379","2","0","3316","zXqLgJwYsUeXOxsV1UPR2A","0","1","83","92","-1","69","44","100","-1","100","75","18535","851817","6","10","30","28","0","0","0","2","66","ef.com","209.235.2.50","39","0","0","0","1504","200","382720","116752","130907","0","428589","142052","2","1","0","599","1.12","EF - Study Abroad, Cultural Exchange and Work Study Opportunities Worldwide","1611","0","0","0","0","3512","Internet Explorer","9.0.8112.16421","2","29","Akamai","0","1","0","2954",

And about those log formats

  • Same type of data is expressed in different ways (like date and time)
  • A lot of logs is a lot hard to read (and learn how to say properly)
2013-11-23 00:00:00 192.168.64.84 GET /Home - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 46
 
2013-11-23 00:00:00 192.168.64.84 GET /Home - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 46
2013-11-23 00:00:00 192.168.64.84 GET /Login/Logout - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 15
2013-11-23 00:00:01 192.168.64.84 GET /Login/Index - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 15

2013-11-23 00:00:00 192.168.64.84 GET /Home - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 46
2013-11-23 00:00:00 192.168.64.84 GET /Login/Logout - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 15
2013-11-23 00:00:01 192.168.64.84 GET /Login/Index - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 15
2013-11-23 00:00:01 192.168.64.84 GET /combres.axd/LoginCss/1791568736/ - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 0
2013-11-23 00:00:01 192.168.64.84 GET /combres.axd/LoginJs/359896920/ - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 0
2013-11-23 00:00:03 192.168.64.84 GET /Content/images/EF-LOGO.png - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 31
2013-11-23 00:00:04 192.168.64.84 GET /Content/images/people.png - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 1156
2013-11-23 00:00:05 192.168.64.84 GET /Content/images/favicon.ico - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 0
2013-11-23 00:00:07 192.168.64.84 GET /combres.axd/LoginJs/359896920/ - 80 - 221.225.145.203 Mozilla/5.0+(Windows+NT+6.1)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1 200 0 0 15
2013-11-23 00:00:08 192.168.64.84 GET /Content/images/people.png - 80 - 221.225.145.203 Mozilla/5.0+(Windows+NT+6.1)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1 200 0 0 343
2013-11-23 00:00:09 192.168.64.84 GET /Content/images/favicon.ico - 80 - 221.225.145.203 Mozilla/5.0+(Windows+NT+6.1)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1 200 0 0 203
2013-11-23 00:00:12 192.168.64.84 POST / - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 62
2013-11-23 00:00:12 192.168.64.84 GET /akamai-test-object.mp4 - 80 - 59.173.17.246 Http+Fetcher 404 0 0 15
2013-11-23 00:00:12 192.168.64.84 GET /Login/Index encodedToken=QAAAANskY%2fHq%2b%2fAmPkfVWXNdEMiiNWRua4GPbP65TU%2bgXMdGF9B2xZw%2b6vb3HiILfSpuakXILjcFRr7y%2fLjP55uFgRUzzLUf3vNc0LLjwPwh%2bXl1VCIueQ%3d%3d%5eFRA%5eCN%5e%25e9%259f%25a9%25e9%259d%259e%5e11745715%5ehanson217076%5e1004%7e1005%7e1006%5e0%5e388048 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 302 0 0 62
2013-11-23 00:00:13 192.168.64.84 GET /Home - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 281
2013-11-23 00:00:15 192.168.64.84 GET /Content/images/loading.gif - 80 - 106.114.67.123 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/31.0.1650.8+Safari/537.36 200 0 0 15

SHOW Clip

The Matrix (1999) 02:05:33

  • That was badass!

Introducing:

E.L.K

  • Collect
  • Democratize
  • Index
  • Visualize

Logstash

  • Plumbing and glue for data
  • 160+ connectors
  • Ritch mans form of ETL

Elasticsearch

  • Near time search and analysis

Kibana

  • Visualize and explore

ELK

  • Scales like crazy
  • Fully open source
  • Commercially supported
    • Development
    • Production support
    • Training

Demo time!

So what's next?

  • Connect with Apache Spark (Streaming)
  • Connect with PrestoDB (SQL)
  • Connect with PredictionIO (AI)
  • Connect with Sentry (DevOPS)

  • Collect from MSSQL
  • Collect from Appliances
  • Collect from IOT

Questions?

Data Exploration With ELK

By Mikael Karon

Data Exploration With ELK

Simple data exploration using the ELK stack

  • 823

More from Mikael Karon