Application Metrics & 

DevOps Awesomeness 

With Graphite and Grafana

Logging Analytics

Who am I? 

Torkel Ödegaard

Stockholm

Sweden

"we are survival machines - robot vehicles blindly programmed to preserve the selfish molecules known as genes" 

Why?

• Monitoring
• Logging
• Alerting
• Analytics

Distributed systems

Standard logging solution

Standard metrics solution (win)

Are  there better options?

Elasticsearch

Kibana

Log -> Elasticsearch

LogStash

input {
    tcp {
        type => "log4j"
        port => 3333
    } 
}
filter {
    grok {
        type => "log4j"
        pattern => "%{LOGLEVEL:severity}\s+%{WORD:category} ..."    
        add_tag => "log4j"
    }
    date {
        type => "log4j"
        timestamp => "MM-dd-yyyy hh:mm:ss.SSS a Z"                
    }
}
output {
    elasticsearch { host => "my-elasic-server" }
}

Inputs

input {
    file {
        'path' => '/var/log/apache2/*.log'
        'type' => 'apache-logs'
    }

    redis {
        host => "127.0.0.1"
        type => "redis-input" 
        data_type => "list"
        key => "logstash"    
        message_format => "json_event"
    }
}

Filters

filter {
  grok {
    pattern => "%{COMBINEDAPACHELOG}"
    singles => true
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    locale => "en"
  }
}

filter {
  geoip {
    source => "clientip"    
  }
  useragent {
    source => "agent"    
  }
}

Outputs

output {
    elasticsearch { 
        host => "my-elasic-server" 
    }
    
    redis {
    }
    
    mongodb {
    }
    
    rabbitmq {
    }
}

Metrics / Measurements

Metric vs Log Event

Graphite

Functions!

Metric Libraries

Metric types

Metric.Increment("user.login");            


Metric.Time("auction_search", 142);            


Metric.Time("auction_search", () => search());            
    

apps.devsum.server-01.counters.auction_search.count   15    123123123131
apps.devsum.server-02.counters.auction_search.count    1    123123123131
apps.devsum.server-03.counters.auction_search.count   35    123123123131

apps.devsum.server-01.timers.auction_search.count    5    123123123131
apps.devsum.server-01.timers.auction_search.mean    10    123123123131
apps.devsum.server-01.timers.auction_search.max     50    123123123131
apps.devsum.server-01.timers.auction_search.min      2    123123123131
    

    

Demo

play.grafana.org

Graphite configuration

[stats]
pattern = ^apps.*
retentions = 10s:6h,1min:7d,10min:5y

[stats]
pattern = ^highres.*
retentions = 1s:6h,1min:1d

[stats]
pattern = ^statsd.*
retentions = 1min:1d,10min:1y



    

Time measurements

Percentiles

Percentiles

StatsD

More demo

Future of metrics

Metrics 2.0

Problems

Metrics 2.0

{  server: webapp-01,  datacenter: eu-01,  unit: bytes,  rate: 10s,  metric_type: counter,  stat: images.upload}

Metrics 2.0

Thanks!