{}
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
}
]
}
(more machines)
for
security, monitoring, messaging, alerting and more..
curl -XPUT 'http://cluster:9200/index/type/' -d@somedata.json
Let's take some data and 'analyze' it...
Input string > character filters > tokenizer > token filters > index
=="inverted index"
Standard Analyzer
Simple Analyzer
Whitespace Analyzer
Stop Analyzer
Keyword Analyzer
Pattern Analyzer
Language Analyzers
Snowball Analyzer
Custom Analyzer
Preprocesses the string of characters before it is passed to the tokenizer.
A character filter may be used to strip out HTML markup,
or to convert "&" characters to the word "and".
Mapping Char Filter
HTML Strip Char Filter
Pattern Replace Char Filter
Breaks a string down into a stream of terms or tokens.
A simple tokenizer might split the string up into terms wherever it encounters whitespace or punctuation.
Standard Tokenizer
Edge NGram Tokenizer
Keyword Tokenizer
Letter Tokenizer
Lowercase Tokenizer
NGram Tokenizer
Whitespace Tokenizer
Pattern Tokenizer
UAX Email URL Tokenizer
Path Hierarchy Tokenizer
Classic Tokenizer
Thai Tokenizer
Accepts a stream of tokens from a tokenizer and can:
modify tokens (eg lowercasing)
delete tokens (eg remove stopwords)
add tokens (eg synonyms)
Standard Token Filter
ASCII Folding Token Filter
Length Token Filter
Lowercase Token Filter
Uppercase Token Filter
NGram Token Filter
Edge NGram Token Filter
Porter Stem Token Filter
Shingle Token Filter
Stop Token Filter
Word Delimiter Token Filter
Stemmer Token Filter
Stemmer Override Token Filter
Keyword Marker Token Filter
Keyword Repeat Token Filter
KStem Token Filter
Snowball Token Filter
Phonetic Token Filter
Synonym Token Filter
Compound Word Token Filter
Reverse Token Filter
Elision Token Filter
Truncate Token Filter
Unique Token Filter
Pattern Capture Token Filter
Pattern Replace Token Filter
Trim Token Filter
Limit Token Count Token Filter
Hunspell Token Filter
Common Grams Token Filter
Normalization Token Filter
CJK Width Token Filter
CJK Bigram Token Filter
Delimited Payload Token Filter
Keep Words Token Filter
Keep Types Token Filter
Classic Token Filter
Apostrophe Token Filter
Decimal Digit Token Filter
Stores:
the original document
+ some metadata
+ terms in inverted index
Input
Tokenizer
Standard
Character Filter
HTMLStripper
TokenFilter
Stopwords
TokenFilter
Lowercase
Index
"<div>The Quick Brown Fox Jumps Over The Lazy Dog</div>"
The Quick Brown Fox Jumps Over The Lazy Dog
"The", "Quick", "Brown", "Fox", "Jumps", "Over", "The", "Lazy", "Dog"
["Quick", "Brown", "Fox", "Jumps", "Over", "Lazy", "Dog"]
["quick", "brown", "fox", "jumps", "over", "lazy", "dog"]
Term | Document |
---|---|
quick | 1,2 |
brown | 1 |
fox | 1,2 |
jump | 2,6,7,8,9 |
over | 1,2,4 |
lazy | 1 |
dog | 1,2 |
and filtering
query DSL (domain specific language) allows us to express complex definitions of how to slice & dice data
curl -XGET 'http://kibana.tre.se:9200/_all/_search' -d '{
{
"query": {
"match_all": {}
}
}'
curl -XGET 'http://kibana.tre.se:9200/_all/_search?pretty' -d '{
"facets": {
"0": {
"date_histogram": {
"field": "@timestamp",
"interval": "30m"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "duration:<100"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1457449871001,
"to": "now"
}
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "logname:(\"messageRestLog\")"
}
},
"_cache": true
}
}
],
"must_not": [
{
"fquery": {
"query": {
"query_string": {
"query": "client:(\"SMARTFunctionalTests\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest-legacy\")"
}
},
"_cache": true
}
}
]
}
}
}
}
}
}
},
"1": {
"date_histogram": {
"field": "@timestamp",
"interval": "30m"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "duration:[100 TO 500]"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1457449871002,
"to": "now"
}
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "logname:(\"messageRestLog\")"
}
},
"_cache": true
}
}
],
"must_not": [
{
"fquery": {
"query": {
"query_string": {
"query": "client:(\"SMARTFunctionalTests\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest-legacy\")"
}
},
"_cache": true
}
}
]
}
}
}
}
}
}
},
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30m"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "duration:[500 TO 2000]"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1457449871002,
"to": "now"
}
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "logname:(\"messageRestLog\")"
}
},
"_cache": true
}
}
],
"must_not": [
{
"fquery": {
"query": {
"query_string": {
"query": "client:(\"SMARTFunctionalTests\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest-legacy\")"
}
},
"_cache": true
}
}
]
}
}
}
}
}
}
},
"6": {
"date_histogram": {
"field": "@timestamp",
"interval": "30m"
},
"global": true,
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"query_string": {
"query": "duration:>2000"
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from": 1457449871002,
"to": "now"
}
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "logname:(\"messageRestLog\")"
}
},
"_cache": true
}
}
],
"must_not": [
{
"fquery": {
"query": {
"query_string": {
"query": "client:(\"SMARTFunctionalTests\")"
}
},
"_cache": true
}
},
{
"fquery": {
"query": {
"query_string": {
"query": "application:(\"smart-rest-legacy\")"
}
},
"_cache": true
}
}
]
}
}
}
}
}
}
}
},
"size": 0
}'
Close
All querys are analyzed.
(So we can use the inverted index we created)
Results are 'scored'
{
"query": {
"bool": {
"must": {
"match": {
"body": "Java"
}
},
"filter": {
"term": {
"comment_count": 5
}
}
}
}
}
doc: |1|2|3|4|5|6|7|8|..
match: |1|0|0|1|0|1|0|1|..
Evaluates to what degree each document matches and results in a scored result.
Is computed every time we query.
can not be cached.
Evaluates whether a document matches or not and results in a filter bitset.
Does not need to be computed every time.
can be cached
{
"@version": "1",
"@timestamp": "2016-03-10T14:43:39.762Z",
"host": "x13067pzz.omaccess.net",
"logtimestamp": "2016-03-10 15:43:39,264",
"callId": "0000015354917041-400150",
"user": "piippo59@gmail.com",
"uri": "/smart/api/rest/customers/525056841829790123726121104565/notifications",
"strippedUri": "/smart/api/rest/customers/NUMBER/notifications",
"method": "GET",
"statusCode": 200,
"startTime": "2016-03-10T15:43:39.206+01:00",
"duration": 58,
"client": "3Sverige",
"accept": "application/vnd.com.hi3gaccess.smart-v2+json",
"agent": "",
"isAdmin": "",
"authMethod": "Interactive",
"size": 6296,
"cache": "NONE",
"applicationId": "se.tre.ios.3sverige",
"applicationVersion": "3.6",
"json_responseBody": "{\"notifications\":[{\"my3WebLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"Mitt3/Faktura/#new-invoice\"},\"id\":\"0a877b32-252f-42ce-bd51-0eb48207e5b4\",\"text\":\"Du har en ny faktura med förfallodag 2016-02-26.\",\"status\":\"UNREAD\",\"ownerId\":\"10079876446\",\"shortText\":\"Ny faktura\",\"my3AppLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"my3://Invoice.details/516521775523\"},\"notificationOwner\":\"account\",\"date\":\"2016-02-25\",\"eventTypeId\":\"newInvoice\"},{\"my3WebLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"Mitt3/Faktura/#new-invoice\"},\"id\":\"e6a50bc2-eb7c-4e95-a769-3c57e9761d7d\",\"text\":\"Du har en ny faktura med förfallodag 2016-01-28.\",\"status\":\"UNREAD\",\"ownerId\":\"10079876446\",\"shortText\":\"Ny faktura\",\"my3AppLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"my3://Invoice.details/516424402423\"},\"notificationOwner\":\"account\",\"date\":\"2016-01-19\",\"eventTypeId\":\"newInvoice\"},{\"my3WebLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"Mitt3/Faktura/#new-invoice\"},\"id\":\"46551726-b721-4ac7-aca3-9d69982bc19f\",\"text\":\"Du har en ny faktura med förfallodag 2015-12-28.\",\"status\":\"READ\",\"ownerId\":\"10079876446\",\"shortText\":\"Ny faktura\",\"my3AppLink\":{\"buttonText\":\"Visa faktura\",\"url\":\"my3://Invoice.details/516287234020\"},\"notificationOwner\":\"account\",\"date\":\"2015-12-14\",\"eventTypeId\":\"newInvoice\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=${isid}\"},\"id\":\"5449f210-f8c6-4205-a5aa-d714b157b93f\",\"text\":\"46760171330 har förbrukat de 512 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"READ\",\"ownerId\":\"163954130\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760171330\"},\"notificationOwner\":\"subscription\",\"date\":\"2015-12-17\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=163954113\"},\"id\":\"c9869eff-df61-402f-a1b6-017cfad6fd5a\",\"text\":\"46760185635 har förbrukat de 20480 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163954113\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760185635\"},\"notificationOwner\":\"subscription\",\"date\":\"2016-02-23\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=163954113\"},\"id\":\"bf5d8ada-6606-44c4-99b2-f0f69744eff0\",\"text\":\"46760185635 har förbrukat de 20480 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163954113\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760185635\"},\"notificationOwner\":\"subscription\",\"date\":\"2016-01-21\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=${isid}\"},\"id\":\"f675e05b-f474-4144-be6f-3552e87c317b\",\"text\":\"46760185635 har förbrukat de 20480 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163954113\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760185635\"},\"notificationOwner\":\"subscription\",\"date\":\"2015-12-16\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=163929804\"},\"id\":\"a427cfa2-1c11-4b28-93b5-b6e6656bfedd\",\"text\":\"46760071705 har förbrukat de 512 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163929804\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760071705\"},\"notificationOwner\":\"subscription\",\"date\":\"2016-03-02\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=${isid}\"},\"id\":\"1385dd11-5f83-4b0e-ae00-74b7f5fe7178\",\"text\":\"46760071705 har förbrukat de 512 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163929804\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760071705\"},\"notificationOwner\":\"subscription\",\"date\":\"2016-01-06\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=${isid}\"},\"id\":\"bb65db9c-4ba6-4097-8f39-4ee328403780\",\"text\":\"46760071705 har förbrukat de 512 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"READ\",\"ownerId\":\"163929804\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760071705\"},\"notificationOwner\":\"subscription\",\"date\":\"2015-12-09\",\"eventTypeId\":\"outOfFreeUnits\"},{\"my3WebLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"Mitt3/Abonnemang/Abonnemangshantering?ent=163929802\"},\"id\":\"2f0654ac-66de-4f7e-a5fb-7ce691b001ab\",\"text\":\"46760071704 har förbrukat de 5120 MB data som ingår per månad. Köp extra surf om du vill fortsätta surfa. Den extra surfmängd du köper gäller tills den är förbrukad men som längst till månadsskiftet.\",\"status\":\"UNREAD\",\"ownerId\":\"163929802\",\"shortText\":\"Inkluderad data är slut\",\"my3AppLink\":{\"buttonText\":\"Köp extra surf\",\"url\":\"my3://Subscription.Voucher.National?msisdn=46760071704\"},\"notificationOwner\":\"subscription\",\"date\":\"2016-01-27\",\"eventTypeId\":\"outOfFreeUnits\"}]}",
"resource": "customers",
"subresource1": "525056841829790123726121104565",
"subresource2": "notifications",
"application": "smart-rest",
"logname": "messageRestLog"
}
Option 1: Let elasticsearch figure it out.
Option 2: Specify exactly what each field means
Fancy Smart Thing
-or-
Finite State Transducer
Buckets
Metrics
Pipelines
Each document is evaluated against the created buckets and each bucket keeps track of what documents “fall” in it.
Typically, metrics aggregations generate numeric stats that are computed over a specific document set.
Avg, Cardinality, Extended Stats, Geo Bounds, Geo Centroid, Max, Min, Percentiles, Percentile Ranks, Scripted Metric, Stats, Sum, Top hits, Value Count
Aggregates on the aggregated data
"Send a message when temperature is over x degrees"
"Send a message when number of failures per minute is above x"
like triggers, but different...
"If document contains X, then add field foo and bar"
tags for example...
Java API
JavaScript API
Groovy API
.NET API
PHP API
Perl API
Python API
Ruby API
+Community Contributed Clients