Instalacja ElasticSearch'a

4. Warto doinstalować Sense (Beta) Chrome plugin

Instalacja ElasticSearch'a

Podstawowe pojęcia

Operacje CRUD

Create

Operacje CRUD

Read

Operacje CRUD

Update

Operacje CRUD

Delete

Wyszukiwanie

URI Search

GET /esmd/movie/_search?q=title:matrix

Term Query

POST /esmd/movie/_search
{
    "query" : {
        "term" : { "title" : "matrix" }
    }
}

Term Query, full text search

POST /esmd/movie/_search
{
    "query" : {
        "term" : { "_all" : "matrix" }
    }
}

Wyszukiwanie

Match query

POST /esmd/movie/_search
{
   "query": {
      "match": {
         "title": {
            "query": "matrix revolution",
            "operator": "or"
         }
      }
   }
}

Wyszukiwanie

Multi Match Query

POST /esmd/movie/_search
{
    "query": {
        "multi_match" : {
            "query":    "2012", 
            "fields": [ "title", "year" ] 
        }
    } 
}

Multi Match Query

POST /esmd/movie/_search
{
    "query": {
        "multi_match" : {
            "query":    "2012", 
            "fields" : [ "title^3", "*id" ]
        }
    } 
}

Wyszukiwanie

Bool query

POST /esmd/movie/_search
{
   "query": {
      "bool": {
         "must": {
            "match": {
               "title": {
                  "query": "american pie",
                  "operator": "and"
               }
            }
         },
         "should": [
            {
                "term": {
                   "year": {
                      "value": "2010"
                   }
                }
            }
         ]
      }
   }
}

Wyszukiwanie

Fuzzy query

POST /esmd/movie/_search
{
    "query" : {
        "fuzzy" : { "title" : "matrixx" }
    }
}

Domyślne odległości dla tekstów

• 0..1 - musi pasować dokładnie
• 1..5 - możliwa jedna edycja
• >5 - możliwe dwie edycje

Wyszukiwanie

Query string query

POST /esmd/movie/_search
{
   "query": {
      "query_string": {
         "default_field": "title",
         "query": "this AND that OR thus"
      }
   }
}

Query string query

POST /esmd/movie/_search
{
   "query": {
      "query_string": {
         "query": "(title:this OR plot:this) AND (title:that OR plot:that)"
      }
   }
}

Wyszukiwanie

Reqexp query

POST /esmd/movie/_search
{
   "query": {
      "regexp": {
         "title": "a.*q"
      }
   }
}

Wyszukiwanie

Poświetlanie znalezionego fragmentu

POST /esmd/movie/_search
{
   "query": {
      "term": {
         "plot": "matrix"
      }
   },
   "highlight": {
      "fields": {
         "title": {
            "type": "plain",
            "force_source" : true
         },
         "plot": {
            "type": "plain",
            "force_source" : true
         }
      }
   }
}

Wyszukiwanie - wszystkie query

Match Query
Multi Match Query
Bool Query
Boosting Query
Common Terms Query
Constant Score Query
Dis Max Query
Filtered Query
Fuzzy Like This Query
Fuzzy Like This Field Query
Function Score Query
Fuzzy Query
GeoShape Query
Has Child Query
Has Parent Query
Ids Query
Indices Query
Match All Query
More Like This Query

Nested Query

Prefix Query
Query String Query
Simple Query String Query
Range Query
Regexp Query
Span First Query
Span Multi Term Query
Span Near Query
Span Not Query
Span Or Query
Span Term Query
Term Query
Terms Query
Top Children Query
Wildcard Query
Minimum Should Match
Multi Term Query Rewrite
Template Query

Filtrowanie

• szybsze niż zwykłe zapytania (o rząd wielkości)
• nie ma wyliczania punktacji
• większość z nich jest automatycznie cache'owa

• do wyszukiwania gdzie odpowiedzą jest tak/nie
• do zapytań o dokładną wielkość

Kiedy używać?

Filtrowanie

Term Filter

POST /esmd/movie/_search
{
    "filter": {
        "term": {
           "year": "2017"
        }
    }
}

Filtrowanie

Range filter

POST /esmd/movie/_search
{
    "filter": {
        "range": {
           "year": {
              "from": 1889,
              "to": 1890
           }
        }
    }
}

Filtrowanie

Exist + not + and filter

POST /esmd/movie/_search
{
    "filter": {
        "and": {
           "filters": [
              {
                  "not": {
                       "exists": {
                           "field": "rating"
                        }
                    }
              },
              {
                    "not": {
                       "exists": {
                           "field": "directors"
                       }
                    }
              }
           ]
        } 
    }
}

Wszystkie filtry

And Filter
Bool Filter
Exists Filter
Geo Bounding Box Filter
Geo Distance Filter
Geo Distance Range Filter
Geo Polygon Filter
GeoShape Filter
Geohash Cell Filter
Has Child Filter
Has Parent Filter
Ids Filter
Indices Filter
Limit Filter

Match All Filter
Missing Filter
Nested Filter
Not Filter
Or Filter
Prefix Filter
Query Filter
Range Filter
Regexp Filter
Script Filter
Term Filter
Terms Filter
Type Filter

Jak to wszystko działa?

Apache Lucene

• Standard dla bibliotek wyszukiwania
• Open Source
• Odwrócony index (inverted index)

Odwrócony index

1. The quick brown fox jumped over the lazy dog

2. Quick brown foxes leap over lazy dogs in summer

Analiza tekstu

Analiza tekstu

• Angielski

Analiza tekstu

Wsparcie dla języków:

Arabic, Armenian, Basque, Brazilian, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English, Finnish, French, Galician, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Kurdish, Norwegian, Persian, Portuguese, Romanian, Russian, Spanish, Swedish, Turkish, and Thai.

Wsparcie do polskiego za pomocą plugina:

Stempel Polish Analysis plugin

Odwrócony index

Obliczanie punktacji

Obliczanie punktacji

Podgląd punktacji

POST /esmd/movie/_search?explain
{
  "query": {
    "term": { "title": "matrix" }
  }
}

  "_explanation": {
    "value": 4.9666376,
    "description": "weight(title:matrix in 21100) [PerFieldSimilarity], result of:",
    "details": [
      {
        "value": 4.9666376,
        "description": "fieldWeight in 21100, product of:",
        "details": [
          {
            "value": 1,
            "description": "tf(freq=1.0), with freq of:",
            "details": [
              {
                "value": 1,
                "description": "termFreq=1.0"
              }
            ]
          },
          {
            "value": 11.352315,
            "description": "idf(docFreq=1, maxDocs=62659)"
          },
          {
            "value": 0.4375,
            "description": "fieldNorm(doc=21100)"
          }
        ]
      }
    ]
  }

Trochę o architekturze

Klaster (Cluster) - identyfikowany przez nazwę

• Węzły (Node) - osobny server

• Skorupy ? (Shard) - indeks Lucynki

• Data - przechowują dane
• Client - load balancer
• Master - zarządzanie klastrem

• primary
• replica

Architektua ElasticSearch'a

Architektua ElasticSearch'a

Zapis

src: https://www.elastic.co/guide/en/elasticsearch/guide/current/distrib-write.html

Architektua ElasticSearch'a

Wyszukiwanie

Query Phase

Agregacje

2 typy agregacji:

• metryki - liczenie czegoś na zbiorze dokumentów
• wiaderkowanie? ("Bucketing") - wrzucanie do różnych worków

• Lepsze "facet's" (strony, aspekty?)
• Mogą być zagnieżdżane

Agregacje

Sum Aggregation

POST /esmd/movie/_search
{
    "size": 0,
    "aggregations" : {
        "ammount of votes" : {
            "sum" : {
                "field": "rating.votes"
            }
        }
    }
}

Agregacje

Stats Aggregation

POST /esmd/movie/_search
{
   "size": 0,
   "aggregations": {
      "stats of votes": {
         "stats": {
            "field": "rating.votes"
         }
      }
   }
}

Agregacje

Terms Aggregation

POST /esmd/movie/_search
{
   "size": 0,
   "aggregations": {
      "possible years of movies": {
         "terms": {
            "field": "year"
         }
      }
   }
}

Agregacje

Zagnieżdżone aggregacje

POST /esmd/movie/_search
{
   "size": 0,
   "aggregations": {
      "possible years of movies": {
         "terms": {
            "field": "year"
         },
         "aggregations": {
            "rank": {
               "terms": {
                  "field": "rating.rank"
               }
            }
         }
      }
   }
}

Agregacje mogą kłamać

Agregacje mogą kłamać

Jak sobie z tym radzić?

Błędy:

• "doc_count_error_upper_bound": 46 (2 + 15 + 29),
• "sum_other_doc_count": 81 (suma pod kreską)

Zwiększenie parametru "size"

• wydajność

Agregacje i analiza

Czasem agregacje dają niespodziewane wyniki

"buckets": [
   {
      "key": "american",
      "doc_count": 1
   },
   {
      "key": "fiction",
      "doc_count": 1
   },
   {
      "key": "pie",
      "doc_count": 1
   },
   {
      "key": "pulp",
      "doc_count": 1
   }
]

POST /moviesindex/movies/_search
{
    "size": 0, 
    "aggregations" : {
        "by titles": {
            "terms": {
                "field": "title"
            }
        }
    }
}

Agregacje i analiza

Konieczne jest wtedy zastosowanie mappingu

PUT /moviesindex
{
  "mappings": {
    "movies": {
      "properties": {
        "title" : {
          "type": "string",
          "fields": {
            "raw" : {
              "type": "string",
              "index": "not_analyzed"
            }
          }
        }
      }
    }
  }
}

Wszystkie agregacje

Min Aggregation
Max Aggregation
Sum Aggregation
Avg Aggregation
Stats Aggregation
Extended Stats Aggregation
Value Count Aggregation
Percentiles Aggregation
Percentile Ranks Aggregation
Cardinality Aggregation
Geo Bounds Aggregation
Top hits Aggregation
Scripted Metric Aggregation
Global Aggregation
Filter Aggregation

Filters Aggregation
Missing Aggregation
Nested Aggregation
Reverse nested Aggregation
Children Aggregation
Terms Aggregation
Significant Terms Aggregation
Range Aggregation
Date Range Aggregation
IPv4 Range Aggregation
Histogram Aggregation
Date Histogram Aggregation
Geo Distance Aggregation
GeoHash grid Aggregation

Sugestie

Sugestie

Term suggester

POST /esmd/_suggest
{
   "sugest in title": {
      "text": "matri",
      "term": {
         "field": "title"
      }
   }
}

Ekosystem

Jak żyć z ElasticSearchem?

Monitoring Marvel

Monitoring Marvel

Monitoring Kopf

Monitoring Elastic HQ

Monitoring Bigdesk

Kibana

Analytics and search dashboard for Elasticsearch

Logstash

Transport and process your logs, events, or other data

Found

Elasticsearch as a Service