Elastic Search

Andrew Johnstone

www.ajohnstone.com

About Elastic Search

Distributed
Highly-availble
RESTful search engine (on top of Lucene)
Document-oriented
JSON-based
schema-free

Restful


Example format

curl -s -XGET 'http://localhost:9200/index1,index2/typeA,type/_search' -d '{
  "query": { "match_all": {} }
}

Mapping

curl -s -XGET 'http://localhost:9200/_mapping?pretty=true'

Modules

  • Discovery 
  • Gateway
  • Transport
  • Network
  • Indicies
  • Cluster
  • Scripting
  • Thread Pool
  • Node
  • Plugins
  • JMX
  • Memcached
  • Thrift

SCALABILITY

  • Elastic search delegates requests to appropriate nodes
  • Automatic discovery of nodes using multicast/unicast.
  • Master election - can allocate multiple masters, clients.
  • Fault detection - master pings nodes, clients ping master.  Identifies when an election process needs to be initiated.
  •  Support for EC2

CLUSTER

Shards allocation is the process of allocating shards to nodes. This can happen during initial recovery, replica allocation, rebalancing, or handling nodes being added or removed.
GET /_cluster/health
GET /_cluster/health/index1,index2
GET /_cluster/nodes/stats
GET /_cluster/nodes/nodeId1,nodeId2/stats
POST /_cluster/nodes/nodeId1,nodeId2/_shutdown
POST /_cluster/reroute # Re-route shards and nodes

Shards

  • A portion of the document space
  • Each one is a separate Lucene index
  • Document is sharded by its _id
    • can be routed to a shard.

PUT /member {
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 2,
}}

SHARDS - Allocations


curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.include.tag" : "value1,value2"
}'

curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.include.group1" : "xxx"
    "index.routing.allocation.include.group2" : "yyy",
    "index.routing.allocation.exclude.group3" : "zzz",
}'
curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
    }
}'

Write consistency

When making bulk calls, you can require a minimum number of active shards in the partition through the consistency parameter. 

For example, in a N shards with 2 replicas index, there will have to be at least 2 active shards within the relevant partition (quorum) for the operation to succeed.

This can be specified node by node.
Valid write consistency values are one, quorum, and all.
Options also exist for Asynchronous Replication, although not enabled by default. 

Routing

  • Master node
    • Maintains cluster state
    • Reassigns shards if nodes leave/join cluster
  • any node can serve as a request router
  • the query is handled via a scatter-gather mechanism

Performance

Call _optimize daily, potentially decrease index size.
disable _all field.

Modify:
ES_MIN_MEM=3000
MES_MAX_MEM=9600
MNFILES=65536

Change search types try search_type=query_then_fetch


Performance

Disable "_all"
curl -XPUT 'http://localhost:9200/_template/template_name/' -d '
{
    "template": "match-*",
    "mappings": {
        "_default_": { 
             "_source": { "compress": "true" },
             "_all" : {"enabled" : false}
        }
    }
}'
Optimize Old Indices
curl -XPOST 'http://localhost:9200/organisations/_optimize?max_num_segments=2'

Use max_num_segments with a value of 2 or 3.

(Setting max_num_segements  IO intensive)

Warming Indexes

Can be created within an index, type or as a template.

curl -XPUT localhost:9200/test/_warmer/warmer_1 -d 
{
   "query":{
      "match_all":{

      }
   },
   "facets":{
      "facet_1":{
         "terms":{
            "field":"field"
         }
      }
   }
}

Warmers

# get warmer named warmer_1 on test index
curl -XGET localhost:9200/test/_warmer/warmer_1 

# get all warmers that start with warm on test index
curl -XGET localhost:9200/test/_warmer/warm* 

# get all warmers for test index
curl -XGET localhost:9200/test/_warmer/

Mapping

Mapping is the process of defining how a document should be mapped to the Search Engine

Mapping types are a way to try and divide the documents indexed into the same index into logical groups.

Templates

curl -XPUT localhost:9200/_template/template_1 -d 
{
   "template":"template*",
   "settings":{
      "number_of_shards":1
   },
   "mappings":{
      "type1":{
         "_source":{
            "enabled":false
         }
      }
   }
}

Analyzers & Tokenizers 



Queries vs Analyzers



Use filters for anything that does not affect the relevance score. Queries Filters Full text & terms Terms only Relevance scoring No scoring Slower Faster No caching Cacheable

Filters are very handy since they perform an order of magnitude better than plain queries since no scoring is performed and they are automatically cached.

Filters can be a great candidate for caching. Caching the result of a filter does not require a lot of memory, and will cause other queries executing against the same filter (same parameters) to be blazingly fast.


Analyzers & Tokenizers

Typically the default analyzers will not match what your expecting. Thus use the analyser.

GET /_analyze?analyzer=standard -d 'testing'
GET /_analyze?tokenizer=snowball&filters=lowercase -d'testing'
GET /_analyze?text=testing
GET /_analyze?field=obj1.field1 -d'testing'

Example


analysis:

 filter:

   ngram_filter:

     type: "nGram"

     min_gram: 3

     max_gram: 8

 analyzer:

   ngram_analyzer:

     tokenizer: "whitespace"

     filter: ["ngram_filter"]

     type: "custom"

ngrams - an n-gram is a contiguous sequence of n items from a given sequence of text 
unigram (1), bigram (2), trigram(3)

Facets

Facets provide aggregated data based on a search query. Allowing the user to refine their query based on the insight from the facet, i.e. restrict the search to a specific category, price or date range. 


Facets supported by Elastic Search


Terms, Range, Histogram, Date Histogram, Filter, Query, Statistical, Terms Stats and Geo Distance

Example

Data import

Elastic search is typically not the primary data store.

Implement a queue or use rivers.

A river is a pluggable service running within elasticsearch cluster pulling data (or being pushed with data) that is then indexed into the cluster.
Currently available rivers include CouchDB, MongoDB, RabbitMQ, Amazon SQS, ActiveMQ etc.

Typically I implement ØMQ as a layer within the application to push data/entities to Elastic search. 

Php Example

Presently spent a two day spike implementing.
As such somewhat incomplete.



Layout

bin/
| bootstrap.php
| search
    | build_index.php
    | index_all.php
application/libs/
| Application
|   | Search
|   |   | Criterion
|   |   |   | Members
|   |   |   |   | Country.php
|   |   |   | README.md
|   |   | Data
|   |   |   | Producer
|   |   |       | Example.php
|   |   |       | README.md
|   |   | Result.php
|   |   | Structure.php
|   | Service

Layout

| Elastica -> /usr/share/php/Elastica
| Photobox
    | Search
    |   | Criterion
    |   |   | Boolean.php
    |   |   | Integer.php
    |   |   | Intersect.php
    |   |   | Keyword.php
    |   |   | String.php
    |   |   | Type.php
    |   |   | Union.php
    |   | Criterion.php
    |   | Data
    |   |   | Producer.php
    |   |   | Transfer.php
    |   | Engine
    |   |   | Elasticsearch
    |   |   |   | StructureAbstract.php
    |   |   | Elasticsearch.php
    |   | Engine.php
    |   | Index
    |   |   | Builder.php
    |   | Result.php
    | Service
        | Search
        | Search.php

Elastica

PHP client for the distributed search engine elasticsearch which is based on Lucene.

https://github.com/ruflin/Elastica

Backoffice Elastic Search

Presently implements criterions for

  • Boolean
  • Integer
  • Intersects
  • Keyword
  • String
  • Type (index)
  • Union

It builds indexes for you against Elastic Search

Application Architecture

Is split between generic implementation with interfaces.
Application specific logic held within lib/Application/Search.

This contains the structure for the index. The implementation can largely be re-factored to use any search engine.


Debugging

In order to debug it is commonly easier to extract the raw query and use the RESTful interface directly. Secondly diagnose whether the criterion's match the terms provided.

curl -s -XGET 'http://localhost:9200/_mapping?pretty=true'
GET /_analyze?field=obj1.field1 -d'testing'

Examples - Gorkana

Analyzers are important to customise for the context of the search. For example a 'quick search' and searching for unusually unique names, such as I from the independent.

Examples - Gorkana

Example searching multiple indexes. Best to create your own quick searches typically best match. (Tokenizer: standard - filters: standard,lowercase, asciifolding, filter_stem_possessive_english, filter_edge_ngram_front). Stop words play an important part for example "The" with "The times"

Example - Gorkana

Multiple matching intersections/unions on hierarchical data sets. Matching permissions, outlets and their departments and their journalists within each. Selections of items maintained throughout pagination (select all M/O and departments).

Example -Gorkana

Multiple facets with Terms and geo-distance search.

PARR - Percolators

 Sends queries, register them, and then sends docs and finds out which queries match that doc. Used for emailing alerts against a matching query.

PARR - DATA Access

All index pages use elastic search, no pages use the database, unless matching specific IDs for supplementary data (even admin pages).

PARR - SEARCH CRITERIONS

A subset of criterions against a few types.

Parr - Data

Data is pushed into the application, the application uses Doctrine 2 and data is mapped to the structure of the referenced doctrine entities.


ØMQ is used to transfer data to Elastic Search, thus any doctrine entity can be pushed into an Elastic Search index.

test

By ajohnstone

Made with Slides.com

test

  • 10,950

More from ajohnstone