Geoposition with Elasticsearch


Philly ES Meetup - Nov 2014

Michael Holroyd, Ph.D.



What is this stuff?

node.js

 server-side event-driven javascript

postgres

 classic row-based SQL server 

elasticsearch

 full-text search using Apache Lucene

 ... supports geospatial queries!

Data

 scraped from instagram.com


Official API endpoint for geolocation based search:
https://api.instagram.com/v1/media/search?lat=48.858844&lng=2.294351

Response:


Go look at downloadInstagram.js and centroids();

Kibana



Maps

leaflet.js

 open-source JS library for interactive maps


  • Handles base-layers
  • Support for tiled raster layers (Anything > 100k elements)
  • Plays nice with d3.js for interactive SVG elements


Maps

Originally we used SVG elements for every dot on the map. Looked awesome, but ran the browser into the ground.

Maps

Instead, rasterize all the dots and use a tile server to load the correct data at run-time.


 psql -c "SELECT latitude, longitude FROM instagram" -F , -t -o instagram.csv

Check out https://github.com/ericfischer/datamaps for multi-core rendering solution. Can still be really slow, future work is to parallelize across machines.


Go look at demo. NO PEEKING AT COOL ELASTICSEARCH FEATURES YET.

Elasticsearch

Yank everything out of postgres and stuff it into elasticsearch
It's important to setup your mapping ahead of time to accommodate geo-position queries:
properties: {
  geo: {
    type: "geo_point",
    lat_lon: true,
    fielddata: {
      format: "compressed",
      precision: "1cm"
    }
  }
}
look at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html

Query

You MUST query in the same geo-format as your data (bug?)


{
  size: 10000,
  "query" : {
    "match_all" : {}
  },
  "filter" : {
    "geo_distance" : {
      "distance" : "100m",
      "distance_type": "plane",
      "geo" : [e.latlng.lng,e.latlng.lat]
    }
  }
}
see  http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-filter.html#query-dsl-geo-distance-filter

What's going on?

Multiple strategies available in Lucene:
BoundingBox, Quadtrees, SpatialPrefixTree

For points, elasticsearch just uses a simple boundingbox strategy. Lots of room for improvement here actually.

For shapes, elasticsearch uses either a geohash strategy (if your data fits) or quadtrees.

Geospatial or Fulltext search


Check out search.js and show Demo

Thanks!


Arqball (computer vision kung-fu):
http://arqspin.com

Knollop (online education search):
http://knollop.com

Learnstream (link curation and sharing):
http://learnstream.com

Michael Holroyd
http://meekohi.com

Geoposition with Elasticsearch

By Michael Holroyd

Made with Slides.com

Geoposition with Elasticsearch

Indexing and searching ~4 million Instagram photos using Elasticsearch.

  • 3,614

More from Michael Holroyd