Amanpreet Singh
Team FDI
Homestead
Fields data store
(DynamoDB)
World
Current Scenario:
- Field queries for single user
- DynamoDB doesn't support complex spatial queries (In memory filtering)
API
Background
Homestead
Fields data store
(DynamoDB)
World
Challenges:
- Bulk Queries
- Complex Spatial Queries
- Queries across all users
API
Problems and Challenges
Example Use Cases
- Notify all users within a boundary
- Find number of fields with area < 20 units
- All the fields with particular source (Equipment generated etc.)
- Get unique acres within a boundary
Allows making bulk and very complex spatial queries on Homestead field data
Homestead
Fields data store
(DynamoDB)
World
API
Spatial
Index
(ElasticSearch)
Spatial Index solves all of the challenges using ElasticSearch query api
Challenge
How do we create the index?
Spatial
Index
(Lissom)
Lissom
Feeder
(One time job)
Homestead
Feeder
Challenges:
- Slow, takes a lot of time to fetch all the fields.
- Error prone, homestead doesn't always return results.
- Testing is hard in elasticsearch.
- Configurability
- One time process
- Feeds elasticsearch by fetching all the fields from homestead
Feeder
Solutions:
- Slow: Use threads -> Claypoole.
- Errors: Log them properly, feed back to feeder (it handles them)
- Testing: Created whole new test utilities set. Makes it lot more easier.
- Config: Takes all possible command line arguments.
- One time process
- Feeds elasticsearch by fetching all the fields from homestead
Challenge
How do we keep the index updated?
Spatial
Index
(Lissom)
Lissom
Feeder
(One time job)
Lissom
Updator
(Keep it updated)
Agni
Homestead
Updator
- Runs as service (in production currently)
- Polls Agni to keep index updated.
Challenges:
- Too many events coming in
- Deletes were not published
- Testing was tough
- How to give alerts?
Updator
Solutions:
Too many: Use core.async, one publisher, many subscribers
Deletes: Publish delete optimistically from Homestead
Testing: Use test utils, integration tests
Alerts: Throws PagerDuty alerts if state not changed in last 15 min.
Challenge
How do we expose the index?
Spatial
Index
(Lissom)
Lissom
Feeder
(One time job)
Lissom
Updator
(Keep it updated)
Lissom
Lib
(Expose it)
Homestead
Internal World
Agni
Homestead
Lissom Lib
- Client to lissom
- Used in homestead to expose spatial index.
- Has support for basic homestead routes and complex spatial queries.
Spatial
Index
(Lissom)
Lissom
Feeder
(One time job)
Lissom
Updator
(Keep it updated)
Lissom
Lib
(Expose it)
Homestead
Homestead Event Pubisher
Internal World
Agni
Homestead
Homestead Route
/spatial/query
Takes in following params:
- pagesize: Number of results to be returned (Cap at 1000)
- offset: Number of results to be skipped before giving results
- filter: Type of filtering to be done on spatial query
- boundary: GeoJSON in which we want fields.
Homestead Route
/spatial/query
Returns:
- Fields results as array
- total number of results
Demo
Metrics
Tech
Clojure
ElasticSearch
Thanks
Got other questions?
Reach out at fdi@climate.com