Spatial

Index

(Lissom)

Lissom

Feeder

(One time job)

Homestead

Feeder

Challenges:

- Slow, takes a lot of time to fetch all the fields.

- Error prone, homestead doesn't always return results.

- Testing is hard in elasticsearch.

- Configurability

- One time process

- Feeds elasticsearch by fetching all the fields from homestead

Feeder

Solutions:

- Slow: Use threads -> Claypoole. 

- Errors: Log them properly, feed back to feeder (it handles them)

- Testing: Created whole new test utilities set. Makes it lot more easier.

- Config: Takes all possible command line arguments.

- One time process

- Feeds elasticsearch by fetching all the fields from homestead

Spatial

Index

(Lissom)

Lissom

Feeder

(One time job)

Lissom

Updator

(Keep it updated)

Agni

Homestead

Updator

- Runs as service (in production currently)

- Polls Agni to keep index updated. 

Challenges:

- Too many events coming in

- Deletes were not published

- Testing was tough

- How to give alerts?

Updator

Solutions:

Too many: Use core.async, one publisher, many subscribers

Deletes: Publish delete optimistically from Homestead

Testing: Use test utils, integration tests

Alerts: Throws PagerDuty alerts if state not changed in last 15 min.

Spatial

Index

(Lissom)

Lissom

Feeder

(One time job)

Lissom

Updator

(Keep it updated)

Lissom

Lib

(Expose it)

Homestead

Internal World

Agni

Homestead

Lissom Lib

- Client to lissom

- Used in homestead to expose spatial index.

- Has support for basic homestead routes and complex spatial queries.

Homestead Route

/spatial/query

Takes in following params:

- pagesize: Number of results to be returned (Cap at 1000)

- offset: Number of results to be skipped before giving results

- filter: Type of filtering to be done on spatial query

- boundary: GeoJSON in which we want fields.

Homestead Route

/spatial/query

Returns:

- Fields results as array

- total number of results