Exploring hospital violations with Elasticsearch

Purpose

Product

Ask yourself:

Where is the community that needs this tool?

What function or need does it address?

What is the discussion in the community?

Note: I've included addendum material in all basement slides. Check it out!

Ask yourself:

Does this already exist?

If so, how does it address a need?

What is the discussion in the community?

Code Newbie Podcast:

Ep. 9 - How To Build A Product

http://www.codenewbie.org/podcast/how-to-build-a-product

There's a lot of CMS data, let's see if we can navigate it using elasticsearch

Goal: Create a product to more easily navigate text data

Sub-goal: Learn elasticsearch, have fun!

The data, the framework, the engine

Data: Center for Medicaid and Medicare Services (publicly available!)
Framework: Rails
- Gems used: Elasticsearch, Chewy
- Gems to use: Kibana-rack
Library: Elasticsearch

Check out these Gems at RubyGems.org

The difficulties

Data:
- CSV to JSON: What about a lot of CSV?
  - Discussion point: Is public data that is extremely difficult to use truly public?
- Seeding the database!
Elasticsearch:
- Use ELK stack, the console, something else?
Constraints:
- Tip, try not to complete your slides in a night.
  - Or the benefits of starting early and getting a draft out.

Publicly Available Data?

Build it,

break it.

Build it again.

Elastic Rails

#Elasticsearch gem
gem 'elasticsearch', '~> 1.0.6'
gem 'elasticsearch-model'
gem 'elasticsearch-rails'
#Chewy for log handling
gem 'chewy', '~> 0.6.0'

def search
  if params[:query].nil?
    @hospitals = []
  else
    @hospitals = Hospital.search params[:query]
  end
end

require 'elasticsearch/model'
 
class Hospital < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end
Hospital.import # for auto sync model with elastic search

All-in-all Ruby had the better free online tutorials.

Discussion point: Will those with more clear, free, tutorials win out?

Gemfile

Search Controller

Hospital Model

Ruby Conf 2013: Elasticsearch Demo

https://www.youtube.com/watch?v=H81IiLUJavI

New Relic Walkthrough

http://www.sitepoint.com/full-text-search-rails-elasticsearch/

It's built, time to break.

Does this achieve the purpose we laid out for it at the beginning?

What if I add this additional functionality...

	
def self.search(query)
  __elasticsearch__.search(
    {
      query: {
        multi_match: {
          query: name,
          fields: ['name^10', 'name']
        }
      },
      highlight: {
        pre_tags: ['<em>'],
        post_tags: ['</em>'],
        fields: {
          name: {},
          hospital: {}
        }
      }
    }
  )
end

Build it again.

Doesn't work, let's go back

Next Steps

Mountains beyond mountains:
- Of data, and it's scattered and disparate
Expanding functionality:
- Making the search smarter, highlighting queries, etc.
Datapalooza:
- Let's start a team.
  - Last year's info: http://healthdatapalooza.org/get-involved/code-a-palooza-at-health-datapalooza-2014/

Will Sankey

wsankey@gmail.com; @wsankey; willsankey.com

TrackMaven Meetup Deck 11/12/2014

By William Sankey

TrackMaven Meetup Deck 11/12/2014

1,834