Polls Database

An Intro (or update if you've been here a while)

The stack

  • Ruby on Rails App
    Rails 4
     
  • Postgres Database
     
  • Deployed on Ec2 using Dokku
    polls.538.io
     
  • Separate Ec2 for updates
    poll-updates.538.io

The Data Structure

  • Poll
    Poll
  • Pollster
    Organization that did the polling
  • Sponsor
    Organization that paid for it
  • Pollster Rating
    Pollster Rating Object, may be tied to multiple pollster objects
  • Pollster Name
    For Scrapers

Caveats

  • There are exceptions to the "Pollster" vs "Sponsor" distinction. For example, Fox News polls are actually conducted by Beacon Research (D) + Shaw and Company Research (R).

Limitations​​

  • Pollster <--> Poll is one-to-one relationship instead of a many-to-one, so when pollsters collaborate, we create a new pollster for that.
  • We don't have the distinction between a pollster and a "survey house" like HuffPost's data structure does

The Data Structure

  • Question
    One poll has many questions
  • Question.type
    VotingQuestion, SecondChoiceVotingQuestion, ApprovalQuestion, FavorabilityQuestion
  • Answer
    VotingAnswer, AlternateAnswer, FavorabilityAnswer, ApprovalAnswer
  • Crosstab (new)
    category: "strength", value: "somewhat", answer_id: 624669, pct: 3.0

Caveats

  • Crosstab is only currently used for favorability questions. It can be expanded if we start collecting more crosstabs, but will need an update before we use it for anything else.

 

Limitations​​

  • Question.type versus QuestionType...Approval Questions have subtypes, which are stored in the QuestionType object

The Data Structure

  • Politician
    A person (biographical info imported from VoteSmart in many cases)
  • Candidate
    A person, in a particular year, running for a particular office, in
  • Office
    Organization that paid for it
  • Matchup
    Pollster Rating Object, may be tied to multiple pollster objects
  • Race (new)
    For Scrapers

Caveats

  • Matchup is mostly a residual table from before we had Race. We keep the table around since it makes some queries a little faster, and so we don't have to modify a bunch of old code...but it may go away.

 

Recent (-ish) Updates

Actions

Pipeline to Interactives

Todo:

  • Devops
  • Optimization
  • Debugifying
  • Refactoring old code
  • Living documentation
  • 100+ Open Tickets
     

Fantasyland

  • Store releases and articles, index them, and make them full-text searchable

cure to link-rot, good resource for political writers, becomes a valuable archive

 

What's your polls db fantasy?

Polls Database

By Dhrumil Mehta

Polls Database

  • 268