Polls Database

An Intro (or update if you've been here a while)

The stack

• Ruby on Rails App
  Rails 4
   
• Postgres Database
   
• Deployed on Ec2 using Dokku
  polls.538.io
   
• Separate Ec2 for updates
  poll-updates.538.io

The Data Structure

https://polls.538.io/

Caveats

• There are exceptions to the "Pollster" vs "Sponsor" distinction. For example, Fox News polls are actually conducted by Beacon Research (D) + Shaw and Company Research (R).

Limitations​​

• Pollster <--> Poll is one-to-one relationship instead of a many-to-one, so when pollsters collaborate, we create a new pollster for that.
• We don't have the distinction between a pollster and a "survey house" like HuffPost's data structure does

The Data Structure

https://polls.538.io/

• Question
  One poll has many questions
• Question.type
  VotingQuestion, SecondChoiceVotingQuestion, ApprovalQuestion, FavorabilityQuestion
• Answer
  VotingAnswer, AlternateAnswer, FavorabilityAnswer, ApprovalAnswer
• Crosstab (new)
  category: "strength", value: "somewhat", answer_id: 624669, pct: 3.0

Caveats

• Crosstab is only currently used for favorability questions. It can be expanded if we start collecting more crosstabs, but will need an update before we use it for anything else.

Limitations​​

• Question.type versus QuestionType...Approval Questions have subtypes, which are stored in the QuestionType object

The Data Structure

https://polls.538.io/

• Politician
  A person (biographical info imported from VoteSmart in many cases)
• Candidate
  A person, in a particular year, running for a particular office, in
• Office
  Organization that paid for it
• Matchup
  Pollster Rating Object, may be tied to multiple pollster objects
• Race (new)
  For Scrapers

Caveats

• Matchup is mostly a residual table from before we had Race. We keep the table around since it makes some queries a little faster, and so we don't have to modify a bunch of old code...but it may go away.

Recent (-ish) Updates

• Old Data Much Cleaner (Thanks Mary!)
• Data cleaning/data tests
  https://github.com/fivethirtyeight/polls/tree/master/test
• Collecting more question types (like Favorables)
• Polls DB is now the source of truth for PollsterRatings 
• Addition of Race table should make analysis easier

Actions

• Daily Polls Updates
  ​https://github.com/fivethirtyeight/polls#daily-polls-update
  • Ingest
  • Export

Pipeline to Interactives

• Export drops data in:
  https://github.com/fivethirtyeight/polls/blob/master/app/models/export/updater.rb
  • S3 or
  • Directly into your GitHub repo (like polls-page). 

Todo:

• Devops
• Optimization
• Debugifying
• Refactoring old code
• Living documentation
• 100+ Open Tickets
   

Fantasyland

• Store releases and articles, index them, and make them full-text searchable

cure to link-rot, good resource for political writers, becomes a valuable archive

What's your polls db fantasy?