B.Y.O.D.
Building your own database
Steven Rich
The Washington Post
#NICAR19
bit.ly/byodatabase
I would prefer not to have to give this talk
But not everything is already in a database or spreadsheet
Why should I build my own database?
There is nothing in this world as satisfying as building your own homemade, artisinal database
Also, you have a data set that is unique and newsworthy
Why shouldn't I build my own database?
-
The database already exists in a database format
-
Someone else has already done it
-
The work involved is insane and will kill you
WARNINGS
-
There is no one-size-fits-all solution
-
The work is often tedious and exacting
-
You need to have a full game plan when you start
Why do we need this session at all?
Examples
Ways to build a database from scratch
- Web scraping
- Public records
- News stories
- By hand
The most important step in this whole process is determining what fields will be kept
Utility vs. Feasibility
Early vs. Late Grouping
Standardization
Scope vs. Time
Thinking about the story
There are a bunch of ways to do this from a technical standpoint
-
Microsoft Excel
-
Google Sheets
-
Google Forms
-
SQL databases
-
Django
How do we build it?
A team approach
Create fail safes
Creating clear standards for data entry
Spot-checking
A programatic approach
A programatic approach
Create fail safes
Creating clear standards for data entry
Spot-checking
Ideal, if possible
May not be necessary
Basic strategies to get into a good place
OCR
-density 300 input.pdf -depth 8 -background white -alpha Off output.tiff
output.tiff input_ocr -l eng pdf
Tabula
tabula.technology
Scripting
Pitfalls
Huge investment
Could be competitive
It's a gamble
It's fragile
Do you continue to keep it?
