Proving an investigative hypothesis

(Or...compared to what?)

Sarah Cohen

The New York Times

sarah.cohen@nytimes.com

Risk assessment

The tip

We hear....

ranchers got drought assistance just because they lived underneath the path of the doomed space shuttle Columbia.

Approaches

Ask the agency / FOIA records directly #fail
Learn how it might happen

What was the fund called?
How did it work?
How would we know it if we saw it? (are there records that would point to an M.O.?)

Broaden to quantify
Minimum / maximum and difficulty assessment

The easy way

Ask the agency
FOIA the records for any space shuttle recipients

Risks

The agency doesn't know
The agency knows you're looking
The agency will pre-empt your story
The agency will give it to someone else

Result of the easy way

The agency has no clue who or how many people got the assistance who "shouldn't" have.

2nd level reporting

Assume it was legal
Assume it was intended
Assume it had unintended consequences

Lesson

Follow the legislative and regulatory record

What we had

The short version

The Livestock Compensation Program had two phases. In the first, only ranchers in counties that had been declared an "agricultural disaster area" got the benefit. In the second, any "disaster area" got money.

Lesson:

What does "agricultural" mean?

Finding the counties

Agricultural disaster areas

There are two disaster programs -- who knew?

Presidential disasters

Compare two lists

Minimum story: counties (and recipients) who got disaster declarations but never had an agricultural one.
What determines an agricultural disaster? (3 farmers hurt by weather.)
Best examples came this way
Effort: Three days of data cleaning; one FOIA and some scraping.

Result of minimum story

That's not enough

What if?

We added up all of the money that didn't go to drought-stricken counties?
What does "drought" mean?
How?

Put together two years of drought maps

Find the money

Result

Why?

Records used

Database of 200 million payments to farmers and ranchers (USDA farm subsidy database)
Agricultural and presidential disaster declarations
Congressional testimony, rules and regulations concerning the Livestock Compensation Program
Drought maps from the weather service
FOIA of congressional correspondence to USDA, much of which was requesting LCP for members' districts

How long? About 2 months.

How to find your "compared to what?"

Find a standard: What is supposed to happen?

Law
What lawmakers say
What leaders claim
What ordinary people are led to believe

Does your example reflect a larger problem?

Is your example a special subgroup that's worse?
Does your example reflect changes over time?

Is your area special?

Bronx courts

Investigative hypothesis

Excessive delays in closing cases lets criminals go free and jails the innocent.

Outside the courtroom, the children of the murdered man waited with their mother. It had taken five years for the Bronx courts to get around to them and to the man with teardrop tattoos charged with killing their father.

The death of Robert Gaston on a bloody bodega floor was one of those murders New York barely notices. The family’s grief had given way to an agonizing wait for what they called their day in court. Two years. Four. Five, as bloodstains and memories faded.

“It should never take five years,” 15-year-old Kaitlynn Gaston said. “All the good parts of New York, the high-class parts of New York, they easily get justice.”

Records

New York State criminal court records

Had versions from 2007, 2008, 2010, 2011 and 2012
Each version had eliminated records for anyone whose case had been dismissed or acquitted.
No records for anyone who was dismissed or acquitted within a year.
Included all charges, hearings and events; all convictions.

Some facts to consider

In New York City, 40% are deemed innocent or their cases are dropped.
Charges are reduced for many of the remaining cases in return for guilty plea
Only 3 percent go to trial

What would you do?

Our attempts

How old were the cases that disappeared between 2010 and 2011? #fail
Do charges get reduced more for cases that have lingered than for cases that get resolved quickly? #fail

Finally

Lessons

Key failures in available records changes what you can prove.
Failing to prove something with flawed records doesn't mean it's not true -- you just wouldn't know it if it exists.
Don't spend too much time when you realize it's not working.

In the end, the court system ran the statistics we needed for the first stories; the lead reporter used the database for the second, after reporting a new angle on the story.

Internet gun sales

The question

How can we learn something new about guns -- their sales, uses, and results -- in the aftermath of the Newtown shootings?

The challenge

No government agency collects information on guns. (They're not allowed to.)
Gun licenses are confidential in many states.
There's little regulation of the gun trade outside of licensed gun stores.
Other news organizations had recently done large projects on the most obvious stories.

What if?

The investigative hypothesis

Some sellers are trading so many guns that they should be federally licensed; some buyers are escaping criminal background checks in Internet gun trades.