Please open

https://data4news.com/numlock-nlp

(It takes a second to load, so just open it up and hang tight, thanks!)

Hi πŸ‘‹

If you have a computer

 

Hello,  my name is...

 

Prof. Dhrumil Mehta

 

 

 

 

 

 

dm3494@columbia.edu

@dmil

πŸ“Numeracy

πŸ“Statistics

πŸ“Polling Data

 

Toaday

About  Me:


  • Columbia University Graduate School of Journalism
    Teach in Data MS & Advisor for CS + Journalism Dual MS
    Assistant Director, Tow Center for Digital Journalism


  • Harvard Kennedy School
    Visiting Associate Professor of Public Policy

  • Formerly:
    • Database Journalist, Politics @ FiveThirtyEight
    • Software Development Engineer @ Amazon
    • Education
      • BA in Philosophy
      • MS in Computer Science

Every Wednesday in Brown Inst.

Not All Numbers Are Created Equal

Let's take a look at this semantic map of Numlock News

 

https://data4news.com/numlock-nlp

Numlock News

Let's take a look at this semantic map of Numlock News

 

https://data4news.com/numlock-nlp

Numlock News

Pick a number you find interesting. Copy and paste the whole snippet (including the links) into this padlet.

 

https://padlet.com/data4news/numlock

We will come back to this later

Learning Objectives

There are different kinds of numbers that must be interpreted and presented differently

 

βœ”οΈ Identify what kind of number you're dealing with

βœ”οΈ Understand it's limitations

βœ”οΈ Know what kinds of questions to ask of it before reporting on it

Not All Numbers Are Created Equal

Counts and Measurements

Summary Statistics

Probabilities

Inferential Statistics

Indexes

Scaled Numbers

Election Forecast

a very complex kind of probabilistic number

Methodological Choices

I'm sure these aren't the only kinds of numbers...

...but they're all very different from one another

 

 

 

more methodological choices to vet

  • your income this year
    (count)
  • the average graduating salary of everyone
    in the last class
    (summary statistic - descriptive)
  • the average income of all Americans
    (summary statistic - inferential)
  • average income of a journalist anywhere in the world
    (index)
  • forecasted journalist income in 10 years given assumptions about world economy
    (probabilistic forecast)

Before you cite a number:

  • What is the original source of this number?
    • Who first collected it and why?
  • What kind of number is it?
    A count? a summary? a prediction? an estimate? an inferential statistic? an index?
  • How was it acquired or calculated?
  • What is the universe of data that it comes from?​​​​​​​
  • What methodological choices were in inherent in 
    • creating the number
    • selecting the number to be presented
  • What methodological choices are you making by finding/selecting/calculating/presenting this number?

Let's take come back to this padlet

 

Numlock News

In the comment for the number you picked, write :


1. what kind of number it is
 

2. what you'd want to know about this number before your cite it in your reporting

Learning Objectives

βœ”οΈ There are different kinds of numbers that must be interpreted and presented differently

Not All Numbers Are Created Equal

Intro to Stats for Journalists

Learning Objectives

 

βœ”οΈ Statistics summarize datasets just like a paragraph can summarize a story. Some things are lost in the process.

 

βœ”οΈ It's important to determine if the summary statistic does a good job of summarizing the underlying distribution before you cite it.

Statistics

Summary Stats

πŸ‘» Scary Math Symbol?

πŸ‘ Seems fine! 

Mean

 

 

 

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

Weighted Average

 

 

Text

Example

Examples

Averages are weighted by the square root of the number of polls that a particular pollster conducted for that particular type of election in that particular cycle.

Median

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

Mode

Examples

 

 

 

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

Range

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

Correlation

https://www.investopedia.com/terms/n/negative-correlation.asp

Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'.

https://m.xkcd.com/552/

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

Correlation & Causation

 

Standard Deviation

 

 

 

 

 

 

Variance

 

 

 

 

 

 

 

Examples

 

 

  

Examples

 

 

  

Ben Orlin β€” Math with bad drawings

Ben Orlin β€” Math with bad drawings

✏️ Mystery Data

 

Learning Objectives

βœ”οΈ Use formulas in spreadsheets to calculate summary statistics (bonus: use pivot tables to do the same)

βœ”οΈ Understanding the limitations of summary statistics

βœ”οΈ Learn to make charts in spreadsheets

Mystery Data

https://data4news.com/mystery-dataset/

Describe 4 mystery datasets with summary statistics...what can you tell me about these mystery data?

 

 

Calculate:

- Mean

- Median

- Mode

- Correlation

- Variance

Done?

Bonus 1: Redo the calculations in the LONG tab with a pivot table.
Bonus 2: Explore the data in the BONUS tab, tell us what you find!

Dataset 1 is done for you

Examples of how to write the formulas are in red

Anascombe's Quartet

Median Grade: 80%

Median Grade: 80%

Median Grade: 80%

What do you do when all the summary statistics fail you? 

Plot the Distribution

https://fivethirtyeight.com/features/al-gores-new-movie-exposes-the-big-flaw-in-online-movie-ratings/

But be careful when plotting

Pearson correlation is ????

Pearson correlation is 0.9909

because there are 40 duplicate data pts in top right and bottom left corner

BE VERY CAREFUL OF SCATTER PLOTS

Correlation != Causation

Normal Distribution

Pay Attention To Distribution Of Data

1977

πŸ“š A Hypothesis Is A Liability

 

Produce checkable artifiacts

Exploratory Data Analysis with LLMs

❌ Hi AI, please answer this question about my data

 

βœ… Hi AI, please write code to answer this question about my data

Editorial Choices

 

Let's look at this rolling average.

Copy of Copy of Intro Stats

By Dhrumil Mehta

Copy of Copy of Intro Stats

intro stats

  • 715