• Karl Ho is:

  • Professor of Instruction at University of Texas at Dallas (UTD) School of Economic, Political and Policy Sciences (EPPS)
  • Co-founder of the UTD Social Data Analytics and Research program (SDAR)
  • Founder of DataGeneration.org
  • Author of Data Programming with R
  • Co-Principal Investigator of the Hong Kong Election Study project
  • Website: karlho.com (talks, lecture, publications)

Speaker bio.

Overview:

Pre-class preparation

In the beginning.....

Prepare for class

Overview:

Why Data Science? Why now?

McKinsey & Co., An Executive’s Guide to AI

Data fluency

Hugo Bowne-Anderson. 2019. "What 300 L&D leaders have learned about building data fluency"

Data fluency

Hugo Bowne-Anderson. 2019. "What 300 L&D leaders have learned about building data fluency"

Everybody has the data skills and literacy to understand and perform data driven documents and tasks

Danger of immature data fluency

Types of Data Science

Rogati AI hierarchy of needs

What is data?

What is data?

What is Big Data?

The Big data is about data that has huge volume, cannot be on one computer. Has a lot of variety in data types, locations, formats and form. It is also getting created very very fast (velocity) (Doug Laney 2001).

The Big data is about data that has huge volume, cannot be on one computer. Has a lot of variety in data types, locations, formats and form. It is also getting created very very fast (velocity) (Doug Laney 2001).

Burt Monroe (2012)

5Vs of Big data 

• Volume

• Variety

• Velocity

• Vinculation

• Validity 

• Programming is a practice of using programming language to design, perform and evaluate tasks using a computer.  These tasks include:
  • Computation
  • Data collection
  • Data management
  • Data visualization
  • Data modeling
• In this course, we focus on data programming, which emphasizes programs dealing with and evolving with data.

What is programming?

Data programming

}

Why learning programming Languages?

- Maribel Fernandez 2014

Compilation

Interpretation

• Machine language
  • Assembly language
  • C

Low-level languages

• BASIC
  • REALbasic
  • Visual Basic
• C++
• Objective-C
  • Mac
• C#
  • Windows
• Java

Systems languages

• Perl
• Tcl
• JavaScript
• Python

Scripting languages

DRY – Don’t Repeat Yourself

Write a function!

Function example

# Create preload function
# Check if a package is installed.
# If yes, load the library
# If no, install package and load the library

preload<-function(x)
{
  x <- as.character(x)
  if (!require(x,character.only=TRUE))
  {
    install.packages(pkgs=x,  repos="http://cran.r-project.org")
    require(x,character.only=TRUE)
  }
}

Why learning programming?

learning how to program can significantly enhance how social scientists can think about their studies, and especially those premised on the collection and analysis of digital data.

   

- Brooker 2019: 

Chances are the language you learn today will quite likely not be the language you'll be using tomorrow.

What is R?

R is an integrated suite of software facilities for data manipulation, calculation and graphical display.

- Venables, Smith and the R Core team

R

Source: Nick Thieme. 2018. R Generation: 25 years of R https://rss.onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2018.01169.x 

What is R?

It is:

• ​Large, probably one of the largest based on the user-written add-ons/procedures

• Object-oriented

• Interactive

• Multiplatform: Windows, Mac, Linux

The script window: 
You can store a document of commands you used in R to reference later or repeat analyses 

Environment:
Lists all of the objects

Console:
Output appears here. The > sign means R is ready to accept commands. 

Plot/Help:
Plots appear in this window. You can resize the window if plots appear too small or do not fit.

The script window: 
You can store a document of commands you used in R to reference later or repeat analyses 

Environment:
Lists all of the objects

Console:
Output appears here. The > sign means R is ready to accept commands. 

Plot/Help:
Plots appear in this window. You can resize the window if plots appear too small or do not fit.

R Programming Basics

Importing data

mydata <- read.csv(“path”,sep=“,”,header=TRUE)
mydata.spss <- read.spss(“path”,sep=“,”,header=TRUE)
mydata.dta <- read.dta(“path”,sep=“,”,header=TRUE)

Note:

• R is absolutely case-sensitive

• R uses extra backslashes to recognize path

• Read data directly from GitHub:

happy=read.csv("https://raw.githubusercontent.com/kho7/SPDS/master/R/happy.csv")

Accessing variables

To select a column use:

mydata$column

For example:

Manipulating variables

Recoding variables

For example:

mydata$Age.rec<-recode(mydata$Age, "18:19='18to19'; 20:29='20to29';30:39='30to39'")

• Start with a project

• Why?
  • File management
  • History
  • Version control using git or svn
  • Read Jenny Byran's advice
    • Start with a project and stick with it
    • Use the here package

Beware of bugs in the above code; I have only proved it correct, not tried it."

 

- Donald Knuth, author of The Art of Computer Programming

Overview

Data Methods

Made

Data

}

}

Found

Data

Data Methods

Data file formats

YAML (Yet Another Markup Language or YAML Ain't Markup Language) is a data-oriented, human readable language mostly use for configuration files)

Since open data are provided by government agencies or research institutions, these data files are often:

• Structured

• Well documented

• Ready for data/research functions

API

• APIs can take many different forms and be of varying quality and usefulness. 

• RESTful API (Representational State Transfer) is a means of transferring data using web protocols

• Example:
  • Crossref API
    http://api.crossref.org/works/10.1093/nar/gni170
  • Taiwan Legislative Yuan API
    https://www.ly.gov.tw/WebAPI/LegislativeBill.aspx?from=1050201&to=1050531&proposer=&mode=json

Like open data, data available through API are generally:

• Structured

• Somewhat documented

• Not necessary fully open

• Subject to the discretion of data providers

• E.g. Not all variables are available, rules may change without announcements, etc.

For the type of found data not available via API or open access, one can use non-API methods to collect this kind of data.  These methods include scraping, which is to simulate web browsing but through automated scrolling and parsing to collect data. These data are usually non-structured and often times noisy.  Researchers also have little control over data generation process and sampling design.

Non-API data are generally:

• Non-structured

• Noisy

• Undocumented with no  or little information on sampling

Illustration: Collecting stock data 

This workshop demonstrates how to collect stock data using:

1. an R package called quantmod

Illustration: Collecting stock data 

This workshop demonstrates how to collect stock data using:

1. an R package called quantmod

Illustration: Collecting stock data 

This workshop demonstrates how to collect stock data using:

Link to RStudio Cloud:

https://rstudio.cloud/project/4631380

- Need a GitHub and RStudio Account

Link to class GitHub:

https://github.com/datageneration/nchu

Assignment 1

1. Install R and RStudio

2. Download the R program in class GitHub (under codes)

   1. DPR_stockdata.R

3. Can you download TSMC's (台積電） data in the last three years？

4. Plot the TSMA data in the last three years using the sample codes (plot and ggplot2)

Illustration: Collecting COVID data 

This workshop demonstrates how to collect COVID data using:

1. API methods
   1. Johns Hopkins University Center for Systems Science and Engineering (CSSE) (map | GitHub)
   2. Our world in Data (website | GitHub)
   3. New York Times (GitHub)

Data: Total cases per million

Data: Daily COVID deaths

Data: Death data (Asia)

Data: Death data (Europe)

Data: COVID cases~predictors

Data: COVID cases~predictors

Assignment 2

1. Download the R program in class GitHub (under codes)

   1. DPR_coviddata1.R

2. Can you download Taiwan and Germany COVID data in last three years?

3. Plot the data using the sample codes (use plot and ggplot2 functions)

Assignment 3 (optional for AP)

1. Download the R program in class GitHub (under codes)

   1. DPR_caret01.R

2. Can you predict the chance of Tsai winning including additional variable "indep" (support for Taiwan's independence")?

3. What is the new accuracy?  Better or worse? 

Illustration: Scraping Twitter data 

This workshop demonstrates how to collect Twitter data using:

1. API method (with Twitter developer account)
2. Non-API method (using Python-based twint)

• Notebook