• Karl Ho is:

  • Professor of Instruction at University of Texas at Dallas (UTD) School of Economic, Political and Policy Sciences (EPPS)
  • Co-founder of the UTD Social Data Analytics and Research program (SDAR)
  • Projects
    • Taiwan Studies Initiative at UTD EPPS funded by MOE
    • Taiwan Research Academy funded by TFD
    • UTD Math and Coding Camp (EPPS)
  • ​Co-PI: Hong Kong Election Study 
  • Co-PI: North Texas Quality of Life
  • Website: karlho.com (talks, lecture, publications)

Speaker bio.

What is data?

What is data?

What is Big Data?

The Big data is about data that has huge volume, cannot be on one computer. Has a lot of variety in data types, locations, formats and form. It is also getting created very very fast (velocity) (Doug Laney 2001).

The Big data is about data that has huge volume, cannot be on one computer. Has a lot of variety in data types, locations, formats and form. It is also getting created very very fast (velocity) (Doug Laney 2001).

Burt Monroe (2012)

5Vs of Big data 

• Volume

• Variety

• Velocity

• Vinculation

• Validity 

• Programming is a practice of using programming language to design, perform and evaluate tasks using a computer.  These tasks include:
  • Computation
  • Data collection
  • Data management
  • Data visualization
  • Data modeling
• In this course, we focus on data programming, which emphasizes programs dealing with and evolving with data.

What is programming?

Data programming

}

Why learning programming Languages?

- Maribel Fernandez 2014

Compilation

Interpretation

• Machine language
  • Assembly language
  • C

Low-level languages

• BASIC
  • REALbasic
  • Visual Basic
• C++
• Objective-C
  • Mac
• C#
  • Windows
• Java

Systems languages

• Perl
• Tcl
• JavaScript
• Python

Scripting languages

DRY – Don’t Repeat Yourself

Write a function!

Function example

# Create preload function
# Check if a package is installed.
# If yes, load the library
# If no, install package and load the library

preload<-function(x)
{
  x <- as.character(x)
  if (!require(x,character.only=TRUE))
  {
    install.packages(pkgs=x,  repos="http://cran.r-project.org")
    require(x,character.only=TRUE)
  }
}

Why learning programming?

learning how to program can significantly enhance how social scientists can think about their studies, and especially those premised on the collection and analysis of digital data.

   

- Brooker 2019: 

Chances are the language you learn today will quite likely not be the language you'll be using tomorrow.

What is R?

R is an integrated suite of software facilities for data manipulation, calculation and graphical display.

- Venables, Smith and the R Core team

R

Source: Nick Thieme. 2018. R Generation: 25 years of R https://rss.onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2018.01169.x 

What is R?

It is:

• ​Large, probably one of the largest based on the user-written add-ons/procedures

• Object-oriented

• Interactive

• Multiplatform: Windows, Mac, Linux

What is Python?

History

• Flexibility and Simplicity - easy to learn.

• Community providing a more standard
  programming language

• Suitability - higher level of abstraction than alternative languages traditionally used.

• Multi-platform: Windows, MacOS and Linux
  Libraries - modules that can be used to extend
  the basic features of the language.

• Free .... and stable

Why Python?

Hunt 2019

Design Philosophy

PEP 20- The Zen of Python

“A large complex system should have multiple levels of extensibility. This maximizes the opportunities for users, sophisticated or not, to help themselves.”

Resources

Installations

Running Python

IDE's

Choice of Integrated Desktop Environment matters!

There are plenty of IDE available for python programming and developments.  To name a few:

1. PyCharm
2. Visual Studio Code
3. IDLE
4. Jupyter Notebook
5. Google Colab (notebook on Google cloud)
6. Spyder
7. Rodeo (RStudio look alike)
8. RStudio (with reticulate)

Interpreter

Python is a script language, which means that your code is converted to machine code by a Python interpreter.

Therefore, choosing which interpreter to use for a project is an important decision

Interpreter

Python 2.7 is behind most production applications.  However, it is reaching deprecation by 2020!

If you are new, it will not hurt to learn from 2.7.  Yet, use 3.x!!

Python 2.7 and 3.x

• Python 2 was launched in 2000 and is still popular.

• Python 3 was launched in 2008 and is not backward compatible with Python 2.7.

• MacOS has built-in Python 2.7.

• Recommended: Install Python3 but not replacing 2.7.

• Check version:

The script window: 
You can store a document of commands you used in R to reference later or repeat analyses 

Environment:
Lists all of the objects

Console:
Output appears here. The > sign means R is ready to accept commands. 

Plot/Help:
Plots appear in this window. You can resize the window if plots appear too small or do not fit.

The script window: 
You can store a document of commands you used in R to reference later or repeat analyses 

Environment:
Lists all of the objects

Console:
Output appears here. The > sign means R is ready to accept commands. 

Plot/Help:
Plots appear in this window. You can resize the window if plots appear too small or do not fit.

R Programming Basics

Importing data

mydata <- read.csv(“path”,sep=“,”,header=TRUE)
mydata.spss <- read.spss(“path”,sep=“,”,header=TRUE)
mydata.dta <- read.dta(“path”,sep=“,”,header=TRUE)

Note:

• R is absolutely case-sensitive

• R uses extra backslashes to recognize path

• Read data directly from GitHub:

happy=read.csv("https://raw.githubusercontent.com/kho7/SPDS/master/R/happy.csv")

Accessing variables

To select a column use:

mydata$column

For example:

Manipulating variables

Recoding variables

For example:

mydata$Age.rec<-recode(mydata$Age, "18:19='18to19'; 20:29='20to29';30:39='30to39'")

• Start with a project

• Why?
  • File management
  • History
  • Version control using git or svn
  • Read Jenny Byran's advice
    • Start with a project and stick with it
    • Use the here package

API

• APIs can take many different forms and be of varying quality and usefulness. 

• RESTful API (Representational State Transfer) is a means of transferring data using web protocols

• Example:
  • Crossref API
    http://api.crossref.org/works/10.1093/nar/gni170
  • Taiwan Legislative Yuan API
    https://www.ly.gov.tw/WebAPI/LegislativeBill.aspx?from=1050201&to=1050531&proposer=&mode=json

Like open data, data available through API are generally:

• Structured

• Somewhat documented

• Not necessary fully open

• Subject to the discretion of data providers

• E.g. Not all variables are available, rules may change without announcements, etc.

For the type of found data not available via API or open access, one can use non-API methods to collect this kind of data.  These methods include scraping, which is to simulate web browsing but through automated scrolling and parsing to collect data. These data are usually non-structured and often times noisy.  Researchers also have little control over data generation process and sampling design.

Non-API data are generally:

• Non-structured

• Noisy

• Undocumented with no  or little information on sampling

This workshop demonstrates how to collect stock data using:

Link to RStudio Cloud:

https://posit.cloud/content/6625059

- Need a GitHub and RStudio Account

Link to class GitHub:

https://github.com/datageneration/nchu

Workshop II: Data collection

Assignment 1

1. Install R and RStudio

2. Download the R program in class GitHub (under codes)

   1. DPR_stockdata.R

3. Can you download TSM's (台積電） data in the last three years？

4. Plot the TSM data in the last three years using the sample codes (plot and ggplot2)

Illustration: Collecting COVID data 

This workshop demonstrates how to collect COVID data using:

1. API methods
   1. Johns Hopkins University Center for Systems Science and Engineering (CSSE) (map | GitHub)
   2. Our world in Data (website | GitHub)
   3. New York Times (GitHub)

Data: Total cases per million

Data: Daily COVID deaths

Data: Death data (Asia)

Data: Death data (Europe)

Data: COVID cases~predictors

Data: COVID cases~predictors