Social and Political Data Science: Introduction

Knowledge Mining

Karl Ho

School of Economic, Political and Policy Sciences

University of Texas at Dallas

Introduction

Illustration: Dimensionality of data

Source: https://medium.com/analytics-vidhya/classifying-malignant-or-benignant-breast-cancer-using-svm-fe36f139dd21

Illustration: Support Vector Machines

Source: https://blog.statsbot.co/support-vector-machines-tutorial-c1618e635e93

Overview

What is Knowledge Mining?
- Knowledge Discovery + Data Mining
- Statistics and Machine Learning
Statistical modeling: the two cultures
Conventional statistical methods and machine learning
Statistical/Machine Learning methods

What is Knowledge Mining?

Knowledge Discovery + Data Mining
Wei et al. (2003):
- Knowledge discovery refers to the overall process of discovering useful knowledge from data
- Data mining refers to the extraction of patterns from data.

What is Knowledge Mining?

Knowledge discovery can be performed on structured databases

Source: Wei, Chih-Ping, Selwyn Piramuthu, and Michael J. Shaw. "Knowledge discovery and data mining." In Handbook on Knowledge Management, pp. 157-189. Springer, Berlin, Heidelberg, 2003.

What is Knowledge Mining?

Data mining generally refers to the methods or techniques used to identify the patterns in data,
It can be broadly structured into several categories:
- classification
- clustering
- dependency analysis
- text mining

What is Knowledge Mining?

Source: Wei, Chih-Ping, Selwyn Piramuthu, and Michael J. Shaw. "Knowledge discovery and data mining." In Handbook on Knowledge Management, pp. 157-189. Springer, Berlin, Heidelberg, 2003.

What is Knowledge Mining?

Both classes of methods are subsumed under Machine Learning now
More generally under Unsupervised Machine Learning
- Pattern recognition
- Group/class identication
- Hypothesis formation (vs. Hypothesis confirmation)

Bonferroni's principle

(roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find crap.

- Rajaraman A., Leskovec J. and Ullman J. Mining of Massive Datasets

Today we live in a data rich, information driven, knowledge strained, and wisdom scant world.

- Graham Williams 2021

- Rajaraman A., Leskovec J. and Ullman J. Mining of Massive Datasets

Data mining overlaps with:

Databases: Large-scale data, simple queries
Machine learning: Small data, Complex models
CS Theory: (Randomized) Algorithms

Different cultures:

To a Database person, data mining is an extreme form of analytic processing – queries that examine large amounts of data
- Result is the query answer
To a Machine Learning person, data-mining is the inference of models
- Result is the parameters of the model

Data Mining and Machine Learning

association rules
recursive partitioning or decision trees, including CART (classification and regression trees) and CHAID (chi-squared automatic interaction detection), boosted trees, forests, and bootstrap forests
multi-layer neural network models and “deep learning” methods
naive Bayes classifiers and Bayesian networks
clustering methods, including hierarchical, k-means, nearest neighbor, linearand nonlinear manifold clustering
support vector machines
“soft modeling” or partial least squares latent variable modeling

Data Mining methods

What is machine learning?

Field of study that gives computers the ability to learn without being explicitly programmed.

- Arthur Samuel 1959

Reference:

Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3), 210-229.

A computer can be programmed so that it will learn to play a better game of checkers than can be played by the person who wrote the program.

Programming computers to learn from experience should eventually eliminate the need for much of this detailed programming effort.

What is machine learning?

The ultimate goal of data modeling is to explain and predict the variable of interest using data. Machine learning is to achieve this goal using computer algorithms in particular to make the prediction and solve the problem.

Source: Tom Mitchell website

According to Carnegie Mellon Computer Science professor Tom M. Mitchell,

"Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience."

What is machine learning?

“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”

Source: Tom Mitchell website

Study the past if you would define the future.

– Confucius

Tom Mitchell. 1997. Machine Learning, McGraw Hill.

What is machine learning?

Machine learning is a science of the artificial. The field's main objects of study are artifacts, specifically algorithms that improve their performance with experience.

- Langley, 1996

Machine learning is programming computers to optimize a performance criterion using example data or past experience.’

- Alpaydin, 2004

What is machine learning?

Machine learning is an area of artificial intelligence concerned with the study of computer algorithms that improve automatically through experience. In practice, this involves creating programs that optimize a performance criterion through the analysis of data.

- Sewell, 2006

What is machine learning?

To statisticians, the “improve through experience” part is the process of validation or cross validation. Learning can be done through repeated exercises to understand data.

What is machine learning?

Machine learning involves having computer or statistics programs do repeated estimations, like human learns from experience and improve actions and decisions. This is called the training process in machine learning.

Statistics, Knowledge Mining and Machine Learning

Statistics refresher

Statistics:

Find and test data (data production and collection)
- Made data (surveys, experiments, interviews) based on theory and hypotheses
- Found data (web data, social data, machine generated data) from all sources
Make data ready for analysis (data management)
Explore data (means, variances, distribution) - Descriptive statistics
Explain data (correlation, cross-tabulation, regression) - Inferential statistics

Parametric vs. Non-parametric models

Sample and population
Generalization
Representation

Leo Breiman

1928 – 2005

Source: https://en.wikipedia.org/wiki/Leo_Breiman

Statistical Modeling: The Two Cultures
CART (Classification and Regression Trees)

What can Machine Learning do and do better?

Machine Learning can do:

Prediction
Classification
Give useful information for problem solving and decision making

Machine Learning and Conventional statistical methods

Statistics: testing hypotheses
Machine learning: finding the right hypothesis
Overlap:
Decision trees (C4.5 and CART)
Nearest-neighbor methods
Bridging the two:
Most machine learning algorithms employ statistical techniques

Statistical Modeling:
The Two Cultures

Leo Breiman 2001: Statistical Science

One assumes that the data are generated by a given stochastic data model.

The other uses algorithmic models and treats the data mechanism as unknown.

Data Model

Algorithmic Model

Small data

Complex, big data

Theory:
Data Generation Process

Data are generated in many fashions. Picture this: independent variable x goes in one side of the box-- we call it nature for now-- and dependent variable y come out from the other side.

Theory:
Data Generation Process

Data Model

The analysis in this culture starts with assuming a stochastic data model for the inside of the black box. For example, a common data model is that data are generated by independent draws from response variables.

Response Variable= f(Predictor variables, random noise, parameters)

Reading the response variable is a function of a series of predictor/independent variables, plus random noise (normally distributed errors) and other parameters.

Theory:
Data Generation Process

Data Model

The values of the parameters are estimated from the data and the model then used for information and/or prediction.

Theory:
Data Generation Process

Algorithmic Modeling

The analysis in this approach considers the inside of the box complex and unknown. Their approach is to find a function f(x)-an algorithm that operates on x to predict the responses y.

The goal is to find algorithm that accurately predicts y.

Theory:
Data Generation Process

Algorithmic Modeling

Source: https://www.mathworks.com

Machine Learning and Conventional statistical methods

Source: Attewell, Paul A. & Monaghan, David B. 2015. Data Mining for the Social Sciences: an Introduction, Table 2.1, p. 27

Machine learning methods

Regression	Classification	Clustering	Q-Learning
Linear regression	Logistic regression	- K-Means Clustering	State Action Reward State Action (SARSA)
Polynomial regression	K-Nearest Neighbors	- Hierarchical Clustering	Deep Q-Network
Support vector regression	Support Vector Machines	Dimensionality Reduction	Markov Decision Processes
Ridge Regression	Kernal Support Vector Machines	Principal Component Analysis	Deep Deterministic Policy Gradient (DDPG)
Lasso	Naïve Bayes	Linear Discriminant Analysis
ElasticNet	Decision Tree	Kernal PCA
Decision tree	Random forest
Random forest

Supervised Learning	Unsupervised Learning	Reinforcement Learning

Machine Learning and Conventional statistical methods

Statistics: testing hypotheses
Machine learning: finding the right hypothesis
Overlap:
Decision trees (C4.5 and CART)
Nearest-neighbor methods
Bridging the two:
Most machine learning algorithms employ statistical techniques

Wickham and Grolemund: Data analytics:

Hypothesis generation
Hypothesis confirmation

Which one goes first?

Trevor Hastie and Robert Tibshirani

Bradley Efron and Trevor Hastie

Judea Pearl and Dana MacKenzie

Everyone is a teacher because the ability to teach anything connects to the ability to learn what is being taught.

- John Sibert

Learn like you are to teach.

Q & A

Question: Why R?

Answer: There are many software/platform options including Python, Weka, SAS JMP and SPSS. R is most accessible and not proprietary to specific method. Coupled with other features and systems (e.g. visualization and parallel processing), it facilitates data programming and presentation in a coherent ecosystem.

Knowledge Mining

Introduction

Illustration: Dimensionality of data

Illustration: Support Vector Machines

Overview

What is Knowledge Mining?

Knowledge Discovery + Data Mining

Statistics and Machine Learning

Statistical modeling: the two cultures

Conventional statistical methods and machine learning

Statistical/Machine Learning methods

What is Knowledge Mining?

Knowledge Discovery + Data Mining

Wei et al. (2003):

Knowledge discovery refers to the overall process of discovering useful knowledge from data

Data mining refers to the extraction of patterns from data.

What is Knowledge Mining?

Knowledge discovery can be performed on structured databases

What is Knowledge Mining?

Data mining generally refers to the methods or techniques used to identify the patterns in data,

It can be broadly structured into several categories:

classification

clustering

dependency analysis

text mining

What is Knowledge Mining?

What is Knowledge Mining?

Both classes of methods are subsumed under Machine Learning now

More generally under Unsupervised Machine Learning

Pattern recognition

Group/class identication

Hypothesis formation (vs. Hypothesis confirmation)

Bonferroni's principle

(roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find crap.

What is machine learning?

What is machine learning?

What is machine learning?

What is machine learning?

Machine learning is a science of the artificial. The field's main objects of study are artifacts, specifically algorithms that improve their performance with experience.

- Langley, 1996

Machine learning is programming computers to optimize a performance criterion using example data or past experience.’

- Alpaydin, 2004

What is machine learning?

Machine learning is an area of artificial intelligence concerned with the study of computer algorithms that improve automatically through experience. In practice, this involves creating programs that optimize a performance criterion through the analysis of data.

- Sewell, 2006

What is machine learning?

To statisticians, the “improve through experience” part is the process of validation or cross validation. Learning can be done through repeated exercises to understand data.

What is machine learning?

Machine learning involves having computer or statistics programs do repeated estimations, like human learns from experience and improve actions and decisions. This is called the training process in machine learning.

Statistics, Knowledge Mining and Machine Learning

Statistics refresher

Statistics:

Parametric vs. Non-parametric models

Sample and population

Generalization

Representation

Leo Breiman

Statistical Modeling: The Two Cultures

CART (Classification and Regression Trees)

What can Machine Learning do and do better?

Machine Learning can do:

Machine Learning and Conventional statistical methods

Statistics: testing hypotheses

Machine learning: finding the right hypothesis

Overlap: Decision trees (C4.5 and CART) Nearest-neighbor methods

Bridging the two: Most machine learning algorithms employ statistical techniques

Statistical Modeling: The Two Cultures

Leo Breiman 2001: Statistical Science

Theory: Data Generation Process

Theory: Data Generation Process

Data Model

Theory: Data Generation Process

Data Model

Theory: Data Generation Process

Algorithmic Modeling

Theory: Data Generation Process

Algorithmic Modeling

Machine Learning and Conventional statistical methods

Machine learning methods

Machine Learning and Conventional statistical methods

Overlap:
Decision trees (C4.5 and CART)
Nearest-neighbor methods

Bridging the two:
Most machine learning algorithms employ statistical techniques

Statistical Modeling:
The Two Cultures

Theory:
Data Generation Process

Theory:
Data Generation Process

Theory:
Data Generation Process

Theory:
Data Generation Process

Theory:
Data Generation Process

Overlap:
Decision trees (C4.5 and CART)
Nearest-neighbor methods

Bridging the two:
Most machine learning algorithms employ statistical techniques