Math Lectures

Combine NLP with supervised and unsupervised learning to classify math lectures

Objective:

Use closed captioning from 92 lectures

Save XML file in a directory

Objective: Use supervised and unsupervised learning techniques to best classify math lectures.

(Clean the data)

- Train Doc2Vec Model

- Dimensionality Reduction for Visualization

Clustering and Similarity

- Clustering the data

- KMeans Silhouette Scores

- 9, 10, and 11 clusters

- Topic Extraction (NMF and LDA)

Modeling

- Tf-idf vectorization

- Initial model

- Parameter Search

- Parts of Speech

- Final Model

Part 2

Part 1

Clean and tokenize the text

lecture id

str

label

label

Spacy Doc

 - extract lemmas, remove punctuation and stop words

Use Doc2Vec to vectorize each lecture

- Converts each lecture into a 65 dimensional vector

[ x 1, x 2 ... x 64, x 65 ]

Lecture

i

i

i

i

i

Epoch 1

Epoch 2

Epoch 3

Epoch 100

...

Extract the numerical representation for each lecture

*tagged_data[0] = 1 full lecture

Calculate cosine similarity of lectures

Metrics for clustering and determining similarity

Reducing the Dimensionality

for Visualization

Using PCA

Using t-SNE

The scatter plot of the t-SNE components is much easier to interpret. For this reason t-SNE was chosen as the prefered method for reducing the dimensionality

Y = pca.fit_transform(vecs)

Yt = tsne.fit_transform(vecs)

Clustering the Data

Agglomerative Clustering - 10 Clusters

Spectral Clustering (damping .8)

Mean Shift (all bandwidth)

KMeans - 10 Clusters

Clustering the Data with KMeans

KMeans 9 Clusters

KMeans 10 clusters

KMeans 11 Clusters

Actual labels

9 clusters by Subject

59/69 points correctly clustered

8/10 subjects correctly clustered

10 Clusters by Subject

22/69 points correctly clustered

3/10 Subjects correctly clustered

11 Clusters by Subject

27/69 points correctly clustered

4/10 Subjects correctly clustered

Results of Clustering

9 Clusters

10 Clusters

11 Clusters

By Subject

By Professor

85.5%

31.88%

39.13%

46.37%

10.14%

33%

Score is based on cluster completeness. (Only lectures perfectly in their true label group are scored)

But wait...

Calculus?

Why here?

?

?

This calculus lecture ended up far from its cluster  during the t-SNE decomposition

Why can't the KMeans discern between data structures and algorithms?

Why does this AI lecture get clustered to Differential Equations?

Calculus is very close to Differential Equations.

Math for Computer Science, Artificial Intelligence, Algorithms, Data Structures in this context are more closely related to one another than the others. This relationship is captured in coordinates of the lectures

Topic Extraction using Non negative matrix factorization

Data Structures

Algorithms

Do these look that different?

Topic Extraction using LDA

Winston AI 10

Differential Equations

(Latent Dirichlet Allocation)

AI

Modeling with TF-IDF vectorization

Initial Results with TF-IDF vectores

Logistic Regression

Multinomial NB

Random Forest

K Neighbors

Logistic Regression

61%

Random Forest

80%

Multinomial NB

57%

KNN

63%

Other feature generation

- Extract parts of speech (POS)

- Count occurrence of POS by lecture

- Divide each POS by lecture length

List of Spacy Docs

True

False

norm=

using POS only

Logistic Regression

Random Forest

90%

74%

using POS / len(total_lecture_pos)

24%

92%

Parameter Search

min_df = 19, score = 95.65%

tf-idf vectors

+

POS vectors

=

Random Forest (n_estimators=200, max_depth=4, min_samples_leaf=4, random_state=43, class_weight='balanced')

tf-idf df_min  = 25

Cross Validation 5 folds

mean

New X

Modeling:

Clustering:

+ Able to identify similar subjects.

- Unable to decipher closely related subjects

- Unable to decipher professors

+ Very accurate

+ able to decipher professors

For further study

- Generate an overall rating of each lecture

- Youtube ratings, comments, view counts

- Predict the quality of newly posted content

 - match most relevant new content for a given user

        - Scale the data collection

- Programmatically obtain lecture subtitles from youtube's API

- Create new features based on sentiment analysis of comments