introduction to machine learning
with hadoop
What do we want?
Data!
Clustering
Classification
Dimensionality Reduction
Clustering
unsupervised
may or may not be exclusive
can choose the number of groups
examples: LDA, Canopy, K-means
classification
requires labelled samples
Training phase - testing phase
examples: SVM, Decision Trees
Recommendation
compute item or user similarity
Predict ratings for other user-item combos
Dimensionality reduction
used in concert with other techniques
identifiES high-information dimensions
reduces sparsity
Now What?
feature SELECTION
How often does the user watch more than 50%?
What proportion of views are TV shows?
How many devices does the user use?
How many devices does the user use?
What genres does the user rate highly?
Mahout vectors
Numerically indexed
Indices must be consistent across all data
Clustering: SequenceFile<IntWritable, VectorWritable>
Classification: SequenceFile<Text, VectorWritable>
Mahout TOols
seqdirectory - create sequence files from input data
seq2sparse - tokenize, vectorize data
seqdumper - debug your pipeline
rowid - automatically generate dictionaries
Apache Crunch
Create complex MR data flows
Easy to have multiple input and output files
Comes with built-in methods for operations like join, group, count
You can write output as Mahout vectors
Production Concerns
Input splits
RAM per slot
Separate clusters?
Thanks!
