How to choose your clustering method ?
Or
What should I do if I have a bunch of data in a n-dimensional space and somehow I need to separate it in different groups, but I'm completely lost in the sea of possibilities given by the different available algorithms ?
Felipe Delestro
delestro@biologie.ens.fr
MACHINE LEARNING JOURNAL CLUB
Tem hora que a gente se pergunta, por que é que não se junta tudo numa coisa só? Fernando Anitelli
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4431909/FernandoAnitelli.png)
Sometimes we ask ourselves, why don't we just group it all together? Fernando Anitelli
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4431909/FernandoAnitelli.png)
Linear separability
TOY DATA-SETS
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
DATASET 01
n: 10.000
std: 0.3
k: 20
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
DATASET 02
n: 10.000 & 20.000
std: 0.8 & 1.6
k: 3
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
DATASET 03
n: 10.000
std: 1.0
k: 1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
DATASET 04
n: 10.000 k: 2
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
DATASET 05
n: 10.000 k: 2
Clustering methods
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432501/sklearn-logo.png)
Methods
- K-Means
- Affinity propagation
- Spectral clustering
- Ward hierarchical clustering
- DBSCAN
- Gaussian mixtures
- Birch
- Mean shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438711/sphx_glr_plot_cluster_comparison_001.png)
K-Means
The KMeans algorithm clusters data by trying to separate samples in n groups of equal variance, minimizing a criterion known as the inertia or within-cluster sum-of-squares. This algorithm requires the number of clusters to be specified. It scales well to large number of samples and has been used across a large range of application areas in many different fields.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
K-means
DATASET 01
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4443361/Dataset01_Kmeans.png)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432762/Dataset01_Kscan.gif)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432924/Dataset01_Trial0_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432925/Dataset01_Trial1_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432926/Dataset01_Trial2_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432927/Dataset01_Trial3_Iter2-25.gif)
K-means
random initialisation
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432997/Dataset01_Kmeans__01_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432999/Dataset01_Kmeans__02_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4433000/Dataset01_Kmeans__03_Iter1-25.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4433001/Dataset01_Kmeans__04_Iter1-25.gif)
K-means
k-means++
DATASET 02
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432773/Dataset02_k1.png)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432775/Dataset02_k2.png)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432776/Dataset02_k3.png)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432778/Dataset02_k4.png)
K-means
DATASET 03
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432796/Dataset03_Kscan.gif)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
K-means
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4433049/Dataset04_ks.gif)
K-means
DATASET 05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
K-means
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4433059/Dataset05_ks.gif)
K-means
AFFINITY PROPAGATION
AffinityPropagation creates clusters by sending messages between pairs of samples until convergence. A dataset is then described using a small number of exemplars, which are identified as those most representative of other samples. The messages sent between pairs represent the suitability for one sample to be the exemplar of the other, which is updated in response to the values from other pairs. This updating happens iteratively until convergence, at which point the final exemplars are chosen, and hence the final clustering is given.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
Affinity propagation
DATASET 01
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436531/Dataset01_Damping0.9_Iter70.png)
Affinity propagation
damping = 0.9
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436529/Dataset01_Damping0.9_Iters.gif)
Affinity propagation
damping = 0.9
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436536/Dataset02_Damping0.95.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
Affinity propagation
DATASET 02
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436539/Dataset03_Damping0.9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
Affinity propagation
DATASET 03
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436562/Dataset04_Damping0.9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
Affinity propagation
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436564/Dataset05_Damping0.9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
Affinity propagation
DATASET 05
SPECTRAL CLUSTERING
It does a low-dimension embedding of the affinity matrix between samples, followed by a KMeans in the low dimensional space. It is especially efficient if the affinity matrix is sparse and the pyamg module is installed. SpectralClustering requires the number of clusters to be specified. It works well for a small number of clusters but is not advised when using many clusters.
Spectral clustering
DATASET 01
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
Spectral clustering
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436575/Dataset01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436579/Dataset02.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
Spectral clustering
DATASET 02
Spectral clustering
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4447205/Dataset02_k4.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436600/Dataset03_k10.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
Spectral clustering
DATASET 03
Spectral clustering
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436601/Dataset04_Affinity-rbf.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436602/Dataset04_Affinity-NearestNeighbors.png)
affinity: rbf
affinity: nearest neighbors
Spectral clustering
Spectral clustering
DATASET 05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
affinity: rbf
affinity: nearest neighbors
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436656/Dataset05_Affinity-rbf.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436662/Dataset05_Affinity-NearestNeighbors.png)
Spectral clustering
WARD
Hierarchical clustering is a general family of clustering algorithms that build nested clusters by merging or splitting them successively. This hierarchy of clusters is represented as a tree (or dendrogram). The root of the tree is the unique cluster that gathers all the samples, the leaves being the clusters with only one sample. Ward minimizes the sum of squared differences within all clusters. It is a variance-minimizing approach and in this sense is similar to the k-means objective function but tackled with an agglomerative hierarchical approach.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436843/Dataset01_Ward.png)
Ward Hierarchical
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436846/Dataset02_Ward.png)
Ward Hierarchical
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436851/Dataset03_Ward.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
Ward Hierarchical
Ward Hierarchical
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436857/Dataset04_Ward.png)
Ward Hierarchical
Ward Hierarchical
# connectivity matrix for structured Ward
connectivity = kneighbors_graph(X, n_neighbors=10, include_self=False)
# make connectivity symmetric
connectivity = 0.5 * (connectivity + connectivity.T)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436859/Dataset04_Ward_WithConnectivity.png)
Ward Hierarchical
Ward Hierarchical
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
DATASET 05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436860/Dataset05_Ward.png)
Ward Hierarchical
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4436862/Dataset05_Ward_WithConnectivity.png)
Ward Hierarchical
DBSCAN
The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. Due to this rather generic view, clusters found by DBSCAN can be any shape, as opposed to k-means which assumes that clusters are convex shaped. The central component to the DBSCAN is the concept of core samples, which are samples that are in areas of high density. A cluster is therefore a set of core samples, each close to each other (measured by some distance measure) and a set of non-core samples that are close to a core sample (but are not themselves core samples).
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
DBSCAN
DATASET 01
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4444042/Dataset01_DBSCAN.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437298/Dataset01_eps_scan.gif)
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
DBSCAN
DATASET 02
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437334/Dataset02_eps_scan.gif)
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437361/Dataset03_eps_scan.gif)
DBSCAN
DATASET 03
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437404/Dataset03_minsamples_scan.gif)
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
DBSCAN
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437426/Dataset04_eps_0.1.png)
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437429/Dataset04_eps_0.08.png)
DBSCAN
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
DBSCAN
DATASET 05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4437492/Dataset05_eps_scan.gif)
DBSCAN
GAUSSIAN MIXTURES
A Gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
Gaussian mixtures
DATASET 01
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438260/Dataset01_GMM_Good.png)
Gaussian mixtures
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438262/Dataset01_GMM_Bad.png)
Gaussian mixtures
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
Gaussian mixtures
DATASET 02
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438266/Dataset02_GMM.png)
Gaussian mixtures
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438268/Dataset02_GMM_Bad.png)
Gaussian mixtures
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438269/Dataset03_GMM.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438273/Dataset03_GMM_elipses.png)
Gaussian mixtures
DATASET 03
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438284/Dataset04_GMM.png)
Gaussian mixtures
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438287/Dataset05_GMM.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
Gaussian mixtures
DATASET 05
BIRCH
Balanced Iterative Reducing and Clustering using Hierarchies is an unsupervised data mining algorithm used to perform hierarchical clustering over particularly large data-sets. An advantage of BIRCH is its ability to incrementally and dynamically cluster incoming, multi-dimensional metric data points in an attempt to produce the best quality clustering for a given set of resources (memory and time constraints).
Birch
DATASET 01
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432433/Dataset01.png)
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4447228/Dataset01_Birch.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
Birch
DATASET 02
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438449/Dataset02_Birch_default.png)
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438455/Dataset02_Birch_thresh0.01.png)
Birch
threshold = 0.10
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438492/Dataset01_thresh_scan.gif)
Birch
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
DATASET 03
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446539/Dataset03_Birch.png)
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
DATASET 04
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438522/Dataset04_Birch.png)
Birch
Birch
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
DATASET 05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438523/Dataset05_Birch.png)
Birch
Mean Shift
Mean shift clustering aims to discover “blobs” in a smooth density of samples. It is a centroid-based algorithm, which works by updating candidates for centroids to be the mean of the points within a given region. These candidates are then filtered in a post-processing stage to eliminate near-duplicates to form the final set of centroids.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4438445/Dataset01_Birch.png)
Mean shift
DATASET 01
Mean shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446330/Dataset01_MeanShift_default.png)
bandwidth = estimate_bandwidth(X, quantile=0.04)
Mean shift
Mean shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446334/Dataset01_MeanShift_Quantile0.04.png)
bandwidth quantile = 0.04
Mean shift
bandwidth quantile = 0.04 | cluster_all=False
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446453/Dataset01_MeanShift_Quantile0.04_WithNoise.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432435/Dataset02.png)
Mean Shift
DATASET 02
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446347/Dataset02_MeanShift_default.png)
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446348/Dataset02_MeanShift_Quantile0.15.png)
bandwidth quantile = 0.15
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432437/Dataset03.png)
DATASET 03
Mean Shift
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446461/Dataset03_MeanShift_default.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432438/Dataset04.png)
DATASET 04
Mean Shift
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446468/Dataset04_MeanShift_default.png)
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446510/Dataset04_MeanShift_Quantile0.10.png)
bandwidth quantile = 0.10
Mean Shift
bandwidth quantile = 0.05
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446511/Dataset04_MeanShift_Quantile0.05.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4432439/Dataset05.png)
DATASET 05
Mean Shift
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446547/Dataset05_MeanShift_default.png)
Mean Shift
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446558/Dataset05_MeanShift_Quantile0.10.png)
bandwidth quantile = 0.10
Summary
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446896/Summary.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/347072/images/4446973/Fluxogram.png)
How to choose your clustering method?
By Felipe Delestro
How to choose your clustering method?
- 902