Data

## install.packages("NbClust")
## install.packages("reshape2")
## install.packages("viridis")
## install.packages("factoextra")
## install.packages("cluster")
## install.packages("magrittr")

library(factoextra)
library(cluster)
library(magrittr)
library(Wu)

# Load  and prepare the data
data("USArrests")

my_data <- USArrests %>%
  na.omit() %>%          # Remove missing values (NA)
    scale() %>%               # Scale variables
    as.data.table()

my_data %>% DT()

Data Summary

## colnames(my_data)

Vars <- colnames(my_data)

FactorVars <- NULL

t <- Table1n(obj = my_data, Vars = Vars, FactorVars = FactorVars)

t %>% prt()

Distance

• Distance by four continuous variables
• Methods for Measuring Distance
• Fred Szabo: Manhattan Distance
• Euclidean Distance \[d_{euc}(x,y)=\sqrt{\sum_{i=1}^n(x_i - y_i)^2}\]
• Manhattan Distance is the distance a car would drive in a city (Manhattan). It is the sum of absolute differences. It is also known as \(L^1\) norm. \[d_{man}(x,y)=\sum_{i=1}^n|(x_i - y_i)|\]
• Pearson correlation distance: it measures the degree of a linear relationship between two profiles. It is another type of dissimilarity measuremeants called correlation-based distance. r values from -1 to 1. It could be converted to range of 0 to 1. \[d_{cor}(x,y) = 1- \frac{\sum_{i=1}^n(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n(x_i - \bar{x})^2}\sum_{i=1}^n(y_i - \bar{y})^2}\] \[d = (1 - r)/2\]
• Spearman correlation distance: computes the correlation between the rank of x and y.
• Kendall Correlation Distance. \[d_{kend}(x,y) = 1 - \frac{n_c - c_d}{n(n-1)/2}\] \[n_c : total\ number\ of\ concordant\ pairs\] \[n_d:\ total\ number\ of\ discordant\ pairs\] \[n(n-1)/2 = total\ number\ of\ possible\ pairings\]

library(factoextra)
res.dist <- get_dist(USArrests, stand = TRUE, method = "manhattan")

fviz_dist(res.dist, gradient = list(low = "#00AFBB", mid = "white", high = "#FC4E07"))

Gower Dissimilarity

• Gower's dissimilarity measure for mixed numeric/categorical data
• The overall similarity score is the sum of all the individual variable similarity divided by the total possible comparisons.

library(cluster)

Dist <- daisy(iris, metric = c("gower"))
library(factoextra)
fviz_dist(Dist)

## hclust(Dist)

Silhouette

• Silhouette measures the consistency within clusters of data
• Its value ranges from -1 to +1, high value indicates the object is well matched to its own cluster and poorly matched to nerighoring clusters.
• a: Mean distance within cluster for i and all other data points.
• b: smallest mean distance of i to all points in any other cluster. The cluster with this smallest mean dissimilarity is said to be the neighboring cluster.
• silhouette values is \(\frac{b - a}{max(a,b}\)
• score is zero if cluster size = 1
• pam: partition around medoids function
• Medoids are representative object within a cluster. It is similar in concept to means or centroids, but medoids are always restricted to be members of the cluster.

## silhouette plot
pamx <- pam(Dist, 3)
sil = silhouette(pamx$clustering, Dist)
plot(sil)

Kmeans Clustering

library("factoextra")

set.seed(123)
fviz_nbclust(my_data, kmeans, nstart = 10, method = "gap_stat", nboot = 500) + labs(subtitle = "Gap statistic method")

Plot Kmeans Clusters

set.seed(123)
km.res <- kmeans(my_data, 3, nstart = 25)
# Visualize
library("factoextra")
fviz_cluster(km.res, data = my_data, ellipse.type = "convex", palette = "jco", ggtheme = theme_minimal())

## GAP Statistic

• [[https://statweb.stanford.edu/~gwalther/gap][GAP Statist]]

To estimate the optimal number of clusters. Provide a statistical procedure to formalize that heuristics.

D: sum of the pairwise distances for all points in cluster r W: Sum of cluster mean of D. If D is the squared Euclidean distance, then W is the pooled within-cluster sum of squared around the cluster means.

• [[https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#:~:text=The%2520gap%2520statistic%2520compares%2520the,yields%2520the%2520largest%2520gap%2520statistic).][Gap Statistic Method]]

The gap statistic compares the total within intra-cluster variation for different values of k with their expected values under null reference distribution of the data. The estimate of the optiomal clusters will be valued that maximize the gap statistics (that yields the largest gap statistic). This means that the clustering structure is far away from the random uniform distribution of points.

1, calculate within-cluster variation W (observed) 2, Generate B reference data sets with a random uniform distribution. calculate W (expected) 3, Compute the estimate gap statistic as the deviation of the observed W from expected W 4, Choose the smallest value of cluster number k, and the gap statistic is within one standard deviation of the gap at k+1

Hierarchical Clustering

• Cluster Dendrogram
• A dendrogram is a diagram that shows the hierarchical relationship between objects.

# Compute hierarchical clustering
library(Wu)
res.hc <- USArrests %>%
  scale() %>%                    # Scale the data
  dist(method = "euclidean") %>% # Compute dissimilarity matrix
  hclust(method = "ward.D2")     # Compute hierachical clustering

# Visualize using factoextra
# Cut in 4 groups and color by groups
fviz_dend(res.hc, k = 4, # Cut in four groups
          cex = 0.5, # label size
          k_colors = c("#2E9FDF", "#00AFBB", "#E7B800", "#FC4E07"),
          color_labels_by_k = TRUE, # color labels by groups
          rect = TRUE # Add rectangle around groups
          )

Determine the Optimal Number of Clusters

set.seed(123)

# Compute
library("NbClust")

res.nbclust <- USArrests %>% scale() %>% NbClust(distance = "euclidean", min.nc = 2, 
    max.nc = 10, method = "complete", index = "all")

*** : The Hubert index is a graphical method of determining the number of clusters. In the plot of Hubert index, we seek a significant knee that corresponds to a significant increase of the value of the measure i.e the significant peak in Hubert index second differences plot.

*** : The D index is a graphical method of determining the number of clusters. In the plot of D index, we seek a significant knee (the significant peak in Dindex second differences plot) that corresponds to a significant increase of the value of the measure.

• Among all indices:
  
• 9 proposed 2 as the best number of clusters
• 4 proposed 3 as the best number of clusters
• 6 proposed 4 as the best number of clusters
• 2 proposed 5 as the best number of clusters
• 1 proposed 8 as the best number of clusters

• 1 proposed 10 as the best number of clusters

             ***** Conclusion *****                            

• According to the majority rule, the best number of clusters is 2

# Visualize
library(factoextra)
fviz_nbclust(res.nbclust, ggtheme = theme_minimal())