Evaluating Clustering Quality in Centroid-Based Clustering Using Counterfactual Distances
摘要
Evaluating clustering quality is essential for selecting the optimal number of clusters and for comparing clustering algorithms. Several internal clustering quality indices have been proposed such as the Silhouette score and the Variance Ratio Criterion (VRC) with well-known advantages and limitations. In this work, we present the idea of counterfactuals to quantify cluster separation. Counterfactuals have been recently introduced in the context of clustering to quantify minimum modifications to be applied to a point of one cluster in order to be assigned to another cluster. We exploit counterfactual distances in the context of k-means clustering and define the separation between clusters considering such distances. Then we use the proposed separation measure to define a clustering quality score as the ratio of total separation over total intra-cluster variance of a clustering solution. We evaluated the effectiveness of the proposed score against Silhouette and VRC using various real-world digit datasets.