The reviewed record of science sign in
Pith

arxiv: 2606.27061 · v1 · pith:PAXIB36S · submitted 2026-06-25 · cs.AI

How to evaluate clustering with ground truth?

Pasi Fr\"anti This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 04:34 UTCgrok-4.3pith:PAXIB36Srecord.jsonopen to challenge →

classification cs.AI
keywords clustering evaluationexternal validity indexescentroid indexground truthset-matching measurescluster accuracy
0
0 comments X

The pith

Centroid index is recommended as the preferred external measure for clustering evaluation when ground truth is available.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reviews common external validity indexes for clustering when ground truth labels exist, with emphasis on set-matching-based measures. It concludes that the centroid index stands out because it operates at the cluster level and yields results that can be directly explained in terms of mismatched clusters. Readers should care since the choice of index shapes how researchers and practitioners judge the quality of clustering algorithms in real applications. When finer point-level detail is required, the pair-set index supplies a normalized score free of cluster-size bias, while clustering accuracy suits cases where every point must count equally.

Core claim

External indexes based on set matching evaluate how well a clustering matches known ground-truth partitions. The centroid index is the recommended choice because it is an intuitive cluster-level measure whose result can be explained directly. When a more detailed point-level score is needed, the pair-set index delivers a normalized value without bias from unequal cluster sizes, and clustering accuracy works if every data point must contribute equally to the score.

What carries the argument

Centroid index (CI), a set-matching measure that counts how many clusters fail to align with the ground-truth partition at the cluster level.

If this is right

  • Clustering results become directly interpretable in terms of which specific clusters mismatch the ground truth.
  • Normalized scores without size bias become available when the pair-set index is chosen instead.
  • Equal weighting of every data point is achieved by selecting clustering accuracy or similar set-matching measures.
  • Evaluation practice shifts away from indexes that lack cluster-level explainability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread adoption of the centroid index could simplify comparisons across different clustering papers.
  • The emphasis on explainability at the cluster level may encourage similar design choices in other unsupervised evaluation settings.

Load-bearing premise

That prioritizing intuitiveness and explainability at the cluster level, within the family of set-matching measures, is the right way to choose an evaluation index.

What would settle it

A controlled user study in which domain experts consistently rate another index, such as the adjusted Rand index, as more useful or accurate than the centroid index across multiple datasets would challenge the recommendation.

read the original abstract

External indexes can be used for cluster evaluation when ground truth is available. We review the most common external validity indexes focusing on set-matching-based measures. We recommend centroid index (CI), because it is an intuitive cluster-level measure with an explainable result. If we need a more fine-tuned, point-level measure, there are more choices. Pair-set index (PSI) provides a normalized score which is not biased by cluster sizes. If all points should matter equally, then clustering accuracy (ACC) or any other set-matching measure is suitable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript reviews common external validity indexes for clustering when ground truth is available, restricting attention to set-matching-based measures. It recommends the centroid index (CI) on grounds of cluster-level intuitiveness and explainability, proposes the pair-set index (PSI) when a normalized score unbiased by cluster size is required, and suggests clustering accuracy (ACC) or similar set-matching measures when every point should contribute equally.

Significance. A systematic review that supplies explicit, reproducible criteria for choosing among set-matching indexes could help standardize evaluation practice in unsupervised learning. The paper's explicit scoping to set-matching measures and its emphasis on cluster-level explainability are strengths if they are accompanied by concrete comparisons that demonstrate when CI outperforms alternatives on the stated criteria.

major comments (2)
  1. [Abstract] Abstract: the recommendations for CI, PSI, and ACC are stated without any comparative table, derivation, or empirical result; the central claim that CI is preferable therefore rests on unshown analysis.
  2. [Recommendation section] Recommendation section (or equivalent): the assertion that CI is 'intuitive' and 'explainable' is presented as decisive, yet no operational definition of these properties or head-to-head evaluation against other indexes (e.g., on bias, normalization, or sensitivity to cluster-size imbalance) is supplied.
minor comments (2)
  1. All indexes discussed should be accompanied by their original citations and, where possible, the exact formulas used in the review.
  2. A short paragraph justifying the exclusion of information-theoretic or pair-counting indexes outside the set-matching family would clarify the review's scope.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address the major comments point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the recommendations for CI, PSI, and ACC are stated without any comparative table, derivation, or empirical result; the central claim that CI is preferable therefore rests on unshown analysis.

    Authors: The abstract is intentionally concise. The manuscript reviews set-matching indexes and grounds the recommendations in their structural properties (cluster-level vs. point-level evaluation, normalization behavior, and cluster-size bias). To address the concern, we will add an explicit comparative table in the revised manuscript. revision: yes

  2. Referee: [Recommendation section] Recommendation section (or equivalent): the assertion that CI is 'intuitive' and 'explainable' is presented as decisive, yet no operational definition of these properties or head-to-head evaluation against other indexes (e.g., on bias, normalization, or sensitivity to cluster-size imbalance) is supplied.

    Authors: We employ 'intuitive' and 'explainable' to indicate that CI reports mismatches at the level of individual clusters rather than an aggregate score. We agree that operational definitions and direct comparisons would strengthen the presentation. In revision we will supply definitions of these terms and a head-to-head comparison on the listed criteria. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a review of existing set-matching external validity indexes for clustering with ground truth. It presents no mathematical derivations, equations, fitted parameters, or predictions that could reduce to inputs by construction. The recommendation of centroid index (CI) rests on explicitly stated qualitative criteria (intuitiveness and explainability at cluster level), not on any self-referential fitting, self-citation chain, or ansatz. No load-bearing step matches any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a review paper that discusses existing indexes without introducing new parameters, axioms, or postulated entities.

pith-pipeline@v0.9.1-grok · 5598 in / 976 out tokens · 36424 ms · 2026-06-26T04:34:16.246662+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

  1. [1]

    Fränti and S

    P. Fränti and S. Sieranoja, Clustering accuracy, Applied Computing and Intelligence, 4 (1), 24 - 44, 2024

    2024

  2. [2]

    Kinnunen and H

    T. Kinnunen and H. Li, An overview of text-independent speaker recognition: From features to supervectors, Speech Communication, 52 (1), 12-40, 2010

    2010

  3. [3]

    Kinnunen, I

    T. Kinnunen, I. Sidoroff, M. Tuononen, and P. Fränti, Comparison of clustering methods: a case study of text -independent speaker modeling, Pattern Recognition Letters , 32 (13), 1604 -1617. October 2011

    2011

  4. [4]

    Fränti, C

    P. Fränti, C. Cariou and Q. Zhao, Cluster overlap as objective function, CMC-Computers, Materials & Continua, 66534, 1-28, 2025

    2025

  5. [5]

    Fränti, M

    P. Fränti, M. Rezaei and Q. Zhao, Centroid index: cluster level similarity measure, Pattern Recognition, 47 (9), 3034-3045, 2014

    2014

  6. [6]

    Fränti, Efficiency of random swap clustering, Journal of Big Data, 5:13, 1-29, 2018

    P. Fränti, Efficiency of random swap clustering, Journal of Big Data, 5:13, 1-29, 2018

    2018

  7. [7]

    Fränti, Genetic algorithm with deterministic crossover for vector quantization, Pattern Recognition Letters, 21 (1), 61-68, 2000

    P. Fränti, Genetic algorithm with deterministic crossover for vector quantization, Pattern Recognition Letters, 21 (1), 61-68, 2000

    2000

  8. [8]

    Likas, N

    A. Likas, N. Vlassis, J.J. Verbeek, The global k-means clustering algorithm, Pattern Recognition, 36 (2), 451-461, 2003

    2003

  9. [9]

    Fränti and S

    P. Fränti and S. Sieranoja, How much k-means can be improved by using better initialization and repeats?, Pattern Recognition, 93, 95-112, 2019

    2019

  10. [10]

    Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J

    P. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J. Comput. Appl. Math. 20, 53–65, 1987

    1987

  11. [11]

    Calinski, J

    T. Calinski, J. Harabasz, A dendrite method for cluster analysis, Commun. Stat. 3, 1–27, 1974

    1974

  12. [12]

    Zhao and P

    Q. Zhao and P. Fränti, WB -index: a sum -of-squares based index for cluster validity, Data & Knowledge Engineering, 92, 77-89, July 2014

    2014

  13. [13]

    Hämäläinen, S

    J. Hämäläinen, S . Jauhiainen, T . Kärkkäinen, Comparison of internal clustering validation indices for prototype-based clustering, Algorithms 10 (3), 105, 2017

    2017

  14. [14]

    Davies, D

    D. Davies, D. Bouldin, Cluster separation measure, IEEE Trans. Pattern Anal ysis and Machine Intelligence, 1 (2), 95–104, 1979

    1979

  15. [15]

    Niemelä, S

    M. Niemelä, S. Äyrämö, and T. Kärkkäinen, Toolbox for distance estimation and cluster validation on data with missing values. IEEE Access, 10, pp.352-367, 2021

    2021

  16. [16]

    Bagirov, R.M

    A.M. Bagirov, R.M. Aliguliyev, N. Sultanova, Finding compact and well -separated clusters: Clustering using silhouette coefficients, Pattern Recognition, 135, 109144, March 2023

    2023

  17. [17]

    Ikotun, F

    A.M. Ikotun, F. Habyarimana, A.E. Ezugwu, Cluster validity indices for automatic clustering: A comprehensive review, Heliyon, 11 (2), e41953, January 2025

    2025

  18. [18]

    K -means properties on six clustering benchmark datasets

    P. Fränti and S. Sieranoja, "K -means properties on six clustering benchmark datasets", Applied Intelligence, 48 (12), 4743-4759, December 2018

    2018

  19. [19]

    Hubert and P

    L. Hubert and P. Arabie, Comparing partitions. Journal of Classification, 2 (1), 193-218 (1985)

    1985

  20. [20]

    Kvålseth, Entropy and correlation: some comments, IEEE Trans

    T.O. Kvålseth, Entropy and correlation: some comments, IEEE Trans. Syst ems, Man and Cybernetic, 17 (3), 517-519, 1987

    1987

  21. [21]

    Rand, Objective criteria for the evaluation of clustering methods

    W.M. Rand, Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association 66: 846–850, 1971

    1971

  22. [22]

    Warrens and H

    M.J. Warrens and H. van der Hoef, Understanding the Rand index. Advanced Studies in Classification and Data Science, pp. 301-313, Springer, Singapore, 2020

    2020

  23. [23]

    Warrens and H

    M.J. Warrens and H. van der Hoef, Understanding the adjusted Rand index and other partition comparison indices based on counting object pairs. Journal of Classification, 39, 487–509, 2022

    2022

  24. [24]

    Rezaei and P

    M. Rezaei and P. Fränti, Set matching measures for external cluster validity, IEEE Trans. on Knowledge and Data Engineering, 28 (8), 2173-2186, August 2016

    2016

  25. [25]

    van Dongen, Performance criteria for graph clustering and Markov cluster experiments, Technical Report INSR0012, Centrum voor Wiskunde en Informatica, 2000

    S. van Dongen, Performance criteria for graph clustering and Markov cluster experiments, Technical Report INSR0012, Centrum voor Wiskunde en Informatica, 2000

    2000

  26. [26]

    Meila and D

    M. Meila and D. Heckerman, An experimental comparison of model based clustering methods, Machine Learning, 41(1-2), pp. 9–29, 2001

    2001

  27. [27]

    D. Cai, X. He, J. Han, Document clustering using locality preserving indexing, IEEE Trans. Knowledge and Data Engineering, 17 (12), 2005, 1624-1637

    2005

  28. [28]

    Rendón, I

    E. Rendón, I. Abundez, A. Arizmendi, and E.M. Quiroz, Internal versus external cluster validation indexes. Int. Journal of computers and communications, 5 (1), 27-34, 2011

    2011

  29. [29]

    de Souto, A.L.V

    M.C.P. de Souto, A.L.V. Coelho, K. Faceli, T.C. Sakata, V. Bonadia, and I.G. Costa, A comparison of external clustering evaluation indices in the context of imbalanced data sets, 2012 Brazilian Symposium on Neural Networks, pp. 49-54, 2012

    2012

  30. [30]

    Zhao and P

    Q. Zhao and P. Fränti, Centroid ratio for pairwise random swap clustering algorithm, IEEE Trans. on Knowledge and Data Engineering, 26 (5), 1090-1101, May 2014

    2014

  31. [31]

    Fränti and M

    P. Fränti and M. Rezaei, Generalized centroid index to different clustering models, S+SSPR’16, Merida, Mexico, LNCS 10029, 285-296, November 2016

    2016

  32. [32]

    Slonim and N

    N. Slonim and N. Tishby, Document clustering using word clusters via the information bottleneck method. Int. ACM SIGIR Conf. on Research and development in information retrieval, 208-215, 2000

    2000

  33. [33]

    Kuhn, The Hungarian method for the assignment problem, Naval Res

    H.W. Kuhn, The Hungarian method for the assignment problem, Naval Res. Logist. Quart., 2 (1- 2), 83–97, 1955

    1955

  34. [34]

    Munkres, Algorithms for the assignment and transportation problems, J

    J. Munkres, Algorithms for the assignment and transportation problems, J. Soc. Ind. Appl. Math., 5 (1), 32–38, 1957

    1957

  35. [35]

    Jonker and A

    R. Jonker and A. Volgenant, A shortest augmenting path algorithm for dense and sparse linear assignment problems, Computing, 38 (4): 325–340, December 1987. doi:10.1007/BF02278710. S2CID 7806079

    work page doi:10.1007/bf02278710 1987

  36. [36]

    Mahmoudi and D

    A. Mahmoudi and D. Jemielniak, Proof of biased behavior of normalized mutual information, Scientific Reports, 14:9021, 1-17, 2024

    2024

  37. [37]

    Rezaei and P

    M. Rezaei and P. Fränti, Can the number of clusters be determined by external indices?, IEEE Access, 8 (1), 89239-89257, December 2020

    2020