Pith. sign in

REVIEW 4 major objections 4 minor 12 references

ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the best LLM-based proxy annotators are statistically indistinguishable from human crowdworkers at judging whether documents fit a topic or cluster, and can substitute for them in automated evaluations.

desk verdict A well-executed protocol for human-grounded topic model evaluation, but the 'statistically indistinguishable' claim is overstated due to an alt-test scale mismatch. read the letter →

arxiv 2507.00828 v1 pith:XX3NXC3X submitted 2025-07-01 cs.CL

Alexander Hoyle , Lorena Calvo-Bartolomé , Jordan Boyd-Graber , Philip Resnik This is my paper · ORCID
classification cs.CL
keywords topicmodelsdocumentclusteringLLM-as-a-judgealternativeannotatortestqualitativecontentanalysismodelevaluationcoherencecrowdsourcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Topic model and document-clustering evaluations usually fall into two camps: automated coherence metrics such as NPMI that correlate poorly with human judgment, and human evaluation that requires expert labels and does not scale. This paper proposes a protocol that mirrors how practitioners actually use these models, namely qualitative content analysis: an annotator reads a sample of a topic's top documents and keywords, writes a category label, and then scores new documents by how well they fit that category. The same three steps are turned into an LLM prompt, producing a 'proxy annotator' called PROXANN. Using crowdworker annotations on two datasets and six LLMs, the paper finds that the largest models—GPT-4o, Llama-3.3-70B, Qwen-3-32B, and Qwen-2.5-72B—are statistically indistinguishable from a randomly chosen human annotator on document-level fit and ranking judgments. If correct, this gives researchers a scalable, application-grounded way to evaluate topic models and clusters without expert labels, and it also indicates that the MALLET implementation of LDA remains at least as competitive as modern neural and clustering approaches.

What carries the argument

The load-bearing machinery is the three-step evaluation protocol itself, designed to emulate qualitative content analysis. In the Label Step an annotator reads seven exemplar documents and fifteen top keywords and writes a free-text category label; in the Fit Step they score seven additional documents from 1 to 5 by how well each fits that label; in the Rank Step they order those documents by representativeness. For the LLM version, Fit scores are computed as a probability-weighted mean over the Likert-scale tokens, and Rank judgments come from pairwise comparisons aggregated through a Bradley–Terry model. Substitutability is then established with the Alternative Annotator Test, a leave-one-out procedure that asks whether the LLM agrees with the rest of the annotator pool more often than a randomly held-out human annotator does.

What would settle it

A direct comparison in which expert qualitative researchers and the best LLM proxies annotate the same topics would settle the claim: if expert–expert agreement is high but expert–LLM agreement is no better than chance, then the proxy has been validated against the wrong gold standard for the intended use case.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the evaluation task itself can be made the proxy: instead of asking whether a topic's top words are coherent, it asks whether a reader—human or LLM—can infer a category from the topic's exemplar documents and keywords, and then apply that category to held-out documents. On this task, the best LLM proxies reach advantage probabilities $\rho > 0.5$ against crowdworker annotators on both datasets, with statistically significant wins under the Alternative Annotator Test at the document level; human–human agreement on well-defined topics is high, and a synthetic label-derived upper bound achieves Krippendorff's $\alpha \ge 0.8$, indicating that the tasks are well specified. The paper also finds that NPMI, the standard automated coherence metric, is not correlated with the human judgments, while PROXANN-based metrics rank topics and models about as well as a held-out human annotator does. On the model-comparison side, the human evaluations show MALLET (classical LDA) performing at least as well as CTM and BERTopic on both Wiki and Bills.

Load-bearing premise

The evaluation assumes that non-expert crowdworkers, screened by attention checks and bonuses, produce the same categories and fit judgments that real qualitative content analysts would, so that a proxy validated against them transfers to actual research practice.

Editorial extensions

If this is right

  • Researchers can substitute a large LLM for a panel of crowdworkers in the fit and ranking steps, making use-oriented topic and cluster evaluation cheap enough to run repeatedly during model development.
  • Model comparisons that currently rely on NPMI coherence should be re-examined, since NPMI shows no correlation with human judgments in this study while PROXANN-derived metrics do.
  • The protocol applies to any method that outputs document-topic scores and top words, including clustering approaches such as BERTopic, not just generative topic models.
  • Classical LDA (MALLET) remains a strong baseline: on the human judgments it performs at least as well as CTM and BERTopic, so new models should be required to beat it on use-oriented tasks rather than only on coherence.
  • LLM-based topic rankings agree with human rankings roughly as well as one human annotator agrees with another, so automated evaluation can support development decisions even when individual topic-level comparisons are noisy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same protocol could be run in reverse to generate multiple category interpretations from a single LLM and use their disagreement as a cheap coherence signal, an idea the paper leaves to future work.
  • Editorial inference: because the study is English-only and uses Wikipedia and U.S. Congressional bills, the substitutability result should be expected to weaken on non-English or highly specialized corpora until separately validated.
  • Editorial inference: the LLM's category label shapes all downstream fit scores, so automated evaluation inherits the model's granularity preferences; a label-level agreement check, not just document-fit agreement, would be a natural extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PROXANN, a use-oriented evaluation protocol for topic models and document clustering in which annotators infer a category from exemplar documents and keywords, then judge whether held-out documents fit that category (Fit Step) and rank their representativeness (Rank Step). The same protocol is instantiated as LLM prompts, and the LLM outputs are compared against crowdworker annotations collected on two datasets (Wiki and Bills) across three models (MALLET, CTM, BERTopic). The central claim is that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore substitute for human evaluation. The paper also reports that the classical MALLET model remains competitive, that NPMI does not align with human judgments, and that PROXANN-based metrics can rank topics comparably to a leave-one-out human baseline. The authors release the annotation data, package, and web interface.

Significance. If the central claim survives scrutiny, the contribution is valuable: it would provide a scalable, off-the-shelf evaluation of topic models and document clusters that is anchored in an actual use case rather than in unsupervised coherence metrics. The empirical design is careful in several respects: multiple annotators per topic, attention checks and training exercises, a label-derived upper-bound pilot, bootstrap confidence intervals, an alternative annotator test, and publicly released data and code. The paper also makes a useful methodological point that NPMI is poorly aligned with human judgments. The main risk is that the headline substitutability claim rests on a comparison metric that may be biased in favor of the LLM because of scale granularity, and the abstract's 'statistically indistinguishable' wording is stronger than the superiority tests actually reported.

major comments (4)
  1. [§3.1, §4.4, Table 3, Table 8] The alternative annotator test is not symmetric between human and LLM response scales. In the Fit Step, human annotators provide integer scores on a 1–5 Likert scale, while PROXANN produces continuous probability-weighted means (e.g., GPT-4o Fit=4.77 in Table 8). The alt-test in §4.4 uses RMSE against the leave-one-out mean of the other annotators. For a given document, a continuous LLM score can always be placed arbitrarily close to that mean, whereas an integer human score carries unavoidable quantization error. This means that even an LLM with no better underlying relevance judgment than a random human will tend to win more binary RMSE comparisons, inflating ρ and the significance stars in Table 3. The slack term ε=0.1 cannot neutralize a systematic granularity advantage of up to 0.5 in score units. The authors should round PROXANN scores to the integer scale before computing RMSE, or use an ordinal/tie-aware agreement measure, and report whether the document-level substitutability claim survives after this alignment.
  2. [Abstract, §4.4, Table 3] The phrase 'statistically indistinguishable from a human annotator' does not match the statistical test performed. The alt-test as described is a one-sided test of whether the LLM wins significantly more than half of the comparisons against human annotators, and Table 3 reports ρ values such as 0.65–0.71 with significance markers. This is evidence of superiority under RMSE, not of statistical indistinguishability in the sense of an equivalence test. The authors should either run a two-one-sided equivalence test with a pre-specified margin on the RMSE difference between LLM and human, or rephrase the claim as 'as good as or better than a randomly chosen human annotator' in the abstract and conclusions.
  3. [§4.3, App. D, App. I] The pilot study on six Wiki topics was used to tune the LLM prompts (App. D and App. I), while the main Wiki evaluation samples eight topics per model from the same corpus (§4.3). The paper does not state whether the six pilot topics were excluded from the main evaluation sample. If any of these topics overlap with the main eight, the Wiki PROXANN results in Fig. 3 and Table 3 could be optimistically biased by prompt tuning on those very topics. Please report the overlap explicitly, and if the pilot topics were not excluded, run a sensitivity analysis that removes them or otherwise quantifies the impact on the Wiki results.
  4. [Limitations, §1, §2.1] The Limitations section concedes that 'the annotators are not experts pursuing a research question.' Given the paper's use-oriented framing and its motivation from qualitative content analysis, the abstract's claim that PROXANN can 'serve as a reasonable substitute in automated evaluations' is established only for crowdworker-style relevance judgments. The manuscript should either add expert validation or explicitly scope the claim in the abstract and title to non-expert human judgments, since the current wording suggests applicability to expert content-analytic practice that the evidence does not yet support.
minor comments (4)
  1. [§7 (Conclusion)] The sentence 'The quality of models is determined their ability to meet real-world needs' is missing 'by' and should read 'determined by their ability.'
  2. [§5.4] The statement 'there is some capacity for PROXANN to accurately rank topics at least as well as an arbitrary annotator' is stronger than Table 4 supports, since most correlations are below 0.5 and the confidence intervals are wide. Consider softening to 'comparable to leave-one-out human agreement levels.'
  3. [§4.4] The description of the alt-test is slightly compressed: it is not immediately clear whether the slack term ε is added to the LLM's similarity or subtracted from the human's, and how the topic-level RMSE is aggregated over the seven documents. A displayed formula for the binary outcome would remove this ambiguity.
  4. [Table 4 footnote] The table reports bootstrapped means and standard deviations, but not the number of topics used for each correlation; since only eight topics per model are evaluated, the effective sample size is small and this should be stated in the caption.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: PROXANN is validated empirically against held-out human annotations, and no predicted quantity is fitted into the proxy by construction.

full rationale

The central claim, that the best LLM proxies are statistically indistinguishable from a human annotator, is supported by direct empirical comparisons between PROXANN responses and crowdworker annotations (Section 4.4, Table 3), not by a derived quantity that equals its own input. The Fit Step uses a probability-weighted mean over Likert tokens following Wang et al. (2025), and the Rank Step uses pairwise comparisons with a Bradley-Terry model; neither is optimized against the human labels used in the alt-test. The only tuning step is prompt engineering on pilot Wiki annotations, which is disclosed and does not determine the Bills results, where the substitutability finding is actually stronger. Self-citations to prior work, such as Hoyle et al. (2021, 2022), provide datasets, models, and framing, but the load-bearing validation is self-contained: human-human agreement, PROXANN-human agreement, and topic-ranking correlations are all computed from data collected in this paper. The Limitations section's concession that annotators are not experts is a gold-standard validity concern, not circularity. A possible asymmetry between continuous LLM scores and integer human scores in the RMSE-based alt-test is a measurement-validity risk and is better handled as a correctness concern, not as a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical evaluation rather than a derivation, so it has no fitted constants in a mathematical model. The main burdens on the reader are hand-chosen protocol parameters (sample sizes, thresholds, alt-test slack) and domain assumptions connecting crowdworker agreement to real-world validity. No new physical or conceptual entities are introduced.

free parameters (3)
  • Sample sizes for protocol (nd=7 exemplar documents, nw=15 keywords, 7 evaluation documents) = 7 documents, 15 keywords
    Set by hand in Section 3.1 to balance annotator burden and informativeness. These choices affect the noise level of measured agreement but not the ranking of models in a way the paper justifies.
  • Alt-test slack epsilon = 0.1
    Adopted from Calderon et al. (2025) in Section 4.4; the threshold for declaring LLM wins over a human annotator. Changing epsilon changes which models pass the substitutability test.
  • Exemplar sampling threshold t_k (elbow detection) = Per-topic threshold from knee-point algorithm
    Appendix A: documents with theta above an automatically detected elbow are used to sample exemplars. The threshold changes which documents annotators see and thus could influence category labels and fit scores.
assumptions (4)
  • domain assumption The primary real-world use of topic models and document clustering is qualitative content analysis, where analysts induce categories from text and apply them to new documents.
    Sections 2.1 and 3 frame the protocol around this use case; if the real use case differs, the evaluation may measure the wrong quantity.
  • domain assumption A topic or cluster is of good quality if a reader can infer a coherent category from its top documents and words and then consistently apply that category to held-out documents.
    Section 2.2 and Fig. 1; this operationalizes validity via inter-subjective agreement rather than external truth.
  • domain assumption Crowdworker judgments, after training and attention filtering, are a valid proxy for expert content-analyst judgments.
    Used throughout; the Limitations section explicitly notes annotators are not experts. The pilot label-derived clusters show task intelligibility, not expert equivalence.
  • standard math Standard statistical tools (Kendall's tau, Krippendorff's alpha, Bradley-Terry/ILSR, bootstrap) correctly estimate agreement and uncertainty for this data.
    Section 4.4 and Appendix I.2; these are standard methods, but small sample sizes (8 topics, 7 documents) make confidence intervals wide.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering." pith.science (2026). https://pith.science/paper/XX3NXC3X

@misc{pith2026250700828,
  author       = {Pith},
  title        = {Pith review of: ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XX3NXC3X}},
  note         = {Machine review of arXiv:2507.00828}
}
read the original abstract

Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann

Figures

Figures reproduced from arXiv: 2507.00828 by the authors.

Figure 1
Figure 1. Our evaluation protocol for topic models and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Annotators review the top documents and words from a single topic and infer a category (Label Step), then [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Correlations between PROXANN and human annotations (Kendall’s τ ) for the relevance judgment (Fit Step) and representativeness ranking (Rank Step) tasks, averaged over topics (pooled over all three models). While GPT-4o has the best overall correlations and relatively low variance, the Qwen family is a reasonable substitute, even at smaller sizes. Dashed horizontal lines are the average leave-one-out human–human cor… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Distribution of the top 1,000 theta values across [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Metrics quantifying the relationship between human annotations and estimated document-topic probabilities [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Metrics quantifying the relationship between human relevance judgments and estimated document-topic [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Distributions of leave-one-out inter-annotator correlations (Kendall’s [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Metrics quantifying the relationship between human relevance judgments and estimated document-topic [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Model rankings based on FIT–τ h:tm/ RANK–τ h:tm (correlation of human scores with document-topic probabilities), FIT–τ lm:tm/RANK–τ lm:tm (correlation of LLM scores with document-topic probabilities), and NPMI coherence. Correlations are computed using Kendall’s τ . Er…
Figure 10
Figure 10. Figure 10: Instructions for the human annotation protocol. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Label Step. Category identification in the human annotation protocol for the practice question. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Fit Step. Relevance judgment in the human annotation protocol for the practice question. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Rank Step. Representativeness ranking in the human annotation protocol for the practice question. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Consent page (shown at beginning) [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    Examine the Keywords: Use the keywords as clues to identify the general subject area or themes present in the documents

    work page

  2. [2]

    Review the Documents: Skim the summaries provided to understand their main ideas and any recurring elements

    work page

  3. [3]

    Applications of topic models. Found. Trends Inf. Retr., 11(2–3):143–296. Jordan L. Boyd-Graber, David Mimno, and David New- man. 2014. Care and feeding of topic models. In Handbook of Mixed Membership Models and Their Applications. Ralph Allan Bradley and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons...

    arXiv 2014

  4. [7]

    In Findings of the Association for Computational Lin- guistics: EACL 2024, pages 1760–1773, St

    Contextualized topic coherence metrics. In Findings of the Association for Computational Lin- guistics: EACL 2024, pages 1760–1773, St. Julian’s, Malta. Association for Computational Linguistics. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Metho...

    work page 2024

  5. [12]

    Generate a Label: Based on the keywords and document content, come up with a single label that best describes the topic connecting all the documents. Examples: -------- {} ######### KEYWORDS: {} DOCUMENTS: {} Based on the keywords and document content, come up with a single category that best describes the topic connecting all the documents. Return just t...

    work page

  6. [2003]

    Journal of machine Learning research, 3(Jan):993–1022

    Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. Jordan Boyd-Graber, Yuening Hu, and David Mimno

    work page

  7. [2014]

    In Proceedings of the 14th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 530–539, Gothenburg, Sweden

    Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 530–539, Gothenburg, Sweden. Association for Computational Linguistics. Zongxia Li, Lorena Calvo-Bartolomé, Alexander Hoyle, Paiheng Xu, Alden ...

    arXiv 2025

  8. [2017]

    In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada

    An automatic approach for document-level topic model evaluation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada. Association for Computational Linguistics. Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a hot topic: Contextualized document embeddings ...

    work page 2017

Show all 12 references
  1. [2020]

    Association for Computa- tional Linguistics

    Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736, Online. Association for Computa- tional Linguistics. Charles P S...

    2020

  2. [2023]

    PLOS ONE, 18(9):1–24

    Donor activity is associated with us legislators’ attention to political issues. PLOS ONE, 18(9):1–24. Justin Grimmer and Brandon Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analy- sis, 21:267 – 297...

    2013 arXiv

  3. [2024]

    Preprint, arXiv:2306.00978

    Awq: Activation-aware weight quantization for llm compression and acceleration. Preprint, arXiv:2306.00978. Philipp Mayring. 2000. Qualitative inhaltsanalyse. Fo- rum Qualitative Sozialforschung / Forum: Qualita- tive Social Research, 1(2). Stephen Merity, Caiming Xiong, James...

    2000 arXiv

  4. [2025]

    banning AR-15s

    Improving llm-as-a-judge inference with the judgment distribution. Preprint, arXiv:2503.03064. Shu Xu and Michael F. Lorber. 2014. Interrater agree- ment statistics with skewed data: evaluation of alter- natives to cohen’s kappa. Journal of consulting and clinical psychology, ...

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.