Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation

Jennifer D'Souza; Zhiyin Tan

REVIEW 5 major objections 5 minor 60 references

Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models, guided by tailored prompts, can evaluate topic model quality on four dimensions—coherence, repetitiveness, diversity, and topic-document alignment—well enough to replace human annotators and narrow statistical…

desk verdict Useful multi-dimensional LLM evaluation framework for topic models with genuinely new mismatch metrics, but no human validation means the central claim about measuring topic quality is not yet supported. read the letter →

arxiv 2502.07352 v1 pith:OXWLT4HL submitted 2025-02-11 cs.CL cs.AIcs.DL

Zhiyin Tan , Jennifer D'Souza This is my paper · ORCID

classification cs.CLcs.AIcs.DL

keywords topicmodelevaluationlargelanguagemodelsLLM-as-evaluatorcoherencediversitytopic-documentalignmentadversarialtestingrepetitivenessdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models, guided by purpose-built prompts, can evaluate topic model output on four quality dimensions—coherence, repetitiveness, diversity, and topic-document alignment—well enough to replace human annotators and narrow statistical metrics. For each dimension it pairs a quick 3-point rating prompt with an explanatory detection prompt (outlier words, duplicate concept pairs, extraneous or missing document themes) and an adversarial test that injects a known intruder to check whether the evaluator catches it. Experiments with three open LLMs on four topic models over two corpora produce interpretable, model-specific results, such as BERTopic scoring high on coherence while also being flagged for redundant words. If the claim holds, digital-library scale taxonomies can be audited automatically and dynamically without manual annotation.

What carries the argument

The load-bearing mechanism is the prompt-conditioned LLM judgment: a family of zero-shot prompts that turn each quality dimension into a rating plus an explainable list. The rating prompts fix scoring conventions (e.g., 1 = unrelated or overlapping, 3 = coherent or distinctive); the detection prompts force the model to output explicit evidence; the adversarial tests measure whether the model can spot an injected anomaly. The framework's claims ride on these prompts transferring across datasets and topic models, and the detection outputs provide the qualitative material for case studies.

What would settle it

Have human annotators rate and flag the same 100 topics and 100 topic-document pairs from 20NG and AGRIS that the three LLMs evaluated, then compute rank correlation between each LLM metric and the human scores; the central claim fails if the correlations are near zero or if adversarial duplicate-detection rates for the chosen evaluator stay near chance.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an LLM with a tailored prompt can serve as a unified multi-dimensional evaluator for topic models, and that this works across model families and datasets enough to rank topic models and expose their failure modes. The coherence dimension uses $C_{rate}$ (semantic consistency on a 1–3 scale) and $C_{outlier}$ (words flagged in at least 3 of 5 iterations); repetitiveness uses $R_{rate}$ and $R_{duplicate}$; diversity uses pairwise $D_{rate}$; alignment uses $A_{ir\text{-}topic}$ and $A_{missing\text{-}theme}$, counts of irrelevant or absent themes averaged over topic-document pairs. Adversarial tests insert an unrelated word (for outliers) or a chosen duplicate (for redundancy) and measure the detection rate; on 20NG and AGRIS the pass rates vary widely across evaluators, e.g., 29% to 93%, which the paper reads as evidence that multiple LLMs should be used. The authors conclude that the framework provides rich, context-sensitive insight into topic quality and that evaluator-specific biases are themselves a finding for downstream users.

Load-bearing premise

The load-bearing premise is that the LLM's judgments are valid proxies for human judgments of topic quality without any calibration against human ratings, so if LLM preferences diverge from human perception the framework measures something other than topic quality.

Editorial extensions

If this is right

Topic model comparisons can be made on multiple axes at once, with scores that carry textual justification rather than a bare number.
A model's high coherence can be cross-checked against repetitiveness; the paper's data suggest BERTopic's coherence is partly inflated by redundant top words, so coherence alone can mislead.
Adversarial pass rates give a per-evaluator, per-dimension trust check, and low rates on duplicate detection for some LLMs (as low as 29% on AGRIS) argue for running several evaluators.
Topic-document alignment metrics quantify both over- and under-representation of themes, which is directly relevant to search, recommendation, summarization, and classification over scholarly text.
The framework scales to 50 and 100 topics in minutes to a few hours on a single A100 GPU, making repeated or dynamic evaluation feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the LLM scores were calibrated against human ratings, the same prompt suite could become a continuous monitoring tool for live taxonomies, with detection lists serving as actionable repair signals.
The mismatch between $D_{unique}$ (lexical uniqueness) and $D_{rate}$ (semantic distinctiveness) hints that LLM ratings could expose cases where clustering-based models are semantically diverse despite lexical overlap; this is a testable hypothesis beyond the paper.
Adversarial pass rates could be used prospectively as a gate: downstream applications could require a minimum detection rate per evaluator and dimension before trusting its scores, a practice the paper motivates but does not prescribe.
A natural extension is to use the LLM's own flagged outliers, duplicates, and missing themes to drive automatic topic refinement, turning evaluation into part of the topic-modeling loop.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth reading if you work on topic model evaluation. It pulls coherence, repetitiveness, diversity, and topic-document alignment into one LLM-based framework, with adversarial tests for the two detection-style metrics. The new metrics A_ir-topic and A_missing-theme are a real contribution: they directly quantify over- and under-coverage of document themes, which standard metrics like C_v and D_unique miss. The prompts are concrete and reusable, the code is public, and the qualitative analysis of evaluator differences is refreshingly honest.

The soft spot is exactly where the stress-test note lands. The only validity evidence is the adversarial tests, and those already show one evaluator failing duplicate detection most of the time: Mistral succeeds in 37% of cases on 20NG and 29% on AGRIS. The three LLMs also disagree substantially on nearly every metric and on model rankings. The paper itself says there is "significant variability among LLM evaluators," but then the central claim remains that these scores reflect topic quality rather than evaluator-specific bias. Without any correlation to human judgments, that claim is unproven. The ten runs per configuration are mentioned, but no error bars or significance tests appear; that matters because the differences between models are often small relative to the evaluator disagreement. The normalization in Section 5.2 rescales each evaluator group internally, which can hide systematic differences and make Mistral, Llama, and Qwen look more interchangeable than they are.

That said, the paper is not sloppy. The experimental design is clearly described, the adversarial tests are a reasonable first step, and the authors explicitly acknowledge the variability rather than sweeping it under the rug. This is a solid proposal that needs stronger validation before it can be used as a drop-in replacement for human evaluation.

For whom: researchers building automated evaluation tools for topic modeling and digital libraries. I would not cite it as evidence that LLM evaluation works, but I would cite it as a framework worth testing and extending.

Recommendation: send it to peer review. The right outcome is conditional acceptance requiring human-judgment correlation studies, significance testing, and a clearer account of how to handle evaluator disagreement.

Referee Report

5 major / 5 minor

Summary. The paper presents an LLM-based evaluation framework for topic models, defining eight prompt-based metrics that cover coherence (rating and outlier detection), repetitiveness (rating and duplicate detection), diversity (pairwise rating), and topic-document alignment (irrelevant words and missing themes). The framework is evaluated on 20NG and AGRIS with four topic models (LDA, ProdLDA, CombinedTM, BERTopic), three open LLM evaluators (Mistral-7B, Llama-3.1-8B, Qwen2.5-14B), and K=50/100 topics, with automated baselines C_v and D_unique. Adversarial tests are used to check outlier and duplicate detection, and the paper reports evaluator-specific patterns, normalization for visualization, and qualitative case studies. The authors conclude that LLMs can provide robust, scalable, multi-dimensional topic-model evaluation.

Significance. If the framework's scores were shown to track human judgments, it would be a valuable low-cost complement to human annotation, especially the interpretable outputs (flagged outliers, duplicate pairs, missing themes). The paper is clearly written, the prompts are concrete, the code is released, and the adversarial protocol is a useful internal check. However, the evidence currently supports the framework only as a descriptive account of what three LLMs say about topics; it does not establish that the scores measure topic quality, because no human-rated validity check is reported and the evaluators disagree on several key rankings.

major comments (5)

[Section 5.1; Tables 1-7] No evidence connects the LLM scores to human perceptions of topic quality, which is the load-bearing claim of the paper. The adversarial tests in Section 5.1 validate only the detection subtasks (outlier and duplicate identification), not the rating scales C_rate, R_rate, D_rate, or the alignment counts A_ir-topic and A_missing-theme, and adversarial success does not imply that the numeric scores will rank topic models the way human raters would. The paper should report correlations (e.g., Spearman) between each LLM metric and human ratings on a sample, or provide another external criterion; without such calibration the framework measures LLM outputs rather than topic quality.
[Section 5.1; Tables 3-4] The reported adversarial pass rates undermine the reliability of R_duplicate for one of the three evaluators, yet Tables 3 and 4 still present Mistral's R_duplicate as an interpretable result. Section 5.1 reports Mistral success rates of only 37% on 20NG and 29% on AGRIS for duplicate detection, meaning Mistral fails the majority of these checks; the paper later calls this 'significant variability' and moves on. The framework needs a defined policy: either exclude or down-weight evaluators that fail adversarial tests at a specified threshold, report per-evaluator confidence, or show that a fixed aggregation rule (e.g., majority or median across evaluators) reproduces the conclusions for all three evaluators.
[Section 4.2; Tables 1-7] The experimental tables report point estimates without variance or significance tests, despite Section 4.2 stating that ten runs were performed for each configuration. Many qualitative conclusions are based on small differences or on one evaluator's ranking; for example, Table 2 (AGRIS, K=50, LDA row) gives C_outlier = 0.778 for Mistral, 1.32 for Llama, and 2.436 for Qwen, a threefold spread. The central comparisons should be accompanied by confidence intervals or pairwise significance tests, and the 'consistently' language in Sections 5.1 and 5.3 should be restricted to differences that survive evaluator variation.
[Section 5.2] The piecewise normalization in Section 5.2 rescales each metric within each evaluator group, so it removes systematic absolute differences between Mistral, Llama, and Qwen (e.g., Qwen's consistently higher C_outlier counts are compressed into the same range as Mistral's). This makes the three evaluators appear more aligned than they are and weakens the robustness claim. The authors should justify the normalization, report raw scores alongside normalized scores in the same figures or tables, and show how ranking-based conclusions change across evaluators without normalization.
[Section 4.3; Tables 6-7] The sampling of topic-document pairs for A_ir-topic and A_missing-theme is not specified precisely enough to be reproducible. Section 4.1 says a single document can be assigned multiple topics, but Section 4.3 does not state the assignment rule (e.g., threshold on posterior probability, top-1 assignment, or the clustering assignment for BERTopic) nor how the 'up to 100 associated documents per topic' were selected. The values in Tables 6 and 7 therefore cannot be independently reconstructed; please define the rule and state whether documents were deduplicated across topics.

minor comments (5)

[Tables 5-7] The layout of Tables 5-7 combines header and data rows into long unbroken lines, making the columns difficult to read; they should be reformatted into readable columns.
[Section 4.1] The AGRIS preprocessing sentence is missing a closing parenthesis and contains an ungrammatical clause ('duplicate records (by DOI, and named it AGRIS'); please rewrite.
[Section 5.2] The normalization is undefined when X_max - X_min = 0, which occurs in practice because Mistral's R_rate is 2.000 for nearly every entry in Tables 3 and 4; please state the fallback for degenerate cases.
[References] References [10] and [37] are the same Stammbach et al. work and should be merged and renumbered throughout the text.
[Section 4.2] The paper says parameters were tuned to optimize C_v and D_unique, but it does not report the chosen hyperparameter values for LDA, ProdLDA, CombinedTM, and BERTopic; include the settings or a pointer to a configuration file for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM-based metrics are operationalized outputs, not derived from a fitted target, and the adversarial tests provide independent checks.

full rationale

The paper does not claim to derive a prediction from first principles; it proposes a framework in which LLM outputs, elicited by the stated prompts, are the evaluation metrics. There is no fitted parameter that is later renamed as a prediction, and no equation defines one quantity in terms of the very quantity it claims to predict. The adversarial tests in Section 5.1 inject known anomalies (e.g., an unrelated word or a manually inserted duplicate) and measure detection success, which is an independent validation signal rather than a circular reduction; the low Mistral duplicate-detection rates are a valid empirical weakness but not a circularity. The only author self-citation in the text is reference [9], a catalog of transformer models cited as background for LLM suitability, and it is not load-bearing for any metric or derivation. The absence of human-judgment correlation is a substantive validity concern about whether the metrics measure topic quality, but that is a correctness/validity issue, not a circularity issue under the criteria used here.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The framework adds no fitted constants, but its validity rests on treating LLM outputs as ground truth and on the ad hoc 3-of-5 outlier threshold. No external human benchmark is used to calibrate or confirm the metrics.

free parameters (1)

Outlier detection threshold = 3 out of 5 LLM iterations
A topic word is counted as an outlier if it is flagged in at least 3 of 5 repeated LLM calls. This cutoff is introduced in Section 3.1 without tuning or a stability analysis.

assumptions (3)

domain assumption LLM semantic judgments are reliable proxies for human topic-quality judgments
All proposed metrics return the LLM's ratings and flags, and the paper does not calibrate those outputs against human annotations or an independent ground truth.
domain assumption Passing adversarial tests indicates real-world metric reliability
The paper validates outlier and duplicate detection only through synthetic injections such as adding 'Shakespeare', and it assumes this performance transfers to unmodified topic lists.
ad hoc to paper The 3-of-5 aggregation rule produces stable outlier identification
Section 3.1 chooses this rule without showing that the results are robust to other thresholds or that five iterations are enough.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation." pith.science (2026). https://pith.science/paper/OXWLT4HL

@misc{pith2026250207352,
  author       = {Pith},
  title        = {Pith review of: Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXWLT4HL}},
  note         = {Machine review of arXiv:2502.07352}
}

read the original abstract

This study presents a framework for automated evaluation of dynamically evolving topic taxonomies in scientific literature using Large Language Models (LLMs). In digital library systems, topic modeling plays a crucial role in efficiently organizing and retrieving scholarly content, guiding researchers through complex knowledge landscapes. As research domains proliferate and shift, traditional human centric and static evaluation methods struggle to maintain relevance. The proposed approach harnesses LLMs to measure key quality dimensions, such as coherence, repetitiveness, diversity, and topic-document alignment, without heavy reliance on expert annotators or narrow statistical metrics. Tailored prompts guide LLM assessments, ensuring consistent and interpretable evaluations across various datasets and modeling techniques. Experiments on benchmark corpora demonstrate the method's robustness, scalability, and adaptability, underscoring its value as a more holistic and dynamic alternative to conventional evaluation strategies.

Figures

Figures reproduced from arXiv: 2502.07352 by the authors.

**Figure 2.** Radar plot comparing the evaluation trends of three LLMs in the results (𝐾 = 50) of the AGRIS themes 𝐴missing-theme than Mistral and Llama. Moreover, all evaluators report higher missing theme counts for 20NG than for AGRIS, implying that 20NG documents exhibit greater thematic diversity and complexity. These insights underscore the influence of evaluator-specific biases on metric outcomes and highlight the importan… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 43 canonical work pages

[1]

Waltman, N

L. Waltman, N. J. Van Eck, A new methodology for constructing a publication-level classification system of science, Journal of the American Society for Information Science and Technology 63 (2012) 2378–2392

work page 2012
[2]

M. F. Moura, R. M. Marcacini, B. M. Nogueira, M. d. S. CONRADO, S. O. Rezende, A proposal for building domain topic taxonomies., In: WORKSHOP ON WEB AND TEXT INTELLIGENCE, 1.; SIMPÓSIO BRASILEIRO EM . . . , 2008

work page 2008
[3]

Dauxais, U

Y. Dauxais, U. Zaratiana, M. Laneuville, S. D. Hernandez, P. Holat, C. Grosman, Towards automation of topic taxonomy construction, in: International Symposium on Intelligent Data Analysis, Springer, 2022, pp. 26–38

work page 2022
[4]

Kotitsas, D

S. Kotitsas, D. Pappas, N. Manola, H. Papageorgiou, Scinobo: a novel system classifying scholarly communication in a dynamically constructed hierarchical field-of-science taxonomy, Frontiers in Research Metrics and Analytics 8 (2023) 1149834

work page 2023
[5]

M. H. Bhat, S. M. Shafi, et al., Taxonomies in knowledge organisation—need, description and benefits, Annals of Library and Information Studies (ALIS) 61 (2014) 102–111

work page 2014
[6]

Shang, X

J. Shang, X. Zhang, L. Liu, S. Li, J. Han, Nettaxo: Automated topic taxonomy construction from text-rich network, in: Proceedings of the web conference 2020, 2020, pp. 1908–1919

work page 2020
[7]

Langlais, T

P. Langlais, T. L. Gao, Rate: a reproducible automatic taxonomy evaluation by filling the gap, in: Proceedings of the 15th International Conference on Computational Semantics, 2023, pp. 173–182

work page 2023
[8]

Amatriain, A

X. Amatriain, A. Sankar, J. Bing, P. K. Bodigutla, T. J. Hazen, M. Kazi, Transformer models: an introduction and catalog, arXiv preprint arXiv:2302.07730 (2023)

arXiv 2023

Show all 60 references

[9]

D’Souza, A catalog of transformer models, 2023

J. D’Souza, A catalog of transformer models, 2023. URL: https://orkg.org/comparison/R609337/. doi:10.48366/R609337

2023 doi
[10]

Stammbach, V

D. Stammbach, V. Zouhar, A. Hoyle, M. Sachan, E. Ash, Revisiting automated topic model evaluation with large language models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 9348–9357

2023
[11]

C. H. Papadimitriou, H. Tamaki, P. Raghavan, S. Vempala, Latent semantic indexing: A probabilistic analysis, in: Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, 1998, pp. 159–168

1998
[12]

Hofmann, Probabilistic latent semantic indexing, in: Proceedings of the 22nd annual interna- tional ACM SIGIR conference on Research and development in information retrieval, 1999

T. Hofmann, Probabilistic latent semantic indexing, in: Proceedings of the 22nd annual interna- tional ACM SIGIR conference on Research and development in information retrieval, 1999. URL: https://sigir.org/wp-content/uploads/2017/06/p211.pdf

1999
[13]

D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, Journal of machine Learning research 3 (2003) 993–1022. URL: https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf?ref= http://githubhelp.com

2003
[14]

T. L. Griffiths, M. Steyvers, Finding scientific topics, Proceedings of the National Academy of Sciences 101 (2004) 5228–5235. URL: https://dyurovsky.github.io/learning-humans-machines/class/ 24-class/papers/griffiths2004.pdf. doi: 10.1073/pnas.0307752101

2004 doi
[15]

D. P. Kingma, M. Welling, Auto-Encoding Variational Bayes, in: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. arXiv:http://arxiv.org/abs/1312.6114v10

2014 arXiv
[16]

Y. Miao, L. Yu, P. Blunsom, Neural variational inference for text processing, in: M. F. Balcan, K. Q. Weinberger (Eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, PMLR, New York, New York, USA, ...

2016
[17]

Y. Miao, E. Grefenstette, P. Blunsom, Discovering discrete latent topics with neural variational inference, in: D. Precup, Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, PMLR, 2017,...

2017
[18]

Srivastava, C

A. Srivastava, C. Sutton, Autoencoding variational inference for topic models, in: International Conference on Learning Representations, 2017. URL: https://openreview.net/forum?id=BybtVK9lg

2017
[19]

A. B. Dieng, F. J. R. Ruiz, D. M. Blei, Topic modeling in embedding spaces, Transactions of the Association for Computational Linguistics 8 (2020) 439–453. URL: https://aclanthology.org/2020. tacl-1.29. doi:10.1162/tacl_a_00325

2020 doi
[20]

S. Sia, A. Dalmia, S. J. Mielke, Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Asso...

2020 doi
[21]

Bianchi, S

F. Bianchi, S. Terragni, D. Hovy, Pre-training is a hot topic: Contextualized document embeddings improve topic coherence, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Intern...

2021
[22]

Grootendorst, Bertopic: Neural topic modeling with a class-based tf-idf procedure, arXiv preprint arXiv:2203.05794 (2022)

M. Grootendorst, Bertopic: Neural topic modeling with a class-based tf-idf procedure, arXiv preprint arXiv:2203.05794 (2022)

2022 arXiv
[23]

C. Pham, A. Hoyle, S. Sun, P. Resnik, M. Iyyer, TopicGPT: A prompt-based topic modeling framework, in: K. Duh, H. Gomez, S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

2024
[24]

Y. Mu, C. Dong, K. Bontcheva, X. Song, Large language models offer an alternative to the traditional approach of topic modelling, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational L...

2024
[25]

Y. Mu, P. Bai, K. Bontcheva, X. Song, Addressing topic granularity and hallucination in large language models for topic modelling, 2024. URL: https://arxiv.org/abs/2405.00611. arXiv:2405.00611

2024 arXiv
[26]

D. M. Blei, J. D. Lafferty, A correlated topic model of science, The Annals of Applied Statis- tics 1 (2007) 17–35. URL: https://projecteuclid.org/journals/annals-of-applied-statistics/volume-1/ issue-1/A-correlated-topic-model-of-Science/10.1214/07-AOAS114.full

2007 doi
[27]

Newman, A

D. Newman, A. Asuncion, P. Smyth, M. Welling, Distributed algorithms for topic models., Journal of Machine Learning Research 10 (2009)

2009
[28]

C. Wang, D. Blei, D. Heckerman, Continuous time dynamic topic models, in: Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI’08, AUAI Press, Arlington, Virginia, USA, 2008, p. 579–586

2008
[29]

G. E. Hinton, R. R. Salakhutdinov, Replicated softmax: an undirected topic model, in: Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, A. Culotta (Eds.), Advances in Neural Information Pro- cessing Systems, volume 22, Curran Associates, Inc., 2009. URL: https://proceedings....

2009
[30]

R. Ding, R. Nallapati, B. Xiang, Coherence-aware neural topic modeling, in: E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belg...

2018 doi
[31]

D. Card, C. Tan, N. A. Smith, Neural models for documents with metadata, in: I. Gurevych, Y. Miyao (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Aus...

2018 doi
[32]

Zhang, B

H. Zhang, B. Chen, D. Guo, M. Zhou, Whai: Weibull hybrid autoencoding inference for deep topic modeling, arXiv preprint arXiv:1803.01328 (2018)

2018 arXiv
[33]

Chang, S

J. Chang, S. Gerrish, C. Wang, J. Boyd-graber, D. Blei, Reading tea leaves: How hu- mans interpret topic models, in: Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, A. Culotta (Eds.), Advances in Neural Information Processing Systems, volume 22, Cur- ran Associates, Inc., ...

2009
[34]

Newman, J

D. Newman, J. H. Lau, K. Grieser, T. Baldwin, Automatic evaluation of topic coherence, in: R. Kaplan, J. Burstein, M. Harper, G. Penn (Eds.), Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics,...

2010
[35]

Mimno, H

D. Mimno, H. Wallach, E. Talley, M. Leenders, A. McCallum, Optimizing semantic coherence in topic models, in: R. Barzilay, M. Johnson (Eds.), Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Edin...

2011
[36]

Aletras, M

N. Aletras, M. Stevenson, Evaluating topic coherence using distributional semantics, in: A. Koller, K. Erk (Eds.), Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, Association for Computational Linguistics, Potsdam, Germany...

2013
[37]

Stammbach, V

D. Stammbach, V. Zouhar, A. Hoyle, M. Sachan, E. Ash, Revisiting automated topic model evaluation with large language models, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Comput...

2023 doi
[38]

Newman, Y

D. Newman, Y. Noh, E. Talley, S. Karimi, T. Baldwin, Evaluating topic models for digital libraries, in: Proceedings of the 10th Annual Joint Conference on Digital Libraries, JCDL ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 215–224. URL: https://doi.or...

2010
[39]

Bouma, Normalized (pointwise) mutual information in collocation extraction, Proceedings of GSCL 30 (2009) 31–40

G. Bouma, Normalized (pointwise) mutual information in collocation extraction, Proceedings of GSCL 30 (2009) 31–40

2009
[40]

J. H. Lau, D. Newman, T. Baldwin, Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality, in: S. Wintner, S. Goldwater, S. Riezler (Eds.), Proceedings of the 14th Conference of the European Chapter of the Association for Computational Ling...

2014 doi
[41]

R. Wang, D. Zhou, Y. He, Atm: Adversarial-neural topic model, Information Processing & Manage- ment 56 (2019) 102098. URL: https://www.sciencedirect.com/science/article/pii/S0306457319300500. doi:https://doi.org/10.1016/j.ipm.2019.102098

2019
[42]

R. Wang, X. Hu, D. Zhou, Y. He, Y. Xiong, C. Ye, H. Xu, Neural topic modeling with bidirectional adversarial training, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Associati...

2020
[43]

X. Wu, X. Dong, T. T. Nguyen, A. T. Luu, Effective neural topic modeling with embedding clustering regularization, in: International Conference on Machine Learning, PMLR, 2023, pp. 37335–37357

2023
[44]

Röder, A

M. Röder, A. Both, A. Hinneburg, Exploring the space of topic coherence measures, in: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 399–408. URL: https://doi.org/...

2015
[46]

S. I. Nikolenko, Topic quality metrics based on distributed word representations, in: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 10...

2016
[47]

Burkhardt, S

S. Burkhardt, S. Kramer, Decoupling sparsity and smoothness in the dirichlet variational au- toencoder topic model, Journal of Machine Learning Research 20 (2019) 1–27. URL: http: //jmlr.org/papers/v20/18-569.html

2019
[48]

F. Nan, R. Ding, R. Nallapati, B. Xiang, Topic modeling with Wasserstein autoencoders, in: A. Ko- rhonen, D. Traum, L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, ...

2019 doi
[49]

Bianchi, S

F. Bianchi, S. Terragni, D. Hovy, D. Nozza, E. Fersini, Cross-lingual contextualized topic models with zero-shot learning, in: P. Merlo, J. Tiedemann, R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistic...

2021 doi
[50]

Terragni, E

S. Terragni, E. Fersini, E. Messina, Word embedding-based topic similarity measures, in: E. Métais, F. Meziane, H. Horacek, E. Kapetanios (Eds.), Natural Language Processing and Information Systems, Springer International Publishing, Cham, 2021, pp. 33–45

2021
[51]

Bhatia, J

S. Bhatia, J. H. Lau, T. Baldwin, An automatic approach for document-level topic model evaluation, in: R. Levy, L. Specia (Eds.), Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Association for Computational Linguistics, Vancouver, C...

2017 doi
[52]

Korenčić, S

D. Korenčić, S. Ristov, J. Šnajder, Document-based topic coherence measures for news media text, Expert Systems with Applications 114 (2018) 357–373. URL: https://www.sciencedirect. com/science/article/pii/S0957417418304883. doi: https://doi.org/10.1016/j.eswa.2018. 07.063

2018 doi
[53]

Bhatia, J

S. Bhatia, J. H. Lau, T. Baldwin, Topic intrusion for automatic topic model evaluation, in: E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics...

2018 doi
[54]

Korenčić, S

D. Korenčić, S. Ristov, J. Repar, J. Šnajder, A topic coverage approach to evaluation of topic models, IEEE access 9 (2021) 123280–123312

2021
[55]

Rahimi, D

H. Rahimi, D. Mimno, J. Hoover, H. Naacke, C. Constantin, B. Amann, Contextualized topic coherence metrics, in: Y. Graham, M. Purver (Eds.), Findings of the Association for Computational Linguistics: EACL 2024, Association for Computational Linguistics, St. Julian’s, Malta, 20...

2024
[56]

X. Yang, H. Zhao, D. Phung, W. Buntine, L. Du, Llm reading tea leaves: Automatically evaluating topic models with large language models, arXiv preprint arXiv:2406.09008 (2024)

2024 arXiv
[57]

D. M. Blei, Probabilistic topic models, Commun. ACM 55 (2012) 77–84. URL: https://doi.org/10. 1145/2133806.2133826. doi:10.1145/2133806.2133826

2012
[58]

Frohmann, I

M. Frohmann, I. Sterner, I. Vulić, B. Minixhofer, M. Schedl, Segment any text: A universal approach for robust, efficient and adaptable sentence segmentation, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural La...

2024
[59]

George, Zipf

K. George, Zipf. human behavior and the principle of least effort, 1949

1949
[60]

A. L. Gerlach M, Shi H, A universal information theoretic approach to the identification of stopwords, Nature Machine Intelligence 1 (2019) 606–612. doi:10.1038/s42256-019-0112-6

2019 doi
[96]

doi:10.18653/v1/2021.acl-short.96

2021 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Waltman, N

L. Waltman, N. J. Van Eck, A new methodology for constructing a publication-level classification system of science, Journal of the American Society for Information Science and Technology 63 (2012) 2378–2392

work page 2012

[2] [2]

M. F. Moura, R. M. Marcacini, B. M. Nogueira, M. d. S. CONRADO, S. O. Rezende, A proposal for building domain topic taxonomies., In: WORKSHOP ON WEB AND TEXT INTELLIGENCE, 1.; SIMPÓSIO BRASILEIRO EM . . . , 2008

work page 2008

[3] [3]

Dauxais, U

Y. Dauxais, U. Zaratiana, M. Laneuville, S. D. Hernandez, P. Holat, C. Grosman, Towards automation of topic taxonomy construction, in: International Symposium on Intelligent Data Analysis, Springer, 2022, pp. 26–38

work page 2022

[4] [4]

Kotitsas, D

S. Kotitsas, D. Pappas, N. Manola, H. Papageorgiou, Scinobo: a novel system classifying scholarly communication in a dynamically constructed hierarchical field-of-science taxonomy, Frontiers in Research Metrics and Analytics 8 (2023) 1149834

work page 2023

[5] [5]

M. H. Bhat, S. M. Shafi, et al., Taxonomies in knowledge organisation—need, description and benefits, Annals of Library and Information Studies (ALIS) 61 (2014) 102–111

work page 2014

[6] [6]

Shang, X

J. Shang, X. Zhang, L. Liu, S. Li, J. Han, Nettaxo: Automated topic taxonomy construction from text-rich network, in: Proceedings of the web conference 2020, 2020, pp. 1908–1919

work page 2020

[7] [7]

Langlais, T

P. Langlais, T. L. Gao, Rate: a reproducible automatic taxonomy evaluation by filling the gap, in: Proceedings of the 15th International Conference on Computational Semantics, 2023, pp. 173–182

work page 2023

[8] [8]

Amatriain, A

X. Amatriain, A. Sankar, J. Bing, P. K. Bodigutla, T. J. Hazen, M. Kazi, Transformer models: an introduction and catalog, arXiv preprint arXiv:2302.07730 (2023)

arXiv 2023