{"id":"c2c14ab3-2bc8-4f38-ad5a-b4f73a0a38e8","arxiv_id":"2412.17338","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A topic-wise contrastive regularizer using precomputed NPMI similarities improves coherence and diversity of neural topic model topics on 20NG, Yahoo, and NYTimes.","lead":"This paper adds a contrastive learning regularizer to neural topic models, grouping words into topics and pushing words from different topics apart during training. The authors report higher topic coherence and diversity on three text corpora, plus better word-intrusion scores in a small human study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-corpus interpretability claim hinges on NPMI as a human-aligned proxy, but the regularizer directly optimizes NPMI and human validation covers only 20NG; this remains the least secure link.","rationale":"I read the paper in good faith. The regularizer is clearly specified, the ablations (Sec. V.G) and backbone-substitution experiments (Sec. V.I) show internal consistency, and the human word intrusion results on 20NG are a genuine positive signal. The reason I focus on NPMI-human alignment rather than, say, missing error bars or unavailable code is that the main empirical claim is about interpretability, not about NPMI. Because Lcon uses NPMI as its similarity function and NPMI is also the headline metric, any experiment reporting NPMI is partially self-confirming. That would be acceptable if NPMI were a settled proxy, but the manuscript itself cites the controversy and offers only one correlation study [40] plus the 20NG human study. That study is too small and too narrow to establish corpus-level generalization, and no significance testing is reported. This makes the cross-corpus interpretability claim the least secure part of the argument. The reader's weakest_assumption is the same concern, and I agree. A direct human evaluation on Yahoo and NYTimes would settle it; if WIS there favors ContraTopic, the central claim holds. If not, the verdict should remain conditional with the claim restricted to NPMI improvement. The appropriate verdict is therefore unchanged from the reader's CONDITIONAL.","tokens_in":18166,"tokens_out":6698,"duration_ms":72070,"concrete_test":"Run the same word-intrusion protocol (Sec. V.J) on Yahoo and NYTimes, sampling topics across NPMI deciles for ContraTopic and the strongest baselines (ETM, NSTM, WeTe, CLNTM), with at least 15 judges per topic as recommended by [4]. Report WIS with bootstrap confidence intervals and a paired significance test. If ContraTopic does not show a significant WIS advantage on these corpora, the claim that the NPMI-driven regularizer yields consistently higher human interpretability is not supported beyond 20NG.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ContraTopic 'consistently produces topics with significantly higher levels of interpretability compared to all baseline approaches' (Abstract, Sec. VII). The regularizer Lcon in Eq. (2) uses precomputed NPMI as the similarity K(·) (Sec. IV.A), while the primary evaluation metric is also average NPMI (Sec. V.B). Consequently, an NPMI improvement is partly the expected outcome of directly optimizing a smoothed version of the evaluation metric and does not by itself demonstrate human interpretability. The only independent human-based evidence is the word intrusion study on 20NG (Table III), with 20 participants, no significance tests, and no error bars. The paper itself notes the debate on whether NPMI tracks human judgment (Sec. IV.A, Refs. [4],[14],[39]), and the Limitation section concedes that precomputed NPMI is only a proxy. Thus the load-bearing assumption is that optimizing NPMI transfers to human interpretability on all three corpora. If that assumption fails, the result is 'higher NPMI with a possible human-judged improvement on one dataset' rather than the claimed consistent interpretability gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ContraTopic, a regularizer for neural topic models that augments the ELBO of an ETM-style backbone with a topic-wise contrastive loss. The regularizer samples v words per topic using a relaxed Gumbel subset sampler, treats words from the same topic as positive pairs and words from different topics as negative pairs, and uses precomputed NPMI scores as the similarity function K(·). The authors report improvements in average NPMI topic coherence and topic diversity over nine baselines on 20NG, Yahoo, and NYTimes, supplemented by ablations, sensitivity analyses, backbone-substitution experiments, a word-intrusion study on 20NG, and qualitative topic lists.","tokens_in":18448,"tokens_out":4142,"duration_ms":43659,"significance":"If the reported improvements are genuine and independently confirmed, ContraTopic would be a simple and useful plug-in: it adds one regularizer to an existing neural topic model, requires no human labels, and improves both coherence and diversity in the reported settings. The paper has real strengths: Table II ablates the positive/negative terms and the similarity function, Figure 6 tests the regularizer across different backbone models, the sensitivity analysis is informative, and Section VI candidly discusses limitations. The main reservations concern the alignment between the optimized objective and the evaluation metric, and the statistical support for the headline claims.","major_comments":[{"comment":"The regularizer's similarity function K(·) is the precomputed NPMI, and the main topic-coherence evaluation is also mean NPMI on the test set. The coherence gains in Figure 2 are therefore, to a substantial degree, the expected consequence of directly optimizing a smoothed surrogate of the evaluation statistic; they do not by themselves establish improved interpretability. The word-intrusion study in Section V.J is the only independent evidence, and it covers only 20NG. Section VI further concedes that precomputed NPMI is only a proxy for human interpretability. I would ask for either (i) human or otherwise independent topic-quality evaluations on at least Yahoo and NYTimes, or a clear reframing of the claim as 'higher NPMI and higher unique-word diversity' rather than 'higher interpretability'; and (ii) an analysis of how much of the NPMI gain persists when K(·) is computed on a held-out corpus or with a different coherence estimator.","section":"Section IV.A, Eq. (2), and Section V.B"},{"comment":"The abstract and Section VII claim that ContraTopic 'consistently produces topics with significantly higher levels of interpretability,' but Figure 2 omits error bars 'for brevity' and Table III reports no variance or significance test. Three random-seed runs with only means reported cannot support the word 'significantly.' The word-intrusion result (ContraTopic 0.80 vs. the best baseline 0.68) needs at least a confidence interval or a paired significance test (e.g., over topics or over participants) before it can be used as evidence for the central claim.","section":"Section V.F, Figure 2, and Section V.J, Table III"},{"comment":"Hyperparameter selection is not fully controlled. ContraTopic's λ, v, τβ, and τg are tuned on a validation split, while baselines are said to use 'best reported settings' or official implementations. Because λ is dataset-specific (40, 40, and 300 for 20NG, Yahoo, and NYTimes), it is unclear how much of the reported improvement comes from the regularizer itself rather than from additional tuning on the target datasets. Reporting validation-tuned results for the baselines, or at least a sensitivity range for key baselines, would make the comparison fairer.","section":"Section V.C and Section V.D"}],"minor_comments":[{"comment":"The text repeatedly says 'non-differential' where 'non-differentiable' is meant; please correct this throughout the methodology.","section":"Section IV.B, Eq. (3)"},{"comment":"The sentence listing tuned hyperparameters says 'λ, v, τg, and τg'; the second τg should be τβ.","section":"Section V.D"},{"comment":"The heading 'The sensitive analysis' should read 'The sensitivity analysis.'","section":"Section V.G"},{"comment":"The spelling 'Contratopic' and 'ContraTopic' is inconsistent; please unify to one form.","section":"Section IV.A and elsewhere"},{"comment":"The questionnaire description should state whether all 300 questions were answered by each participant in a fixed order or a randomized order, and should report per-participant variability in addition to the mean WIS.","section":"Section V.J"},{"comment":"The similarity function K(·) in Eq. (2) has no temperature or scaling parameter; since K is an NPMI value, please clarify whether it is used directly or scaled, because the scale interacts with the choice of λ.","section":"Section V.E and Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The core methodological concern is the circularity between the NPMI-based similarity in the regularizer and the NPMI-based evaluation metric. This is fixable within the manuscript's scope by adding independent human or LLM-based evaluation on more than one dataset, by reporting significance tests and error bars, and by tempering the abstract's unqualified interpretability claim. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful paper on a real problem, with a genuinely new regularizer and reasonable experiments, but the headline \"significantly higher interpretability\" overreaches: the main metric is one the regularizer directly optimizes, and the human evidence is a single 20-person study on one dataset.\n\nWhat's new: a topic-wise contrastive loss for NTMs, using words as samples, precomputed NPMI as the similarity function, and Gumbel-softmax subset sampling to get differentiable top-word draws. That combination is absent from prior contrastive NTMs like CLNTM and TSCTM, which work document-wise. The ablations show positive and negative terms each contribute, NPMI beats embedding similarity for K(·), and the relaxed sampler helps. The backbone-substitution experiment (ETM to WLDA and WeTe) is a nice touch—it makes the regularizer look architecture-agnostic. The paper is also honest in the Limitations section about the NPMI proxy and the education bias in the participant pool.\n\nWhere it wobbles: the main coherence metric is average NPMI, and Lcon is built from NPMI pairs, so the coherence improvement is partly expected. They evaluate on held-out test NPMI, which breaks the tightest loop, but it's still the same statistic. The human word-intrusion study is on 20NG only, with 20 participants, no significance tests or error bars, so the cross-corpus claim rests on NPMI. Error bars are omitted from the main figures, which is unfortunate given the variance shown in the ablation table. And the code is only promised in \"supplementary materials\"—not linked in the arXiv version I read.\n\nNet: the central claim should be softened to \"improves NPMI coherence and diversity and shows promising human-judged interpretability on one dataset.\" As a method paper, it deserves review. A serious referee should ask for error bars, at least one more human evaluation dataset (or a more robust analysis of the existing one), and a working code link.","headline":"A genuinely new topic-wise contrastive regularizer that likely improves topic coherence and diversity, though the headline cross-corpus interpretability claim outruns the evidence.","tokens_in":18975,"tokens_out":2188,"would_cite":true,"duration_ms":22481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A topic-wise contrastive regularizer makes neural topic models produce more coherent, more distinct topics.","keywords":["neural topic models","topic interpretability","contrastive learning","topic coherence","topic diversity","NPMI","Gumbel-Softmax","regularization"],"falsifier":"Run the paper's word intrusion study on Yahoo or NYTimes rather than only 20NG. If ContraTopic's word intrusion scores do not exceed the baselines on those corpora, or if a corpus is constructed where high-NPMI word sets are not semantically coherent to annotators, the claim that optimizing NPMI via topic-wise contrastive learning robustly improves human interpretability would be refuted.","tokens_in":17993,"feed_emoji":"🧩","tokens_out":5836,"duration_ms":49919,"temperature":0.7,"pith_summary":"Neural topic models are typically trained to maximize the likelihood of observed text, but that objective does not guarantee that the discovered topics make sense to a person. This paper argues that topic interpretability should be optimized directly, and proposes ContraTopic, a regularizer that measures interpretability during training. The regularizer treats words drawn from the same topic as positive samples and words from different topics as negative samples, with word similarity scored by precomputed NPMI. Training with this term alongside the usual reconstruction and KL terms yields topics that the paper reports are more coherent and more diverse than those from ten baseline models across 20NG, Yahoo, and NYTimes. A human word intrusion study on 20NG gives ContraTopic the highest score, supporting the claim that the regularizer improves interpretability as people perceive it.","feed_headline":"New regularizer beats neural topic models on coherence and diversity","feed_subtitle":"It adds a contrastive regularizer that boosts both coherence and distinctness of discovered topics.","key_machinery":"The load-bearing component is the topic-wise contrastive regularizer Lcon defined by Eq. (2): a supervised contrastive loss whose samples are words, not documents. Positive pairs are words sampled from the same topic; negative pairs are words from different topics; and the similarity kernel K(·) is the precomputed NPMI matrix over the vocabulary. To keep the sampling differentiable, the paper uses a Gumbel-Softmax reparameterization with a relaxed subset-sampling procedure (Eqs. (3)-(5)) that draws top-v words per topic without replacement. This mechanism lets gradients flow from an interpretability-style objective directly into the topic-word distribution, which is what allows the model to trade likelihood for more human-readable topics.","core_discovery":"ContraTopic's central claim is that topic interpretability can be decomposed into internal coherence and external distinctiveness and steered directly by a contrastive loss on words, not documents. For each topic, the model samples its top-v words using a relaxed Gumbel-Softmax subset sampler, then applies a supervised contrastive loss where words from the same topic are positive pairs and words from different topics are negative pairs. The similarity function K(·) is the precomputed NPMI between word pairs, which the paper adopts because NPMI is a standard coherence metric and because maximizing in-topic NPMI while minimizing cross-topic NPMI is equivalent to maximizing mutual information within topics and minimizing it between topics. The full objective is Ltr = Lrec + Lkl + λLcon, and the paper reports that this regularizer consistently improves topic coherence and topic diversity over ETM, ProdLDA, WLDA, NSTM, WeTe, NTM-R, VTMRL, and CLNTM on three datasets.","pith_inferences":["If NPMI-based coherence is a bottleneck, one could swap in a learned or LLM-based similarity function that still admits the same contrastive loss, potentially improving alignment with human judgments on corpora where NPMI is weak.","The topic-wise regularizer could be combined with document-wise contrastive objectives in a unified multi-level framework, possibly improving document representation quality without sacrificing topic interpretability.","The metric-overfitting danger suggests testing on held-out corpora with a different distribution than the training set; the paper already evaluates coherence on unseen test data, but a cross-corpus transfer test would sharpen the claim."],"forward_implications":["Adding the ContraTopic regularizer to an NTM is a plug-in way to improve topic coherence and diversity without changing the backbone architecture; the paper shows it works with ETM, WLDA, and WeTe as backbones.","Because the regularizer couples to word similarity via NPMI, replacing the similarity kernel (e.g., with embeddings or LLM-based scores) is a direct lever for steering topic quality toward different notions of interpretability.","The ablation results imply that both positive and negative pairs contribute: dropping negative pairs costs about 5% and dropping positive pairs about 12% on the reported metrics, so the method's benefit depends on the full topic-wise contrastive structure."],"supporting_citations":[{"why":"Supplies the empirical justification that NPMI correlates with human coherence judgments across general and specialized corpora, grounding the choice of K(·).","marker":"[40]"},{"why":"Provides the supervised contrastive loss form that the topic-wise regularizer adapts from image representation learning.","marker":"[28]"},{"why":"Introduces the Gumbel-Softmax reparameterization that makes sampling from the topic-word distribution differentiable.","marker":"[18]"},{"why":"Provides the relaxed subset sampling algorithm used to draw multiple words per topic without replacement.","marker":"[41]"},{"why":"Defines the ETM backbone with word embeddings and topic embeddings that ContraTopic builds on and shares hyperparameters with.","marker":"[11]"},{"why":"Introduces CLNTM, the document-wise contrastive NTM baseline that ContraTopic distinguishes itself from and outperforms.","marker":"[34]"},{"why":"Proposes a differentiable coherence-aware neural topic model (NTM-R), the closest prior regularizer that only targets coherence, which ContraTopic extends by also addressing diversity.","marker":"[13]"},{"why":"Provides the word intrusion evaluation design and also raises the debate about whether automated coherence metrics track human interpretability, motivating the human study.","marker":"[4]"}],"fun_headline_variants":["Contrastive topic learning boosts coherence and diversity","Topic-wise contrastive loss yields clearer topics","ContraTopic: contrastive regularizer for interpretable topics","New contrastive method improves topic interpretability","Training NTMs with contrastive topical coherence wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on treating NPMI scores computed on the training corpus as a reliable, corpus-independent proxy for what humans find coherent; if NPMI and human judgment diverge on a given corpus, optimizing it could raise reported coherence without improving real interpretability.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive topic learning boosts coherence and diversity","Topic-wise contrastive loss yields clearer topics","ContraTopic: contrastive regularizer for interpretable topics","New contrastive method improves topic interpretability","Training NTMs with contrastive topical coherence wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2656,"prompt_tokens":933,"completion_tokens":1723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1651}},"tokens_in":549,"tokens_out":1723,"duration_ms":12737,"temperature":1.0,"reasoning_tokens":1651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:34:11.773843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's word intrusion study on Yahoo or NYTimes rather than only 20NG. If ContraTopic's word intrusion scores do not exceed the baselines on those corpora, or if a corpus is constructed where high-NPMI word sets are not semantically coherent to annotators, the claim that optimizing NPMI via topic-wise contrastive learning robustly improves human interpretability would be refuted.","supporting_citations":[{"cited_title":"Large-scale correlation analysis of automated metrics for topic models,","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical justification that NPMI correlates with human coherence judgments across general and specialized corpora, grounding the choice of K(·)."},{"cited_title":"Supervised contrastive learning,","cited_arxiv_id":null,"evidence_quote":"Provides the supervised contrastive loss form that the topic-wise regularizer adapts from image representation learning."},{"cited_title":"Contrastive learning for neural topic model,","cited_arxiv_id":null,"evidence_quote":"Introduces CLNTM, the document-wise contrastive NTM baseline that ContraTopic distinguishes itself from and outperforms."},{"cited_title":"Coherence-aware neural topic modeling,","cited_arxiv_id":null,"evidence_quote":"Proposes a differentiable coherence-aware neural topic model (NTM-R), the closest prior regularizer that only targets coherence, which ContraTopic extends by also addressing diversity."},{"cited_title":"Is automated topic model evaluation broken? the incoherence of coherence,","cited_arxiv_id":null,"evidence_quote":"Provides the word intrusion evaluation design and also raises the debate about whether automated coherence metrics track human interpretability, motivating the human study."}],"review_version":1}