{"id":"c663bd09-54bc-4336-95b2-68186c171565","arxiv_id":"2412.13534","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A clustering method that replaces document embeddings with language-model probabilities over generated texts achieves state-of-the-art results on four document datasets.","lead":"This paper clusters documents by using the probabilities a language model assigns to texts it generates from each document, rather than clustering the documents' usual embedding vectors. The method reports large gains over previous clustering techniques on four standard datasets and also improves generative document retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The state-of-the-art claim rests on the regularization exponent α=0.25, which was selected using ACC/NMI/ARI on the same four labeled test sets; at α=1 the gains largely vanish, so the reported superiority may be an artifact of test-set hyperparameter selection.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing concern. The paper's abstract claims state-of-the-art performance with large margins, but the margin comes almost entirely from the choice α=0.25. The paper provides no validation-based procedure for selecting this value; instead, Figure 1 sweeps α on the same datasets and metrics that are later reported as results, which constitutes test-label leakage in hyperparameter selection. At α=1 the method is no longer competitive, so the entire empirical contribution hinges on this single fitted scalar. A held-out validation test would determine whether α=0.25 generalizes or whether the reported numbers are an artifact of tuning on the evaluation sets. The mathematical propositions in the appendix appear correct for the estimated objective, and the code/data availability is a positive sign, but the empirical central claim is conditional. The reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":16866,"tokens_out":10818,"duration_ms":100103,"concrete_test":"Re-run the Table 2 experiments for GC with a proper validation split: for each of the four datasets, randomly hold out 20% of documents with labels, select α on that validation subset using ACC/NMI/ARI over the same grid as Figure 1, then evaluate the chosen α on the remaining 80% of documents. If the selected α is not 0.25 or the held-out metrics fall below the reported values by more than the reported standard deviations, the SOTA claim depends on test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that GC outperforms all previous clustering methods. The size of that advantage is driven by the choice α=0.25: Table 2 shows that using α=1 (the unbiased KL estimator) collapses R2 NMI from 77.8 to 25.8 and substantially reduces every other metric. The paper states in §5.2 that α was set to 0.25 for all experiments, and Figures 1(a–d) sweep α on the four datasets while measuring ACC, NMI, and ARI—metrics that require ground-truth labels. There is no held-out validation split and no theoretical derivation that selects 0.25; §8 offers only a qualitative bias-variance argument. Therefore the reported numbers are effectively selected on the test labels, so the claim that GC 'outperforms any previous clustering method' is not established for a genuinely unsupervised setting. This is load-bearing because without α=0.25 the method's advantage largely disappears.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'generative clustering' (GC), in which each document is represented by the conditional distribution p(Y|x) over texts generated by a large language model, and cluster centroids are distributions over the same space. The distortion between a document and a centroid is the KL divergence, estimated by regularized importance sampling with a proposal distribution p(Y), a clipping step, and an exponent alpha that controls the bias-variance tradeoff. The authors provide an alternating assignment/centroid-update algorithm, prove convergence and centroid optimality for the estimated objective, and report experiments on R2, R5, AG News, and Yahoo! Answers claiming state-of-the-art clustering performance. They also apply the method to hierarchical indexing for generative document retrieval and report improved retrieval accuracy on MS Marco Lite and NQ320K.","tokens_in":17068,"tokens_out":3510,"duration_ms":33646,"significance":"If the central claim were established for a genuinely unsupervised setting, this would be a valuable contribution: it gives a principled information-theoretic formulation of LLM-based document clustering, introduces an importance-sampling estimator with an explicit bias-variance knob, and demonstrates a credible application to generative retrieval. The algorithmic ideas are novel relative to the cited k-means-on-embeddings baselines, the code is released, and the appendix contains formal propositions for convergence and centroid optimality. However, the paper's headline 'state-of-the-art, outperforming any previous clustering method' claim is currently tied to a hyperparameter value (alpha=0.25) that is selected using ground-truth labels on the same four evaluation datasets, and the unbiased version (alpha=1) performs far worse, often below standard SBERT baselines. That weakness is load-bearing for the main empirical claim.","major_comments":[{"comment":"The value alpha=0.25 is selected by evaluating ACC, NMI, and ARI on the same labeled datasets that are later reported as the evaluation results. The ablations in Table 2 show this choice is decisive: with alpha=1, R2 NMI drops from 77.8 to 25.8, and the other datasets show large losses as well, often falling below the best SBERT baselines. Since clustering is presented as an unsupervised task, tuning the core hyperparameter on the test labels invalidates the claim that GC 'outperforms any previous clustering method' in a genuinely unsupervised setting. The authors need a held-out validation split, a nested cross-validation procedure, or a theoretical derivation that selects alpha=0.25 without label access. The qualitative bias-variance argument in Section 8 does not supply such a derivation.","section":"§6.2, §6.3, Figure 1(a-d), Table 2"},{"comment":"The paper proves optimality of the centroid update for the estimated objective and derives the proposal that minimizes the second moment of the importance weights, but it never shows that the biased estimator with alpha<1 preserves the cluster structure of the true KL divergence, i.e., that the ordering of pairwise distances d(x,k) is approximately maintained. This is not merely a technical gap: the alpha=1 ablation performs so much worse that the claimed superiority is attributable to the particular bias induced by alpha=0.25 rather than to the information-theoretic objective itself. A formal or systematic empirical statement about when the biased estimator preserves cluster structure is needed to support the method's name and its claimed advantage.","section":"§5.2, Eq. (5), Propositions 2-3"},{"comment":"The abstract and Section 6.3 claim that GC outperforms 'any previous clustering method', but Table 2 contains no recent LLM-based clustering baselines, even though Viswanathan et al. (2023) and Zhang, Wang, and Shang (2023) are cited in Related Work. The comparison set consists of older k-means, DEC/IDEC/STC/DFTC, and embedding methods. The state-of-the-art claim is therefore not supported by the experiments as presented. The authors should either benchmark against those recent methods or explicitly qualify the claim to the compared baselines.","section":"§2.1, §6.3, Table 2"},{"comment":"The retrieval experiment repeats the same alpha-selection issue: alpha=0.25 gives the best retrieval numbers, and the ablations show alpha=1 is substantially worse (e.g., MS Marco Lite Rec@1 drops from 32.41 to 24.43). The paper states that alpha=0.25 is a 'robust choice' without a validation procedure for the retrieval task. If alpha is selected using the clustering labels on the four datasets and then transferred to retrieval, the retrieval result is not an independent confirmation; it is a second instance of the same fitted hyperparameter.","section":"§7, Table 3"}],"minor_comments":[{"comment":"The phrase 'The condition probability' should be 'The conditional probability'.","section":"§3, first paragraph"},{"comment":"Equation (4) has mismatched parentheses: the norm expression 'E[p(Y |f (x))∥2' is missing a closing bracket and uses a nonstandard norm notation; it should read '∥E[p(Y |x)] − E[p(Y |f(x))]∥²'.","section":"§4, Eq. (4)"},{"comment":"Algorithm 1 says Y is generated by i.i.d. sampling 'from the language model', but the actual procedure in Section 5.3 first samples a document uniformly and then generates a text from it. The algorithm should state this two-step sampling procedure explicitly.","section":"§5.1, Algorithm 1 line 1 and §5.3"},{"comment":"The proof ends with 'This completes the proof of Proposition 5.4', but the proposition is numbered Proposition 2 in the appendix. The cross-reference should be corrected.","section":"Appendix A.2, proof of Proposition 2"},{"comment":"The notation 'yj ∼ ϕ(Y)' under the summation is confusing; it should be 'yj ∼ ϕ' or 'yj drawn from ϕ'.","section":"§5.2, Eq. (5)"},{"comment":"Standard deviations are reported only for the GC row. Reporting them for the strongest baselines would help the reader judge whether margins of a few points (e.g., Yahoo! Answers ACC 60.7 vs. 58.2 for all-minilm-l12-v2) are meaningful.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core algorithm is interesting and the experiments are extensive, but the main empirical claim currently rests on test-set hyperparameter selection. The authors should be asked to provide a validation-based or theoretically grounded choice of alpha and to benchmark against recent LLM-based clustering methods before the state-of-the-art claim can be sustained. If they can do that, the paper would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new idea: represent each document as an LLM's conditional probability distribution over generated texts, then cluster by KL divergence estimated via regularized importance sampling. The formalization is coherent, the appendix propositions are correct for the estimated objective, and the code is public. The authors also show a sensible application to generative retrieval.\n\nWhat it does well: the importance-sampling formulation with the exponent alpha is a real contribution, and the writing is clear about the method's mechanics. The ablations are honest enough to include alpha=1 and naive p(Y), which is more than many papers do. The math in the appendix checks out as far as I can tell, and the retrieval gains are consistent with the clustering results, even if smaller.\n\nWhere it gets soft: the central empirical claim — that GC outperforms every previous clustering method — is not actually supported by the protocol. The authors state in Section 5.2 that alpha=0.25 for all experiments, and Figures 1(a-d) sweep alpha on the four datasets while measuring ACC, NMI, and ARI, all of which require ground-truth labels. There is no held-out validation split and no derivation that selects 0.25. Section 8 offers only a qualitative bias-variance discussion. This is load-bearing because Table 2 shows that alpha=1 collapses R2 NMI from 77.8 to 25.8 and substantially reduces every other metric. So the reported superiority is effectively a fitted result on the test sets. The paper would need a validation-based selection or a principled derivation of alpha before the SOTA claim holds. Also minor: generation parameters and a pinned code commit are missing, though the code itself is a plus.\n\nBottom line: this is a serious paper with a promising formulation and correct math, but the headline result is conditional on a hyperparameter chosen with label leakage. The right response is to engage with it in peer review and require the authors to fix the selection protocol or soften the claims. The idea deserves a bench, but not as-is.\n\nRecommendation: send to peer review with major revision required.","headline":"Novel LLM-distribution clustering with correct math, but the SOTA claim rests on alpha=0.25 selected on test labels; at alpha=1 the gains mostly vanish.","tokens_in":17569,"tokens_out":1561,"would_cite":false,"duration_ms":16519,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Clustering by LLM-generated text outdoes every baseline","keywords":["generative clustering","document clustering","large language models","KL divergence","importance sampling","generative document retrieval","regularized importance sampling","LLM-generated texts"],"falsifier":"On any held-out corpus, compute GC's ACC, NMI, and ARI with $\\alpha=0.25$ and with $\\alpha$ tuned on a disjoint validation split; if the tuned value differs substantially or the fixed 0.25 version loses to the best SBERT embedding baseline, the claim that 0.25 is a universally robust setting collapses.","tokens_in":16638,"feed_emoji":"🗂️","tokens_out":6206,"duration_ms":51382,"temperature":0.7,"pith_summary":"Instead of clustering documents directly, generative clustering (GC) turns each document $x$ into a probability distribution $p(Y|x)$ over texts that a large language model might generate from $x$, and measures how far two documents are by the KL divergence between these distributions. The paper's central claim is that this information-theoretic reformulation, together with a regularized importance-sampling algorithm, clusters documents more accurately than any previous method, often by a large margin, on R2, R5, AG News, and Yahoo! Answers. The same construction, applied hierarchically to build document indexes for generative retrieval, improves Recall@1 by up to 36 percent over a strong retrieval baseline on MS Marco Lite. A sympathetic reader would care because the result suggests that the unsaid knowledge an LLM supplies about a text is genuinely useful for recovering cluster structure, and because the formulation replaces heuristic vector-plus-Euclidean-distance pipelines with a principled distributional objective.","feed_headline":"Clustering by LLM-generated text outdoes every baseline","feed_subtitle":"One method, four datasets: generated texts plus KL divergence beat all existing clusterers by wide margins.","key_machinery":"The central object is the per-document distribution $p(Y|x)$ over the infinite set $Y$ of all possible word sequences, computed by multiplying word-level probabilities from an LLM. The load-bearing identity is the importance-sampling rewrite of the KL divergence: $\\mathbb{E}_{Y\\sim p(Y|x)}\\log\\frac{p(Y|x)}{p(Y|k)} = \\mathbb{E}_{Y\\sim\\phi}\\left[\\frac{p(Y|x)}{\\phi(Y)}\\log\\frac{p(Y|x)}{p(Y|k)}\\right]$, which turns an intractable divergence into a finite-sample estimator over a shared set of generated texts. The proposal $\\phi$ is set to the second-moment-minimizing distribution (Eq. 7), the estimator uses regularized weights $W_{ij}=(p(y_j|x_i)/\\phi(y_j))^\\alpha$ with $\\alpha=0.25$, and cluster centroids are normalized sums of those weights, making the whole procedure a Bregman hard-clustering iteration.","core_discovery":"Generative clustering defines a document not by a fixed vector but by the conditional distribution $p(Y=y|x)$ over the infinite set of all word sequences, with each word probability computed autoregressively by an LLM. The distortion between document $x$ and cluster centroid $k$ is the KL divergence $\\mathrm{KL}(p(Y|x)\\|p(Y|k))$; because $Y$ is infinite, this is estimated by regularized importance sampling from a proposal $\\phi$ chosen to minimize the second moment of the importance weights, yielding the estimator in Eq. (5) with exponent $\\alpha=0.25$. The algorithm alternates between assigning each document to its closest centroid and updating each centroid to the normalized mean of the importance-weight rows in its cluster, a procedure proven to converge to a local minimum. On four datasets, this method is reported to outperform all compared baselines on accuracy, NMI, and ARI, with the largest gap on R2 (96.1% accuracy and 77.8 NMI versus 65.6 NMI for the best SBERT baseline). The paper also reports that applying the same KL-based hierarchical clustering to build prefix-code indexes raises retrieval accuracy in generative document retrieval beyond two prior indexing methods.","pith_inferences":["The $\\alpha=0.25$ choice was made by evaluating on the same datasets reported; a fairer test would tune $\\alpha$ on a validation split, and the paper does not show that 0.25 transfers to unrelated corpora.","If the gains come from the LLM's ability to explicate implicit content, then using a more capable or domain-adapted generator should improve clustering further; likewise, datasets whose documents are not query-like may show smaller gains, since the doc2query model is specialized for generating queries.","The same regularized-importance-sampling trick could be applied to other infinite-space divergence problems, such as clustering images or code by their generation probabilities, whenever a generative model supplies exact conditional probabilities.","A direct extension would be to use the generated-text distribution as a feature for supervised tasks, or to check whether the importance-weight matrix $W$ itself provides interpretable cluster labels through its highest-weight texts."],"forward_implications":["On the four tested datasets, GC beats every compared clustering baseline on ACC, NMI, and ARI; on R2 it cuts the error rate from 8.0% to 3.9%.","Using $\\alpha=0.25$ instead of the unbiased $\\alpha=1$ improves NMI on R2 from 25.8 to 77.8, showing that variance reduction, not unbiasedness, is what makes the estimator recover cluster structure.","Performance is stable once the number of generated texts $J$ is at least 50 on small datasets and 384 on Yahoo! Answers, below the 768-dimensional BERT embedding.","When $K$ is misspecified, GC still outperforms the best SBERT baseline across $K$ from 2 to 20 on R5.","Applied hierarchically to prefix-code indexing, GC raises Recall@1 on MS Marco Lite by 36% over BMI and MRR@100 by 25%.","The proposal distribution remains effective when localized to sub-clusters during hierarchical clustering, and removing that localization reduces retrieval accuracy."],"supporting_citations":[{"why":"Defines Bregman hard clustering and supplies the two-step iteration and centroid-optimality results that Algorithm 1 generalizes.","marker":"(Banerjee et al. 2005)"},{"why":"Introduces importance sampling, the mechanism used to estimate the infinite-space KL divergence.","marker":"(Kloek and Van Dijk 1978)"},{"why":"Provides regularized importance sampling, whose bias-variance tradeoff motivates the $\\alpha=0.25$ estimator.","marker":"(Korba and Portier 2022)"},{"why":"Supplies the R2 and R5 dataset construction and the DFTC deep clustering baseline that GC is compared against.","marker":"(Guan et al. 2022)"},{"why":"Provides the doc2query T5 model used to generate $Y$ and to evaluate generation probabilities.","marker":"(Nogueira, Lin, and Epistemic 2019)"},{"why":"Introduces generative document retrieval with prefix-code indexing that the hierarchical GC application builds on.","marker":"(Tay et al. 2022)"},{"why":"Provides the NCI retrieval baseline and training setting used for the generative retrieval comparison.","marker":"(Wang et al. 2022)"},{"why":"Provides the BMI baseline and the MS Marco Lite evaluation setup used in retrieval experiments.","marker":"(Du, Xiu, and Tanaka-Ishii 2024)"}],"fun_headline_variants":["Cluster by LLM text, beat all baselines","LLM-generated text redefines document clustering","Information-theoretic clustering via LLM sampling","Clustering with KL divergence on generated text","Generative clustering outperforms every previous method"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's superiority rests on the regularization exponent $\\alpha=0.25$, which was selected by measuring clustering performance on the same four datasets that are later reported as results; the paper gives no derived or held-out justification that this value generalizes to new collections.","fun_headline_variants_meta":{"raw":{"variants":["Cluster by LLM text, beat all baselines","LLM-generated text redefines document clustering","Information-theoretic clustering via LLM sampling","Clustering with KL divergence on generated text","Generative clustering outperforms every previous method"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2180,"prompt_tokens":897,"completion_tokens":1283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1214}},"tokens_in":513,"tokens_out":1283,"duration_ms":9565,"temperature":1.0,"reasoning_tokens":1214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:01:23.188035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On any held-out corpus, compute GC's ACC, NMI, and ARI with $\\alpha=0.25$ and with $\\alpha$ tuned on a disjoint validation split; if the tuned value differs substantially or the fixed 0.25 version loses to the best SBERT embedding baseline, the claim that 0.25 is a universally robust setting collapses.","supporting_citations":[{"cited_title":"S.; Ghosh, J.; and Lafferty, J","cited_arxiv_id":null,"evidence_quote":"Defines Bregman hard clustering and supplies the two-step iteration and centroid-optimality results that Algorithm 1 generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces importance sampling, the mechanism used to estimate the infinite-space KL divergence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides regularized importance sampling, whose bias-variance tradeoff motivates the $\\alpha=0.25$ estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the R2 and R5 dataset construction and the DFTC deep clustering baseline that GC is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the doc2query T5 model used to generate $Y$ and to evaluate generation probabilities."},{"cited_title":"W.; and Metzler, D","cited_arxiv_id":null,"evidence_quote":"Introduces generative document retrieval with prefix-code indexing that the hierarchical GC application builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NCI retrieval baseline and training setting used for the generative retrieval comparison."}],"review_version":1}