Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A statistically consistent measure of semantic uncertainty using Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that spectral clustering turns noisy language-model pairwise judgments into a statistically consistent estimate of semantic entropy.

desk verdict The paper has a sensible idea and a badly broken proof: the claimed consistency rests on a false inequality in Lemma 3.1. read the letter →

arxiv 2502.00507 v3 pith:Z42DMQXC submitted 2025-02-01 cs.CL cs.AI

Yi Liu This is my paper · ORCID
classification cs.CLcs.AI MSC 62H3062G2060F15
keywords semanticspectralentropyclusteringstochasticblockmodelestimationstrongconsistencylargelanguagemodelsequivalenceuncertaintyquantification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most language-model uncertainty scores either need internal token probabilities or make no formal guarantee that they approach the true semantic spread. This paper defines semantic spectral entropy: ask an off-the-shelf language model whether pairs of generated texts are semantically equivalent, put those yes/no answers into a graph, cluster the graph with spectral clustering, and estimate entropy from cluster sizes. The main result is that this estimate is strongly consistent — the error vanishes almost surely as the number of texts grows — under a stochastic block model for the judge's errors. The paper also supplies finite-sample error rates and proves convergence to the true entropy when texts are drawn from a generative process. If correct, this is the first semantic uncertainty measure with a proven convergence guarantee, and it works with black-box models that only return text.

What carries the argument

Semantic spectral entropy, the named object, is the entropy of the empirical distribution of texts over clusters recovered by spectral clustering. The machine that carries the argument is the stochastic block model on the judge's pairwise equivalence labels, with parameters $p$ and $q$; the paper imports strong-consistency and finite-sample bounds for spectral clustering in this model and translates misclustering error into entropy error through Lemma 3.1. The number of clusters $K$ is an input to the clustering step, with cross-validation discussed as a practical way to choose it.

What would settle it

Generate a corpus with known semantic equivalence classes, ask a language model to judge every pair repeatedly, and test for constant per-pair error probabilities and independence across pairs, for example by comparing observed between-pair error variance to the binomial null and checking whether judgments correlate with pair order or shared wording. Rejecting the stochastic-block-model null would show that the assumptions behind Theorems 3.2 and 3.4 do not hold for that judge, so the claimed almost-sure convergence lacks its stated basis.

Watch

Extended reading notes

Core claim

The central claim, stated as Theorem 3.2 and Theorem 3.4, is that $|\bar{E}(T)-\hat{E}(T)| \to 0$ almost surely and that, under a generative model, $|E-\hat{E}|$ has explicit finite-sample bounds that go to zero. The estimator is built from a graph whose edges are the language model's pairwise equivalence decisions $e(t_i,t_j)$; when those decisions follow a homogeneous stochastic block model — equivalent pairs judged equivalent with probability $p$, non-equivalent pairs judged equivalent with probability $q$ — spectral clustering recovers the semantic clusters well enough that entropy estimated from cluster sizes converges to the entropy one would get from perfect equivalence judgments. A key structural fact is that the entropy error is controlled by the misclustering error $M_{\mathrm{error}}$ (Lemma 3.1).

Load-bearing premise

The theory's load-bearing premise is that a language model's equivalence judgments are independent draws with the same probability q of judging every non-equivalent pair equivalent and the same probability p of judging every equivalent pair equivalent, and that the number of semantic clusters is known; if error rates vary from pair to pair, judgments are correlated, or K is misspecified, the quoted consistency theorems no longer apply.

Editorial extensions

If this is right

  • No token-level probabilities are needed; the same estimator applies to any judge model that can answer yes/no equivalence questions, so black-box API models can be scored.
  • For a fixed number of semantic clusters and a stable judge, the entropy error shrinks as $O(1/n)$ (Corollary 3.3.1), so collecting more sampled outputs tightens the uncertainty estimate.
  • When outputs come from a generative process, consistency to the true entropy holds with an explicit probabilistic bound, and the number of clusters may grow with sample size as $K=o(n^{1/2-\delta})$ (Section 3.3).
  • The simulation identifies $p-q$, the judge's tendency to distinguish non-equivalent pairs, as the practical driver of accuracy, with a sharp phase change near $p-q=0.4$.
  • Because the method only needs pairwise text comparisons and not the generation mechanism, it can be applied to any collection of texts, not only strings produced by a language model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the homogeneous stochastic block model is unlikely to hold exactly for real language models; pair-specific text difficulty, lexical overlap, and correlated judgments will produce variable $p$ and $q$. A natural follow-up is a degree-corrected or pair-random-effect version of the estimator.
  • The sharp change in simulation error around $p-q \approx 0.4$ suggests a cheap validation protocol: benchmark a judge model on a small labeled set of equivalence pairs, and only deploy it for semantic entropy when its judged-equivalence gap clears that threshold.
  • Since the method treats the judge as a black box returning only text, it transfers directly to document-level diversity measurement, not just language-model generation uncertainty.
  • A stronger empirical test than the paper's proposition-set simulations would compare estimated entropy against human-labeled paraphrase equivalence on free-form question-answer outputs, which would reveal whether the stochastic block model is a faithful noise model in realistic settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes semantic spectral entropy, a measure of semantic uncertainty for a set of generated texts. The method asks a pretrained language model to judge pairwise semantic equivalence, records these judgments as edges in a random graph, applies spectral clustering with a prespecified number of clusters K, and computes the entropy of the resulting cluster-size distribution. The main theoretical claims are strong consistency and finite-sample rates for this estimator under a stochastic block model on the language model's pairwise judgments, with an O(1/n) rate in the fixed-K setting. The paper also reports simulations using several off-the-shelf LMs on two synthetic datasets of paraphrased propositions.

Significance. The idea of treating LM equivalence judgments as a random graph and using spectral clustering is natural and potentially valuable, since it offers a route to semantic entropy estimation without token-level probabilities. If the consistency claims were established, this would be a useful formal contribution to the growing literature on semantic uncertainty. The paper also provides a concrete algorithm and code. However, the central proof contains an invalid inequality, and several theorem statements are internally inconsistent, so the main claim of formal convergence is not currently established. The empirical section is too thin to compensate for the theoretical gaps.

major comments (4)
  1. [Appendix B.2.1 / Lemma 3.1] The proof of Lemma 3.1 uses the inequality |Σ_j p̂_j log(p̂_j/p̄_j)| ≤ |Σ_j (p̂_j − p̄_j)/p̄_j|. This inequality is false. For example, with p̄ = (0.5, 0.5) and p̂ = (0.1, 0.9), the left-hand side is |0.1 ln 0.2 + 0.9 ln 1.8| ≈ 0.368, while the right-hand side is |(0.1−0.5)/0.5 + (0.9−0.5)/0.5| = 0. Since Lemma 3.1 is the bridge from misclassification error to entropy error used in Theorems 3.2 and 3.4, the claimed almost-sure consistency is not proven by the manuscript. A termwise bound with absolute values may repair the lemma, but the proof as written is invalid.
  2. [Theorem 3.3 and Corollary 3.3.1] Theorem 3.3 states the condition p = α_n = α_n(q + λ) with α_n ≥ log(n), while Corollary 3.3.1 and the Remark assume α_n = α, a constant. A constant α cannot satisfy α_n ≥ log(n) for large n. Moreover, the proof in Appendix B.3 cites Lei and Rinaldo's condition α_n ≥ log n / n, which is different from the statement in the main text. As written, the theorem is internally contradictory, and the claimed O(1/n) rate in equation (9) does not follow.
  3. [Section 2.3 and Appendix B.2] Algorithm 1 computes the unnormalized graph Laplacian L = D − E, while the theoretical analysis in Appendix B.2 and Theorem B.1 is developed for the normalized Laplacian L = D^{−1/2} E D^{−1/2}. The paper does not reconcile this mismatch or prove that the consistency theorems apply to the algorithm as presented. This gap affects the central claim that Algorithm 1 is strongly consistent.
  4. [Sections 2.2, 2.4, and 3.1] The consistency theorems assume that the number of clusters K is known and fixed (Section 3.1), but Section 2.4 acknowledges that K is generally unavailable and describes a cross-validation method without proving its consistency. Consequently, the theorems do not cover the full procedure that a practitioner would run, and the abstract's phrase 'mild assumptions' overstates the scope. The K-selection step must either be incorporated into the consistency analysis or explicitly excluded from the theoretical claims.
minor comments (5)
  1. [Appendix B.2] The block probability matrix is defined as B_{k1 k2} = p if k1 = k2 and 1 − q otherwise, but the model in Section 2.1, equations (3)–(4), specifies that non-equivalent pairs are judged equivalent with probability q. This is presumably a typo and should be corrected.
  2. [Lemma B.8] The Chernoff bound is written as exp(−n p(C_j)/2), omitting the factor δ² that appears in the subsequent derivation of δ. The final formula for δ is correct, but the intermediate expression is missing a term.
  3. [Appendix B.1 / Proposition 2.1] The proof of reflexivity is garbled: it argues from t_i ∼ t_j to t_i ∼ t_j rather than proving t_i ∼ t_i. The three parts of the equivalence relation proof are not cleanly separated.
  4. [Section 4 / Table 1] The simulation results report averages over only 10 iterations with no variance, standard errors, or confidence intervals. The statement that a phase change occurs at p − q = 0.4 (Figure 1) is asserted without any statistical analysis, which is insufficient for the abstract's claim of 'comprehensive simulation studies'.
  5. [Throughout] There are several typos and notation conflicts: 'Sermantic' in Algorithm 1, 'semantic spectral clustering' used for the entropy measure in Section 5, and the symbol p used both for the edge probability in the stochastic block model and for the cluster probabilities p(C_j), which is confusing in Theorem 3.4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the entropy target is defined from an independent ground-truth partition, and the consistency proof rests on explicit stochastic-block-model assumptions and external spectral-clustering theorems.

full rationale

The paper's central claim is that the plug-in spectral-clustering entropy estimator (Eq. 5) converges to the true semantic entropy (Eq. 1) when the language model's pairwise equivalence judgments follow the stochastic block model in Eqs. 3-4. The true entropy is defined from an independent latent partition C1,...,CK, not from the estimator or from the LM's observed judgments. The derivation chain is: external spectral-clustering consistency for SBMs (Lei and Rinaldo 2015; Su et al. 2019) gives vanishing misclassification error; Lemma 3.1 bounds the entropy error by the misclassification error; Theorems 3.2-3.4 combine these bounds with a Hoeffding/Chernoff argument for the empirical-vs-true entropy gap. Each link is either an external theorem or an explicitly stated statistical assumption; no fitted parameter is renamed as a prediction, and no load-bearing step cites the author's own prior work. The proof of Lemma 3.1 appears to contain an invalid absolute-value inequality, and the alpha_n condition is stated inconsistently between Theorem 3.3 and Corollary 3.3.1, but these are mathematical correctness concerns rather than circularity. Appendix A's limitation about partial text agreement is an acknowledged scope restriction, not a definitional shortcut. The small simulation size and missing variance reporting affect empirical support, not the definitional independence of the estimator from the target.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces a new estimator and a proof, but relies on essentially three unverified inputs: the SBM noise model for LLM judgments, the logical equivalence definition of semantics, and the applicability of external spectral clustering theorems. There are no new physical or probabilistic entities beyond the proposed estimator itself.

free parameters (1)
  • K (number of semantic clusters) = assumed known in Section 3.1; selected via network cross-validation in Section 2.4
    The consistency theorems require K to be specified in advance. In practice K must be chosen, and the theory does not analyze the effect of K misspecification on the consistency results.
assumptions (3)
  • domain assumption The language model's pairwise equivalence judgments follow a stochastic block model: for every equivalent pair, the judgment is Bernoulli(p), and for every non-equivalent pair, Bernoulli(q), independent across pairs (Equations 3 and 4).
    This is the central modeling assumption that connects the randomness of LLM outputs to the spectral clustering consistency theory. It is stated in Section 2.1 but not empirically validated.
  • domain assumption The relation 'ti is true iff tj is true' is the correct notion of semantic equivalence (Proposition 2.1 in Section 2.1).
    The measure's 'true' entropy is defined relative to this equivalence. It only makes sense for truth-apt propositions, and the paper's own limitation section (Appendix A) acknowledges it cannot handle partial agreement.
  • standard math The spectral clustering consistency theorems of Su et al. (2019) and Lei and Rinaldo (2015) apply, including their regularity conditions (Assumptions B.1-B.4 in Appendix B).
    The paper imports these external results, so the final guarantee inherits their unstated regularity conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A statistically consistent measure of semantic uncertainty using Language Models." pith.science (2026). https://pith.science/paper/Z42DMQXC

@misc{pith2026250200507,
  author       = {Pith},
  title        = {Pith review of: A statistically consistent measure of semantic uncertainty using Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z42DMQXC}},
  note         = {Machine review of arXiv:2502.00507}
}
read the original abstract

To address the challenge of quantifying uncertainty in the outputs generated by language models, we propose a novel measure of semantic uncertainty, semantic spectral entropy, that is statistically consistent under mild assumptions. This measure is implemented through a straightforward algorithm that relies solely on standard, pretrained language models, without requiring access to the internal generation process. Our approach imposes minimal constraints on the choice of language models, making it broadly applicable across different architectures and settings. Through comprehensive simulation studies, we demonstrate that the proposed method yields an accurate and robust estimate of semantic uncertainty, even in the presence of the inherent randomness characteristic of generative language model outputs.

Figures

Figures reproduced from arXiv: 2502.00507 by the authors.

Figure 1
Figure 1. A scatter plot of [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Epistemic warranted reliance on generative AI requires, jointly and non-fungibly, epistemic humility, epistemic access, and resistance to epistemic injustice.

  2. When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Propagational Proxy Voting driven by letter entropy and centered reasoning embeddings beats majority by +2.24 pp on non-trivial MMLU-Pro questions without labels or training.

Reference graph

Works this paper leans on

42 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565

    arXiv 2016

  5. [5]

    Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408.04667

    arXiv 2024

  6. [6]

    Kehui Chen and Jing Lei. 2018. Network cross-validation for determining the number of communities in network data. Journal of the American Statistical Association, 113(521):241--251

    work page 2018

  7. [7]

    Davin Choo, Christoph Grunau, Julian Portmann, and V \'a clav Rozhon. 2020. k-means++: few more steps yield constant approximation. In International Conference on Machine Learning, pages 1909--1917. PMLR

    work page 2020

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

Show all 42 references
  1. [9]

    Irving M Copi, Carl Cohen, and Kenneth McMahon. 2016. Introduction to logic. Routledge

    2016

  2. [10]

    Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295--302

    2020

  3. [11]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.276 Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models ...

    2024 doi

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Francesco Esposito. 2024. Programming Large Language Models with Azure Open AI: Conversational Programming and Prompt Engineering with LLMs. Microsoft Press

    2024

  6. [14]

    Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and Andr \'e FT Martins. 2021. Uncertainty-aware machine translation evaluation. arXiv preprint arXiv:2109.06352

    2021 arXiv

  7. [15]

    Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916

    2021 arXiv

  8. [16]

    Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. 1983. Stochastic blockmodels: First steps. Social networks, 5(2):109--137

    1983

  9. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  10. [18]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  11. [19]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962--977

    2021

  12. [20]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  13. [21]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  14. [22]

    Amit Kumar, Yogish Sabharwal, and Sandeep Sen. 2004. A simple linear time (1+/spl epsiv/)-approximation algorithm for k-means clustering in any dimensions. In 45th Annual IEEE Symposium on Foundations of Computer Science, pages 454--462. IEEE

    2004

  15. [23]

    Jing Lei. 2016. A goodness-of-fit test for stochastic block models. The Annals of Statistics, 44(1):401

    2016

  16. [24]

    Jing Lei and Alessandro Rinaldo. 2015. Consistency of spectral clustering in stochastic block models. The Annals of Statistics, pages 215--237

    2015

  17. [25]

    Martin Liebeck. 2018. A concise introduction to pure mathematics. Chapman and Hall/CRC

    2018

  18. [26]

    Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs, 1(9)

    2021

  19. [27]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2024. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research

    2024

  20. [28]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

    2023 arXiv

  21. [29]

    Vasilios Mavroudis. 2024. Langchain

    2024

  22. [30]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. arXiv preprint arXiv:2405.20003

    2024 arXiv

  23. [31]

    Nvidia. 2024. https://docs.nvidia.com/cuda/floating-point/index.html Floating point and ieee 754 compliance for nvidia gpus

    2024

  24. [32]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

    2011

  25. [33]

    Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888--905

    2000

  26. [34]

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2024. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. arXiv preprint arXiv:2407.10457

    2024 arXiv

  27. [35]

    Liangjun Su, Wuyi Wang, and Yichong Zhang. 2019. Strong consistency of spectral clustering for stochastic block models. IEEE Transactions on Information Theory, 66(1):324--338

    2019

  28. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  29. [37]

    A. Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. https://ap...

    2024

  30. [38]

    Chi Wang, Xueqing Liu, and Ahmed Hassan Awadallah. 2023. Cost-effective hyperparameter optimization for large language model generation inference. In International Conference on Automated Machine Learning, pages 21--1. PMLR

    2023

  31. [39]

    Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, and Da-Chang Juan. 2020. Contextual temperature for language modeling. arXiv preprint arXiv:2012.13575

    2020 arXiv

  32. [40]

    YX Rachel Wang and Peter J Bickel. 2017. Likelihood-based model selection for stochastic block models. The Annals of Statistics, 45(2):500

    2017

  33. [41]

    Wikipedia contributors . 2024. https://en.wikipedia.org/wiki/December_3 December 3 . [Online; accessed 03-Dec-2004]

    2024

  34. [42]

    Ludwig Wittgenstein. 2023. Tractatus logico-philosophicus. Linkgua

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.