Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A re-analysis finds that min-p sampling does not improve output quality or diversity over basic, top-k, or top-p sampling.

desk verdict A careful, mostly convincing re-analysis of min-p's evidence base, but the paper's own headline conclusion is stronger than its own significant high-temperature result allows. read the letter →

arxiv 2506.13681 v2 pith:Q3526YJV submitted 2025-06-16 cs.CL cs.LG

Rylan Schaeffer , Joshua Kazdan , Yegor Denisov-Blanch This is my paper
classification cs.CLcs.LG
keywords min-psamplinglanguagemodeldecodingmethodscomparisonhumanevaluationhyperparametersweepLLM-as-a-judgecommunityadoptionreplicationstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper re-examines the four lines of evidence behind the claim that min-p sampling makes language-model output both better and more diverse. It re-analyzes the original human evaluations, sweeps the original NLP benchmarks while giving every sampler the same number of tuned hyperparameters, inspects the LLM-as-a-judge comparisons, and checks the community-adoption numbers. Across all four, the paper finds the evidence does not support min-p's superiority: under matched tuning, min-p is largely indistinguishable from basic, top-k, and top-p sampling. If this conclusion holds, a widely adopted sampling method is being used on the strength of claims that closer analysis does not sustain.

What carries the argument

The argument's technical load is carried by two re-analysis tools. First, a re-running of the original human-evaluation scores with one-sided paired t-tests, a Bonferroni correction, and an Intersection-Union Test, which converts the original 'consistently better' claim into a testable hypothesis about all twelve comparisons. Second, a Best-of-N hyperparameter sweep on GSM8K chain-of-thought: nine models, thirty-one temperatures, three seeds, and six hyperparameter values per sampler, compared by subsampling equal numbers of hyperparameters per sampler so that each method receives an identical tuning budget. The sweep answers the question 'best performance at equal tuning cost' rather than 'best performance when each paper chooses its own grid.'

What would settle it

Run a pre-registered, large-N human evaluation of basic, top-p, top-k, and min-p sampling across the original temperatures and diversity settings, report all scores, and test the 'min-p better in all settings' claim with an Intersection-Union Test; or run a benchmark sweep where min-p is given equal or fewer hyperparameter settings than the baselines on an out-of-distribution task and still beats them by a margin larger than the reported standard errors.

Watch

Extended reading notes

Core claim

The central claim is that min-p sampling does not improve quality, diversity, or the quality-diversity trade-off relative to commonly used samplers. The paper reaches this by re-analyzing the original paper's own data: the human-evaluation results lose significance when the omitted basic-sampler scores are restored and the tests are run correctly (only 1 of 12 comparisons survives Bonferroni correction); the GSM8K benchmark sweep shows min-p matching or underperforming other samplers once each sampler receives the same number of hyperparameter settings; the LLM-as-a-judge results are under-specified, and the reported table chose the higher win rate for min-p and the lower for top-p; and the advertised adoption figures were retracted as unsubstantiated. The paper's conclusion is negative: the published evidence fails to show that min-p outperforms the alternatives.

Load-bearing premise

The benchmark comparison assumes that giving each sampler the same number of swept hyperparameter values is the fair way to compare them, and that the chosen grids (six values per sampler) represent each method fairly; a different fairness rule could change the ordering.

Editorial extensions

If this is right

  • Practitioners should not assume min-p is a strict upgrade; at equal tuning budget, basic, top-k, and top-p samplers perform about as well.
  • Claims of community adoption should be verified at the repository level before being treated as evidence of a method's value.
  • Human-evaluation studies in this area need to report all participants' scores, pre-specify their statistical tests, and account for multiple comparisons.
  • LLM-as-a-judge comparisons should either compare samplers directly or justify why an indirect comparison is valid, given that judge preferences may not be transitive.
  • Sampling-method choice may be a smaller lever than model choice or prompt format for output quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equal-hyperparameter-volume standard, if applied more broadly, could be a template for re-evaluating other decoding-method claims.
  • A reasonable next test is whether min-p's possible benefit at very high temperatures (where all samplers do worse) is worth the cost in absolute performance; the paper leaves that as a possible niche.
  • The pattern of reviewers being swayed by unverified adoption statistics suggests that venue-level practices for checking empirical claims deserve scrutiny.
  • One could extend the Best-of-N sweep to other tasks (e.g., GPQA, summarization, creative writing) to see whether the null result generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript re-examines the evidence presented in Nguyen et al. (2024) for min-p sampling, arguing that the original paper's four lines of evidence—human evaluations, NLP benchmarks, LLM-as-a-Judge evaluations, and community adoption claims—do not support min-p's superiority. The authors re-analyze the original human evaluation data (finding omitted basic-sampler scores and conducting corrected hypothesis tests), add a new human study conducted by the original authors, perform large GSM8K hyperparameter sweeps with equalized hyperparameter counts, investigate the LLM-as-a-Judge methodology and reporting, and document that the GitHub adoption figures were retracted. The paper concludes that min-p does not improve quality, diversity, or the quality-diversity trade-off relative to basic, top-k, and top-p sampling.

Significance. If the conclusions are correct and properly scoped, this is a valuable corrective to a high-profile ICLR 2025 oral paper. The re-analysis of the original human evaluation data is concrete and reproducible: the discovery that one-third of the human scores were omitted, the corrected Bonferroni and IUT tests, and the documentation of the Table 3(b) reporting inconsistency are substantive contributions. The paper also ships public code and W&B sweeps, which strengthens the benchmark portion. The documented retraction of the community adoption numbers is an important scientific-record correction. However, the paper's central claim is overstated relative to its own evidence, so its significance depends on whether the conclusions are narrowed and the internal inconsistency in the treatment of the significant τ=3.0 quality result is resolved. As written, the blanket 'does not improve' claim in the abstract is not supported even by the paper's own Table 1.

major comments (3)
  1. [Sec. 2.2/Table 1 and Sec. 6] The abstract and Section 1 claim, without qualification, that min-p "does not improve quality or diversity or the trade-off between quality and diversity." Yet Table 1 reports a Bonferroni-corrected significant win for min-p over basic on quality at temperature 3.0 (t=3.13, p=.001). The paper's response in Section 2.4—that this advantage occurs only under conditions with lower absolute scores—is a value judgment about which operating points matter, not a statistical refutation of a relative improvement claim. Furthermore, Section 6 itself concedes that "min-p sampling can sometimes provide a benefit at higher temperatures," which directly contradicts the unconditional wording of the abstract. The conclusion should be narrowed, e.g., to "no Pareto improvement over the best achievable quality-diversity frontier across all temperatures" or "no consistent improvement across all settings," and the abstract and Section 1 revised accordingly. As written, the central claim is internally inconsistent with the paper's own accepted significant result.
  2. [Sec. 3.1] The abstract states that "comprehensively sweeping the original paper's NLP benchmarks reveals min-p does not surpass baselines when controlling for the number of hyperparameters," but the manuscript only sweeps GSM8K Chain-of-Thought. The original paper's NLP benchmark evaluations also included GPQA (5-shot), which is not evaluated here. The manuscript acknowledges this limitation ("we only evaluated GSM8K CoT") but still draws a benchmark-wide conclusion. The claim should be explicitly restricted to GSM8K CoT, or GPQA must be swept before the broad statement is made.
  3. [Sec. 3.1, Best-of-N analysis] The core benchmark argument depends on the fairness criterion of equalizing the number of hyperparameter settings per sampler. This criterion is asserted, not justified. The hyperparameter grids (six values per sampler, "taken from the original paper; some were lightly edited") are hand-picked and may not be representative of each sampler's practical tuning range. Moreover, for basic sampling, temperature is the only hyperparameter, so the "Best-of-N" subsampling with N ranging up to 100 is not well-defined unless sampling with replacement is explicitly specified; the manuscript does not specify this detail. Without a justification of the fairness criterion or a sensitivity analysis over alternative grids (e.g., controlling for compute or for best score per hyperparameter dimension), the benchmark-based rejection of min-p's superiority is not fully established.
minor comments (4)
  1. [Sec. 2.4] The claim that the value 7.80 in the original Table 15 should be 5.80 is specific and testable; given that the paper presents this as a typo in the original authors' work, it would strengthen the report to include the relevant row from the publicly posted data in a supplementary table.
  2. [Sec. 5.1] The phrase "in our opinion, were preposterous" is subjective and not needed to make the point that the numbers were unverified; the subsequent analysis of GitHub repositories is sufficient. Consider removing such value-laden wording.
  3. [Sec. 2.3] The manual annotation of qualitative responses is a reasonable exploratory step, but the paper does not report inter-annotator agreement or whether the annotation procedure was pre-registered. A sentence acknowledging this subjective component would increase methodological transparency.
  4. [Sec. 6] The section "What Went Wrong During the ICLR 2025 Review Process?" goes beyond the paper's stated scientific scope and includes speculative statements about reviewer behavior. While the review-process critique is motivated by the evidence, the presentation would be more measured if the section focused on the documented failures rather than on evaluative comments about the reviewers.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the negative conclusion rests on independent reanalysis of external data and new sweeps, not on a self-referential derivation or fitted prediction.

full rationale

This paper is a critical re-analysis of an external paper's claims about min-p sampling. Its central conclusion—that the original evidence fails to show min-p improves quality, diversity, or their trade-off—is supported by (1) re-running statistical tests on the original authors' released human-evaluation data, (2) a new human evaluation conducted by the original authors in response to the critique, (3) an independent, large-scale GSM8K hyperparameter sweep, and (4) inspection of the original authors' public code and Telegram-reported scores. None of these steps defines the target conclusion into existence. The paper does not fit parameters and then rename the fit a prediction; the closest step is the benchmark 'best-of-N controlling for hyperparameter volume' analysis, but that is a stated fairness criterion, not a tautology. The self-citations (e.g., Schaeffer et al. 2025a,b, Hughes et al. 2024) appear only as methodological references for best-of-N sampling and for the gold-standard status of human evaluation; they are not load-bearing for the empirical claim about min-p. The discussion section explicitly concedes that 'min-p sampling can sometimes provide a benefit at higher temperatures' and that 'new evidence might lead to different conclusions,' which further indicates the conclusion is an inductive assessment of evidence rather than a circular derivation. No step reduces by construction to its own inputs, and no uniqueness theorem or self-citation chain forces the result. The only minor observation is that the authors cite their own prior work in methodological contexts, but this is non-load-bearing and does not raise the circularity score beyond a minimal level.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims do not introduce new entities or fitted model parameters. The main hand-made choices are the hyperparameter grids and sweep ranges that define the 'equal tuning budget' comparison, plus assumptions about the authenticity of the original authors' released data and the validity of human evaluation as ground truth.

free parameters (4)
  • Hyperparameter grids per sampler = top-k: 10,30,50,100,150,200; top-p: 0.99,0.98,0.95,0.9,0.8,0.7; min-p: 0.01,0.02,0.05,0.1,0.2,0.3
    Hand-chosen from the original paper 'lightly edited'; the equal-hyperparameter-volume comparison and the paper's rejection of min-p superiority depend on these values being a fair representation of each sampler's tuning space.
  • Temperature sweep = 0.0 to 3.0 in 0.1 increments (31 values)
    Chosen for the GSM8K sweep; the best-score envelope for every sampler depends on this range and resolution.
  • Sampling seeds = {0,1,2}
    Three seeds averaged in the benchmark analysis; conclusions could shift with more seeds, though this is a reasonable compute-limited choice.
  • Best-of-N subsampling repeats = 150
    Number of random hyperparameter subsets used to estimate best achievable performance; affects the smoothness of the comparison.
assumptions (4)
  • domain assumption Human evaluations are the gold standard that can adjudicate min-p's quality and diversity claims.
    Invoked in Sec. 2 with multiple citations (Van Der Lee et al., Roller et al., etc.); if human judgment is not a valid gold standard, the central human-eval refutation loses force.
  • domain assumption The released scores and code from the original authors are the actual records of the original experiments.
    The re-analysis of the omitted basic-sampler data and the LLM-as-a-Judge score discrepancy relies on the public data, the first author's GitHub, and a Telegram link (Sec. 4.3). These artifacts are treated as authentic and complete.
  • ad hoc to paper Equalizing the number of hyperparameter settings is a fair way to compare samplers.
    The Best-of-N analysis (Sec. 3.1) defines fairness by hyperparameter volume; this choice is not externally grounded and is the load-bearing assumption behind the benchmark conclusion.
  • standard math Absence of statistically significant superiority in 12 t-tests implies no consistent superiority claim is supported.
    The use of Bonferroni and Intersection-Union tests is standard, but the interpretation that 'insufficient evidence to support consistent outperformance' is only about evidence, not about actual equivalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models." pith.science (2026). https://pith.science/paper/Q3526YJV

@misc{pith2026250613681,
  author       = {Pith},
  title        = {Pith review of: Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3526YJV}},
  note         = {Machine review of arXiv:2506.13681}
}
read the original abstract

Sampling from language models impacts the quality and diversity of outputs, affecting both research and real-world applications. Recently, Nguyen et al. 2024's "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs" introduced a new sampler called min-p, claiming it achieves superior quality and diversity over established samplers such as basic, top-k, and top-p sampling. The significance of these claims was underscored by the paper's recognition as the 18th highest-scoring submission to ICLR 2025 and selection for an Oral presentation. This paper conducts a comprehensive re-examination of the evidence supporting min-p and reaches different conclusions from the original paper's four lines of evidence. First, the original paper's human evaluations omitted data, conducted statistical tests incorrectly, and described qualitative feedback inaccurately; our reanalysis demonstrates min-p did not outperform baselines in quality, diversity, or a trade-off between quality and diversity; in response to our findings, the authors of the original paper conducted a new human evaluation using a different implementation, task, and rubric that nevertheless provides further evidence min-p does not improve over baselines. Second, comprehensively sweeping the original paper's NLP benchmarks reveals min-p does not surpass baselines when controlling for the number of hyperparameters. Third, the original paper's LLM-as-a-Judge evaluations lack methodological clarity and appear inconsistently reported. Fourth, community adoption claims (49k GitHub repositories, 1.1M GitHub stars) were found to be unsubstantiated, leading to their removal; the revised adoption claim remains misleading. We conclude that evidence presented in the original paper fails to support claims that min-p improves quality, diversity, or a trade-off between quality and diversity.

Figures

Figures reproduced from arXiv: 2506.13681 by the authors.

Figure 1
Figure 1. Visualizing Human Evaluators’ Scores from Nguyen et al. (2024)’s Data Demonstrates Min-p Does Not “Consistently" Outperform Other Samplers. Rather, the original paper’s data suggest min-p is largely indistinguishable from other samplers based on 95% confidence intervals. Recently, the paper “Turning Up the Heat: Min-P Sampling for Creative and Coherent LLM Outputs" (Nguyen et al., 2024) introduced a new sampling met… view at source ↗
Figure 2
Figure 2. Manual Annotation of Human Evaluators’ Qualitative Responses Fail to Support Claim that Min-P Was the Preferred Sampler. We manually annotated responses from human annotators regarding their preferred sampler(s) at the end of the original paper’s study. The responses suggest min-p was not the most preferred sampler. We provide example responses in Sec. 2.3. diversity), the paper pooled data across all settings and p… view at source ↗
Figure 3
Figure 3. New Human Evaluation Study Suggests Min-p Does Not Outperform Baselines in Quality, in Diversity or in a Pareto-Optimal Tradeoff Between Quality and Diversity. Visu￾alization of scores from Nguyen et al. (2024)’s second human experiment. Min-p’s performance advantage relative to basic and top-p sampling is observed in conditions (e.g., higher temperatures) where absolute quality and absolute diversity scores across … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Min-P Does Not Consistently Outperform Other Samplers on GSM8K When Con￾trolling For Hyperparameter Volume. In our first analysis, we measured how the maximum Exact Match (Strict) for each sampler improves as the number of hyperparameters increases. Basic sampling has …
Figure 5
Figure 5. Figure 5: Min-P Does Not Consistently Outperform Other Samplers on GSM8K When Control￾ling For Hyperparameter Volume. In our second analysis, we measured how the difference between min-p’s highest score and the best non-min-p sampler’s highest score changes as the number of swep…
Figure 6
Figure 6. Figure 6: Nguyen et al. (2024)’s LLM-As-A-Judge Evaluations Suggest Min-p Typically Matches Other Samplers Despite 2× to 10× More Hyperparameter Tuning. Left: Nguyen et al. (2024) swept min-p with more than twice as many hyperparameters as top-p and more than ten times as many h…
Figure 7
Figure 7. Figure 7: Min-P Does Not Consistently Outperform Other Samplers on GSM8K When Control￾ling For Hyperparameter Volume. We reran our GSM8K sweep using “standard" formatting rather than “Llama" formatting and observed qualitatively similar data. 17 [PITH_FULL_IMAGE:figures/full_fi…
Figure 8
Figure 8. Figure 8: Min-P Does Not Consistently Outperform Other Samplers on GSM8K When Control￾ling For Hyperparameter Volume. We reran our GSM8K sweep using “standard" formatting rather than “Llama" formatting and observed qualitatively similar data. 18 [PITH_FULL_IMAGE:figures/full_fi…
Figure 9
Figure 9. Figure 9: GSM8K Scores By Model, Sampler and Sampler Hyperparameters. Many models achieve their highest scores at low temperatures across samplers. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ML conferences should create an official peer-reviewed track dedicated to refuting and critiquing previously published work.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    A learning algorithm for boltzmann machines

    David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9 0 (1): 0 147--169, 1985

    1985

  2. [2]

    Mirostat: A neural text decoding algorithm that directly controls perplexity

    Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint arXiv:2007.14966, 2020

    arXiv 2007

  3. [3]

    Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick,...

    arXiv 2024

  4. [4]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132

    arXiv 2024

  5. [5]

    All that’s ‘human’is not gold: Evaluating human evaluation of generated text

    Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that’s ‘human’is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long ...

    work page 2021

  6. [6]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  7. [7]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36: 0 30039--30069, 2023

    2023

  8. [8]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018

    arXiv 2018

Show all 34 references
  1. [9]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Truncation sampling as language model desmoothing

    John Hewitt, Christopher D Manning, and Percy Liang. Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 3414--3427, 2022

    2022

  4. [12]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020

    2020

  5. [13]

    Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions

    David M Howcroft, Anya Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In 13th...

    2020

  6. [14]

    Best-of-n jailbreaking, 2024

    John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. Best-of-n jailbreaking, 2024. URL https://arxiv.org/abs/2412.03556

    2024 arXiv

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  8. [16]

    Smith, and Daniel S

    Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. Genie: Toward reproducible and standardized human evaluation for text generation, 2022. URL https://arxiv.org/abs/2101.06561

    2022 arXiv

  9. [17]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  10. [18]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  11. [20]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  12. [21]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  13. [22]

    Open-domain conversational agents: Current progress, open problems, and future directions, 2020

    Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, Pratik Ringshia, Kurt Shuster, Eric Michael Smith, Arthur Szlam, Jack Urbanek, and Mary Williamson. Open-domain conversational agents: Current...

    2020 arXiv

  14. [23]

    How do large language monkeys get their power (laws)?, 2025 a

    Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. How do large language monkeys get their power (laws)?, 2025 a . URL https://arxiv.org/abs/2502.17578

    2025 arXiv

  15. [24]

    Chatterji, Vedanuj Goswami, Sergey Edunov, Dieuwke Hupkes, Sanmi Koyejo, and Sharan Narang

    Rylan Schaeffer, Punit Singh Koura, Binh Tang, Ranjan Subramanian, Aaditya K Singh, Todor Mihaylov, Prajjwal Bhargava, Lovish Madaan, Niladri S. Chatterji, Vedanuj Goswami, Sergey Edunov, Dieuwke Hupkes, Sanmi Koyejo, and Sharan Narang. Correlating and predicting human evaluat...

    2025 arXiv

  16. [25]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33: 0 3008--3021, 2020

    2020

  17. [26]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  18. [27]

    Best practices for the human evaluation of automatically generated text

    Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pp.\ 355--368, 2019

    2019

  19. [28]

    Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025

    Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025

    2025 arXiv

  20. [29]

    Investigating non-transitivity in llm-as-a-judge, 2025

    Yi Xu, Laura Ruis, Tim Rocktäschel, and Robert Kirk. Investigating non-transitivity in llm-as-a-judge, 2025. URL https://arxiv.org/abs/2502.14074

    2025 arXiv

  21. [30]

    A careful examination of large language model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems,...

    2024

  22. [31]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  23. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  24. [33]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  25. [34]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  26. [35]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.