Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

On Robustness and Reliability of Benchmark-Based Evaluation of LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Paraphrasing benchmark questions drops LLM accuracy significantly while preserving model rankings.

desk verdict Large, reproducible study showing LLM rankings survive paraphrasing while absolute scores drop; main caveat is unvalidated paraphrase fidelity, which the authors themselves flag. read the letter →

arxiv 2509.04013 v1 pith:ZBGWQIP3 submitted 2025-09-04 cs.CL cs.AI

Riccardo Lunardi , Vincenzo Della Mea , Stefano Mizzaro , Kevin Roitero This is my paper
classification cs.CLcs.AI
keywords LLMevaluationbenchmarkreliabilityparaphraserobustnesslinguisticvariabilitymultiple-choicequestionansweringrankingstabilityconsistencydatacontamination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are usually scored on benchmarks with fixed, standardized wording. This paper asks whether those scores hold up when the same questions are rephrased. Across six multiple-choice benchmarks and 34 models, it finds that relative rankings barely move (Kendall's tau above 0.9) while absolute accuracy falls: most models score worse on paraphrased inputs than on the original wording. The authors read this as evidence that current benchmarks provide a valid comparative ordering but overestimate real-world robustness to linguistic variability. They also find that consistency across paraphrases is only a meaningful sign of correctness for larger models; smaller models can be consistently wrong.

What carries the argument

The load-bearing mechanism is the paraphrase perturbation protocol. For each original question, an automatic paraphrasing model generates five rewrites that preserve meaning, avoid negations, and keep question form and answer order fixed; every LLM then answers all six versions under zero-shot, top-1 decoding. Accuracy is compared through 1,000 sampled paraphrase sets, producing a boxplot of what the model would score under realistic wording variability, and ranking stability is measured with Kendall's tau between original and median paraphrased accuracy.

What would settle it

Take a random sample of 500 original questions per benchmark and their five paraphrases, and have human judges who do not know the model answers answer both versions. If humans find the paraphrases equally easy, or if the paraphrases are judged to change the intended answer, then the observed model accuracy drop would be an artifact of the paraphrase generation rather than evidence about model robustness.

Watch

Extended reading notes

Core claim

The paper's central claim is that the standard practice of reporting one accuracy number on fixed benchmark wording produces a misleading picture. When each of 52,966 questions across six multiple-choice benchmarks is automatically reworded into five paraphrases and answered by 34 instruction-tuned large language models, the relative ordering of model accuracies is preserved (Kendall's tau above 0.9 on every benchmark), but the absolute accuracy distribution shifts downward: on MMLU, 28 of 34 models score worse under paraphrased samples than on the original wording, 5 overlap, and 1 improves. The authors interpret this decoupling as evidence that current benchmarks are reliable for comparing

Load-bearing premise

The generated paraphrases preserve the original questions' meaning and difficulty, so the accuracy drops measure model fragility to wording rather than changes in the task itself.

Editorial extensions

If this is right

  • Reported benchmark numbers should be read as upper bounds on performance under fixed wording, not as expected accuracy in applications where users rephrase questions.
  • Existing leaderboards remain useful for relative comparison; a model ranked higher on original wording will almost always rank higher under paraphrases.
  • Robustness-aware evaluation should report accuracy distributions or paraphrase sensitivity alongside single scores.
  • Consistency-under-paraphrase should not be used as a standalone quality metric, especially for small models, where it can indicate stubbornness rather than understanding.
  • The negative correlation between benchmark release date and the number of models in the 'Over' category is consistent with data contamination in older benchmarks; fresh paraphrased test sets can expose memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If paraphrase sensitivity were added to leaderboards as a second axis, the Pareto frontier would change: some models with lower original accuracy but high consistency could be preferable for deployment.
  • A direct contamination test is available: compare 'Over' counts for models whose pretraining cutoff predates a benchmark's release against models trained after it; the paper's release-date correlation predicts a clear difference.
  • The protocol can be extended to a continuous robustness metric, such as accuracy variance across paraphrases, and to human-authored paraphrases, which may produce larger drops than automatic ones.
  • For open-ended generation tasks, the same decoupling might not hold, because correctness judgments themselves are noisy; the multiple-choice setting is the cleanest place to observe the effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper systematically examines whether LLM benchmark scores are robust to paraphrased question inputs. Using GPT-4o mini, the authors generate five paraphrases for each question in six multiple-choice benchmarks (ARC-C, HellaSwag, MMLU, OpenBookQA, RACE, SciQ), yielding roughly 265k paraphrased instances. They evaluate 34 instruction-tuned LLMs in a zero-shot, top-1 decoding setting. The central finding is that while model rankings remain stable across paraphrased inputs (Kendall's tau > 0.9 for all benchmarks), absolute accuracy drops substantially: in Figure 6, most models are categorized as 'Over,' meaning their original-question accuracy exceeds their paraphrase distribution. The paper interprets this as evidence that LLMs are brittle to linguistic variation and that current benchmarks overestimate real-world robustness. It also reports consistency analyses, showing that a substantial fraction of questions elicit different answers across paraphrases, and a size-dependent correlation between accuracy and consistency.

Significance. If the central claim holds, the paper makes a valuable empirical contribution by scaling paraphrase-robustness evaluation to 34 models, six benchmarks, and over 260k paraphrased questions. The public release of data, code, predictions, and plots is a strength, as is the breadth of the evaluation. The finding that rankings are stable while absolute scores drop is consistent with a growing body of work on prompt sensitivity and has concrete implications for how benchmark scores should be reported and interpreted. However, the significance depends critically on the unvalidated assumption that the automatically generated paraphrases preserve both meaning and difficulty. The paper itself acknowledges this in Section 5 ('Further validation of the generated paraphrases is needed'), and the internal checks in Section 4.1.3 do not establish semantic equivalence. The study is therefore best viewed as a large-scale, reproducible probe that currently supports a conditional conclusion.

major comments (4)
  1. [Section 3.3 and Section 5] The central claim that absolute accuracy drops because of LLM brittleness rests on the assumption that the GPT-4o mini generated paraphrases preserve semantic content and question difficulty. The only validation offered (Section 4.1.3) is indirect: adding paraphrases increases answer diversity, reverse order gives similar curves, and first/last paraphrase accuracies are equal on average. None of these checks establishes semantic equivalence; a systematic difficulty shift (e.g., adding or removing clues, changing implicit reasoning requirements) would produce exactly the same observable patterns. The paper's own Section 5 states that 'Further validation of the generated paraphrases is needed.' This is a load-bearing gap because if the paraphrases are not semantically equivalent, the accuracy drop is an artifact of the generation process, not evidence of LLM brittleness.
  2. [Section 3.1] The methodology combines test and training instances: 'we present results over both test and sampled training data, treating them uniformly in our evaluation.' This is problematic for a claim about generalization. Models may have seen training instances during pretraining or instruction tuning, especially for older benchmarks that the paper itself suspects contamination (Section 4.2). Figure 6, however, states that sampling was performed 'only on the test set and on the validation set,' creating an inconsistency across the paper. The central analyses should be run on held-out test/validation data only, and the paper should clarify which figures and numbers use the combined set.
  3. [Section 4.2 and Figure 6] The abstract and conclusions use the word 'significantly' for the absolute performance drop, but no statistical significance test is reported. The Over/In/Under categories are defined by whether the original accuracy falls outside the interquartile range of the 1,000-sample paraphrase distribution, which is not a hypothesis test. To support 'significant' decline, the authors should provide paired tests (e.g., bootstrap confidence intervals for per-model differences, or a Wilcoxon signed-rank test across models) and report effect sizes for each benchmark.
  4. [Section 3.3 and Table 2] The paraphrases are generated by GPT-4o mini, which is also one of the 34 evaluated models (model 5 in Table 2). This does not make the experiment circular, but it is a potential confound: the paraphrase distribution may reflect GPT-4o mini's stylistic preferences, and GPT-4o mini itself may have a systematic advantage or disadvantage on these paraphrases. The paper should discuss this and ideally demonstrate robustness by generating paraphrases with a second, independent model (e.g., an open-source model) and repeating a subset of the experiments.
minor comments (5)
  1. [Section 4.1.3] The sentence 'This means that paraphrasing indeed introduces increasing confusion for LLMs' is an interpretive leap. The observed increase in answer diversity is consistent with meaningful linguistic variation, but it is also consistent with paraphrase-induced task alteration. Please rephrase to separate observation from interpretation.
  2. [Section 3.4] The claim that 'we observed no changes in model responses when modifying the phrasing of the prompt' is not backed by a systematic experiment. Either provide details of this check or soften the claim.
  3. [Table 2 / Figure 6] Models 30-34 show identical or near-identical accuracies (0.22-0.25) across all benchmarks, which likely indicates degenerate output behavior (e.g., always selecting the same option). The paper excludes them from Figure 6 but includes them in earlier analyses. This should be discussed explicitly, as these models can affect the accuracy-consistency correlation in Section 4.1.2.
  4. [Figure 7] The correlations with benchmark release date are all non-significant (p = 0.53, 0.60, 0.33), yet the conclusion states 'We also uncover some evidence of potential data contamination.' The text should clearly mark this as a speculative conjecture, not a finding, and avoid the word 'evidence' without stronger support.
  5. [Section 4.2] The text reports average accuracy of 0.54 for both first and last paraphrase. This is used to support semantic equivalence, but equality of average accuracy does not imply that individual questions have the same difficulty. Please clarify the logical scope of this observation.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: paraphrase semantic fidelity is validated using the same model-response variability that the paper interprets as brittleness; the measured accuracy drop is otherwise an independent empirical result.

  1. self definitional [Section 4.1.3 (Evaluation of Generated Paraphrases)]
    "The lines are well-separated and ordered as expected, i.e., from '0' to '0,1,2,3,4,5': this confirms that the generated paraphrases introduce meaningful variability while preserving enough semantic fidelity to challenge the models."

    The paper's central conclusion is that LLMs are brittle to linguistic variation because accuracy drops on paraphrased questions. That interpretation requires the paraphrases to preserve meaning and difficulty. The only evidence offered for this premise is that model answer diversity increases monotonically as more paraphrases are added, and that reversing order gives a similar curve. But increasing answer diversity is exactly the phenomenon that the conclusion attributes to model brittleness. Thus the input (paraphrase semantic fidelity) is validated by the same output (model answer instability) whose interpretation is at stake; the validation does not independently rule out the alternative that paraphrases altered the task or its difficulty. The paper itself concedes in Section 5 that 'Fu

  2. self definitional [Section 4.2 (RQ2, paragraph on first/last paraphrase accuracy)]
    "On average, the accuracy across all models is 0.54 for both the first and the last paraphrase, underscoring the semantic equivalence of the variants discussed in Section 4.1.3."

    Equal average accuracy on the first and fifth paraphrase is presented as evidence of semantic equivalence. However, two sets of paraphrases that are semantically different in different ways can also yield equal average accuracy; this check only rules out a systematic order/position effect, not semantic drift or difficulty changes. The inference uses model accuracy — the very dependent variable of the study — to certify the semantic validity of the input, while the study's conclusion is also about model accuracy across paraphrases. This is a secondary instance of the same circular validation pattern.

full rationale

The paper is an empirical evaluation rather than a formal derivation, so most of the circularity patterns (self-definitional equations, fitted parameters renamed as predictions, imported uniqueness theorems) do not apply. No model parameters are fit, no predictions are generated from a theoretical model, and the headline measurements — accuracy on original vs. paraphrased questions and Kendall's tau above 0.9 — are directly observed and independently reproducible from the released data. However, the paper's central claim that accuracy drops reflect model brittleness rather than paraphrase artifacts depends entirely on the assumption that the GPT-4o-mini-generated paraphrases preserve semantic content and difficulty. The validation of that assumption in Section 4.1.3 is circular: it uses the models' increasing answer diversity across paraphrases as evidence for semantic fidelity, but that same diversity is the phenomenon the paper interprets as lack of robustness. The equal first/last paraphrase accuracy check in Section 4.2 is also insufficient because it only controls for order effects. The paper explicitly acknowledges the need for further paraphrase validation, which confirms the load-bearing nature of this unvalidated input. There is also a self-citation to the authors' prior work [23], but it is used only as background support and is not load-bearing. Overall, the core empirical result is not reduced by construction, but the interpretive claim of 'benchmarks overestimate robustness' is partially circular in its current validation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper depends on benchmark ground truth, on unvalidated paraphrase equivalence, on the chosen zero-shot protocol, and on the representativeness of its 34-model sample. No novel theoretical entities are introduced; all free parameters are design choices rather than fitted values.

assumptions (4)
  • domain assumption Ground-truth labels in ARC-C, HellaSwag, MMLU, OpenBookQA, RACE, and SciQ are correct for their intended benchmarks.
    The paper never verifies these labels; evaluation accuracy is computed against them (Section 3.4).
  • domain assumption The five GPT-4o-mini paraphrases preserve the semantic content and difficulty of each original question.
    This is the load-bearing premise for interpreting accuracy drops as robustness failures; the paper defers validation to future work (Section 5) and relies on indirect consistency evidence (Section 4.1.3).
  • domain assumption Zero-shot multiple-choice accuracy with deterministic top-1 decoding is a meaningful measure of LLM effectiveness for these benchmarks.
    All results are computed under this protocol (Section 3.4); the paper does not compare against other protocols such as few-shot or free-form generation.
  • domain assumption The 34 selected models are representative of current instruction-tuned LLMs across sizes.
    Results are summarized as averages over these models (Section 3.2); claims about 'LLMs' generally generalize from this sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Robustness and Reliability of Benchmark-Based Evaluation of LLMs." pith.science (2026). https://pith.science/paper/ZBGWQIP3

@misc{pith2026250904013,
  author       = {Pith},
  title        = {Pith review of: On Robustness and Reliability of Benchmark-Based Evaluation of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBGWQIP3}},
  note         = {Machine review of arXiv:2509.04013}
}
read the original abstract

Large Language Models (LLMs) effectiveness is usually evaluated by means of benchmarks such as MMLU, ARC-C, or HellaSwag, where questions are presented in their original wording, thus in a fixed, standardized format. However, real-world applications involve linguistic variability, requiring models to maintain their effectiveness across diverse rewordings of the same question or query. In this study, we systematically assess the robustness of LLMs to paraphrased benchmark questions and investigate whether benchmark-based evaluations provide a reliable measure of model capabilities. We systematically generate various paraphrases of all the questions across six different common benchmarks, and measure the resulting variations in effectiveness of 34 state-of-the-art LLMs, of different size and effectiveness. Our findings reveal that while LLM rankings remain relatively stable across paraphrased inputs, absolute effectiveness scores change, and decline significantly. This suggests that LLMs struggle with linguistic variability, raising concerns about their generalization abilities and evaluation methodologies. Furthermore, the observed performance drop challenges the reliability of benchmark-based evaluations, indicating that high benchmark scores may not fully capture a model's robustness to real-world input variations. We discuss the implications of these findings for LLM evaluation methodologies, emphasizing the need for robustness-aware benchmarks that better reflect practical deployment scenarios.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

    cs.AI 2026-05 conditional novelty 6.0 of 10

    Models flip between correct and incorrect answers on over 23% of questions under meaning-preserving paraphrases, so single-prompt accuracy overstates reliable knowledge.

Reference graph

Works this paper leans on

51 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Austin et al

    J. Austin et al. Program synthesis with large language models. arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    Bai et al

    G. Bai et al. Mt-bench-101: A fine-grained benchmark for evaluat- ing large language models in multi-turn dialogues. arXiv:2402.14762, 2024

    arXiv 2024

  3. [3]

    Bailey, N

    P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yil- maz. Relevance assessment: are judges exchangeable and does it matter. In Proc. of SIGIR, page 667–674, 2008

    work page 2008

  4. [4]

    Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020

    work page 2020

  5. [5]

    Burnell et al

    R. Burnell et al. Rethink reporting of evaluation results in AI. Science, 380(6641):136–138, 2023

    work page 2023

  6. [6]

    Carterette, J

    B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proc. of SIGIR, page 268–275, 2006

    work page 2006

  7. [7]

    Carterette, A

    B. Carterette, A. Bah, and M. Zengin. Dynamic test collections for retrieval evaluation. In Proc. of the 2015 International Conference on The Theory of Information Retrieval, page 91–100, 2015

    work page 2015

  8. [8]

    Chen et al

    M. Chen et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021

    arXiv 2021

Show all 51 references
  1. [9]

    E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y . Choi, P. Liang, and L. Zettlemoyer. QuAC: Question answering in context. In Proc. of the 2018 EMNLP, pages 2174–2184, 2018

    2018

  2. [10]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  3. [11]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021

    2021 arXiv

  5. [13]

    Frohberg and F

    J. Frohberg and F. Binder. CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models. In Proc. of 13th LREC, pages 2126–2140, 2022

    2022

  6. [14]

    Guiver, S

    J. Guiver, S. Mizzaro, and S. Robertson. A few good topics: Experi- ments in topic set reduction for retrieval evaluation. TOIS, 27(4), 2009

    2009

  7. [15]

    D. Harman. Information retrieval evaluation . Morgan & Claypool, 2011

    2011

  8. [16]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring Massive Multitask Language Understanding. ICLR, 2021

    2021

  9. [17]

    Hendrycks et al

    D. Hendrycks et al. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874, 2021

    2021 arXiv

  10. [18]

    Kim et al

    S. Kim et al. Prometheus: Inducing fine-grained evaluation capability in language models. In ICLR, 2023

    2023

  11. [19]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large Lan- guage Models are Zero-Shot Reasoners. In NeurIPS, volume 35, pages 22199–22213, 2022

    2022

  12. [20]

    G. Lai, Q. Xie, H. Liu, Y . Yang, and E. Hovy. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proc. of the 2017 EMNLP, pages 785–794, 2017

    2017

  13. [21]

    Lewis and M

    M. Lewis and M. Mitchell. Evaluating the robustness of analogical reasoning in large language models. arXiv:2411.14215, 2024

    2024 arXiv

  14. [22]

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for co...

    2021 arXiv

  15. [23]

    Lunardi, D

    R. Lunardi, D. La Barbera, and K. Roitero. The Elusiveness of Detect- ing Political Bias in Language Models. In Proc. 33rd ACM Int. Conf. on Information and Knowledge Management, page 3922–3926, 2024

    2024

  16. [24]

    Mihaylov, P

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answer- ing. In Proc. of the 2018 EMNLP, pages 2381–2391, 2018

    2018

  17. [25]

    Mitchell

    M. Mitchell. How do we know how smart AI systems are? Science, 381 (6654):eadj5957, 2023

    2023

  18. [26]

    Mitchell

    M. Mitchell. Abstraction and analogy in AI. Annals of the New York Academy of Sciences, 1524(1):17–21, 2023

    2023

  19. [27]

    Nguyen, M

    T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. MS MARCO: A human generated machine reading com- prehension dataset. CoRR, abs/1611.09268, 2016

    2016 arXiv

  20. [28]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kel- ton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instruction...

    2024

  21. [29]

    Parry, M

    A. Parry, M. Fröbe, H. Scells, F. Schlatt, G. Faggioli, S. Zerhoudi, S. MacAvaney, and E. Yang. Variations in relevance judgments and the shelf life of test collections. arXiv:2502.20937, 2025

    2025 arXiv

  22. [30]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conf. on Language Modeling, 2024

    2024

  23. [31]

    Reuel-Lamparth, A

    A. Reuel-Lamparth, A. Hardy, C. Smith, M. Lamparth, M. Hardy, and M. J. Kochenderfer. BetterBench: Assessing AI Benchmarks, Uncover- ing Issues, and Establishing Best Practices. Advances in Neural Infor- mation Processing Systems, 37:21763–21813, 2024

    2024

  24. [32]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9):99–106, 2021

    2021

  25. [33]

    E. S. Salido, J. Gonzalo, and G. Marco. None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple- Choice LLM Evaluation Benchmarks. arXiv:2502.12896, 2025

    2025

  26. [34]

    Sanderson

    M. Sanderson. Test collection based evaluation of information retrieval systems. Foundations and Trends in IR, 4(4):247–375, 2010

    2010

  27. [35]

    Sclar, Y

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024

    2024

  28. [36]

    Sparck Jones and C

    K. Sparck Jones and C. J. Van Rijsbergen. Information retrieval test collections. Journal of documentation, 32(1):59–75, 1976

    1976

  29. [37]

    C. E. Stevenson, A. Pafford, H. L. van der Maas, and M. Mitchell. Can large language models generalize analogy solving like people can? arXiv:2411.02348, 2024

    2024

  30. [38]

    E. M. V oorhees. Variations in relevance judgments and the measure- ment of retrieval effectiveness. InProc. of SIGIR, pages 315–323, 1998

    1998

  31. [39]

    E. M. V oorhees. Evaluation by highly relevant documents. In Proc. 24th SIGIR, page 74–82, 2001

    2001

  32. [40]

    E. M. V oorhees. The philosophy of information retrieval evaluation. In Workshop of the cross-language evaluation forum for european lan- guages, pages 355–370, 2001

    2001

  33. [41]

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le. Finetuned language models are zero-shot learners. arXiv:2109.01652, 2021

    2021 arXiv

  34. [42]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proc. of the 36th NeurIPS, 2022

    2022

  35. [43]

    Wei et al

    J. Wei et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  36. [44]

    Welbl, N

    J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing Multiple Choice Science Questions. In Proc. of the 3rd Workshop on Noisy User- generated Text, pages 94–106, 2017

    2017

  37. [45]

    W.-w. Yim, Y . Fu, A. Ben Abacha, N. Snider, T. Lin, and M. Yetisgen. Aci-bench: a novel ambient clinical intelligence dataset for benchmark- ing automatic visit note generation. Scientific data, 10(1):586, 2023

    2023

  38. [46]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi. HellaSwag: Can a machine really finish your sentence? In Proc. of the 57th ACL , pages 4791–4800, 2019

    2019

  39. [47]

    Y . Zhao, L. Yan, W. Sun, G. Xing, S. Wang, C. Meng, Z. Cheng, Z. Ren, and D. Yin. Improving the Robustness of Large Language Models via Consistency Alignment. In LREC-COLING, pages 8931–8941, 2024

    2024

  40. [48]

    Zheng et al

    L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in NeurIPS, 36:46595–46623, 2023

    2023

  41. [49]

    Zhong et al

    M. Zhong et al. QMSum: A new benchmark for query-based multi- domain meeting summarization. arXiv:2104.05938, 2021

    2021 arXiv

  42. [50]

    L. Zhu, X. Wang, and X. Wang. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv:2310.17631, 2023

    2023 arXiv

  43. [51]

    T. Y . Zhuo et al. Bigcodebench: Benchmarking code generation with di- verse function calls and complex instructions.arXiv:2406.15877, 2024

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.