Pith. sign in

REVIEW 3 major objections 6 minor 88 references

The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark score is computed from one phrasing of each problem, but rephrasing a problem while keeping its answer fixed routinely flips an LLM's correctness in both directions, so the reported number is just one point inside a wide range.

desk verdict Useful, well-built audit method; the 'stronger models are more wording-dependent' claim is confounded by rate definitions and needs conditional-rate analysis before it can be believed. read the letter →

arxiv 2608.11694 v1 pith:AOKQ6YYV submitted 2026-08-12 cs.CL cs.AI

Shailja Thakur , Sungeun An , Chad DeLuca , Hima Patel This is my paper · ORCID
classification cs.CLcs.AI
keywords benchmarkdriftwordingsensitivitymeaning-preservingrephrasingLLMevaluationpositiveandnegativepromptrobustnessBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a benchmark score, computed from one phrasing of each problem, is not a stable estimate of what an LLM can do. Meaning-preserving rephrasing routinely flips correctness in both directions: some wrong answers become right and some right answers become wrong, across eight models and three benchmarks. Averaged over the 24 model–benchmark pairs, worst-case accuracy (correct under every phrasing) and best-case accuracy (correct under at least one) are separated by 74.7 percentage points. The paper further argues that phrasing sensitivity does not fade as models get stronger; instead it changes sign, so strong models lose far more correct answers to rephrasing than they gain, while weak models gain more than they lose. If this is right, leaderboard scores can rank wording luck as much as capability.

What carries the argument

The central object is BenchDrift, a four-role pipeline: a generator proposes meaning-preserving rephrasings of each problem, a validator keeps only those that still yield the original ground-truth answer, the target model answers the original and all surviving variations, and a judge scores the answers. The argument is carried by the drift identities $Best = Rep. + Pos.$ and $Worst = Rep. - Neg.$, where $Pos.$ is the fraction of problems wrong on the original but right on some rephrasing and $Neg.$ is the fraction right on the original but wrong on some rephrasing. A shared denominator makes the two directions comparable, and every flip is tied to one of four axes — linguistic, referential, pragmatic, and structural — and to a specific transformation within that axis. These identities turn the reported score into one point in an exactly computable interval.

What would settle it

Take a random sample of the variations the validator accepted, ask independent human annotators whether each one still has the same ground-truth answer, re-measure drift using only human-approved variations, and compare. If the 74.7-point average range collapses or the positive–negative asymmetry reverses, the original numbers were set by the validator's blind spot rather than by the models' wording sensitivity.

Watch

Extended reading notes

Core claim

The paper establishes that correctness under meaning-preserving rephrasing behaves like a two-way drift, with every flip attributed to a specific transformation. For a fixed problem and answer, a model's accuracy can be as low as the share of problems it gets right under every tested rephrasing and as high as the share it gets right under at least one, and the reported single-phrasing score sits inside that range. Across GSM8K, MMLU, and MATH-Hard, this range averages 74.7 percentage points. The direction of the drift tracks baseline accuracy almost linearly (Pearson $r = 0.98$, slope 1.08): above a 60% baseline, negative drift exceeds positive drift in all twelve model–benchmark pairs, so the strongest models are precisely the ones whose scores depend most on the wording they were given. Models also agree on which rephrasings cost the most correct answers (mean pairwise Spearman $\rho = 0.77$), and rephrasing breaks a substantial share of answers the model was most confident about, whether the rephrasing makes the problem shorter or longer.

Load-bearing premise

The load-bearing premise is that the automated validator and judge reliably identify which reworded problems truly keep the original answer and which answers are correct; the paper reports no human validation and cannot rule out a shared blind spot, so any drift measured could mix wording effects with actual answer changes.

Editorial extensions

If this is right

  • A single benchmark score should be read as one point inside a range; across the 24 model–benchmark pairs, the worst-to-best gap averages 74.7 percentage points.
  • For models above a roughly 60% baseline, rephrasing costs more correct answers than it recovers, so the reported score tends to overstate how much of a model's success is robust to wording.
  • Because models largely agree on which rephrasings break the most answers, a wording transformation that hurts one model is likely to hurt others; fragility is a property of the rephrasing, not just of the model.
  • Model confidence cannot filter fragile answers: even among the answers the model was most confident about, almost one in five is lost to a meaning-preserving rephrasing.
  • An audit using only five rephrased variations per problem recovers about three-quarters of the drifting problems, so measuring drift does not require a full-scale generation campaign.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if fragility belongs to the rephrasing, then benchmark items could be ranked by how many models they break, and weighting items by this fragility would produce a more stable evaluation score; the paper does not construct such an item-level ranking.
  • One testable extension the paper leaves implicit is whether drift on one benchmark predicts drift on another for the same model; the near-linear asymmetry result suggests it might, but no cross-benchmark correlation is reported.
  • Because the authors find rephrasing breaks confident answers, a selective-prediction system that only answers when the model shows high confidence would still lose accuracy under rephrasing; this follows from their data but is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces BenchDrift, a pipeline that generates variations of benchmark problems along four axes (linguistic, referential, pragmatic, structural), validates with an LLM that each variation preserves the ground-truth answer, and measures how often correctness flips relative to the original phrasing. Over eight models and three benchmarks (GSM8K, MMLU, MATH-Hard), the authors report an average best-to-worst accuracy span of 74.7 percentage points, a two-way drift in which weak models gain more than they lose and strong models lose far more than they gain, and transformation-level fragility rankings that are consistent across models. They also report reliability checks that swap generator/validator/judge models, a subsampling analysis of the number of variations needed, and a comparison with prompt optimizers. The paper's central claims are that benchmark scores are not stable estimates of capability and that the best models are the most wording-dependent.

Significance. If the findings hold, the paper provides a valuable methodology and measurement: it turns a single-point benchmark score into an interval, attributes flips to concrete transformations, and shows cross-model agreement about which rephrasings are dangerous. The open release of code, data, and the variation taxonomy is a strength, as are the bootstrap CIs, the role-swap reliability checks, and the permutation-based axis test. However, the two headline interpretive claims—the sign change with model strength and the assertion that the best models depend most on wording—rest on unconditional drift rates whose denominators impose algebraic caps, and the answer-preservation guarantee relies on LLM validator/judge instruments with no human validation. The paper itself flags the latter in Appendix A. These issues are addressable with additional analyses on the released data, but they currently leave the strongest conclusions underdetermined.

major comments (3)
  1. [Section 2.2 and Section 4.1] The claim that phrasing sensitivity changes sign as models improve (and the derived statement that the best models are the most wording-dependent) is based on comparing Pos and Neg as unconditional fractions of the problem count N. By definition, positive drift can only occur on problems the model answered incorrectly on the original phrasing, and negative drift only on problems it answered correctly, so Pos ≤ 1−Rep and Neg ≤ Rep. Under a baseline-only null model with constant recovery rate r_p for original failures and constant fragility rate r_n for original successes, Neg − Pos = Rep(r_n + r_p) − r_p, which is linear in Rep with slope r_n + r_p. The reported near-linear relation (Pearson r = 0.98, slope 1.08) is exactly the curve such a null produces, so it does not by itself establish a model-dependent change in sensitivity. The manuscript never reports the conditional rates—positive drift among original failures, Pos/(1−Rep), and negative drift among original successes, Neg/Rep—nor fits a constant-rate null model. Recomputing from Table 2 shows the direction can reverse under conditioning: GPT-OSS-20B on GSM8K has a conditional recovery rate of 80.8% (4.2/5.2) versus a conditional fragility rate of 59.9% (56.8/94.8). The authors should report the conditional rates for all 24 pairs, fit the constant-rate null, and re-evaluate the sign-change and 'most wording-dependent' conclusions on that basis.
  2. [Appendix A and Section 2.5] The validator and the judge are LLMs, and the paper states that no human validation was run. The authors correctly note that a judge that is directionally more permissive (e.g., more willing to accept borderline answers on variations than on the original) could inflate or deflate the asymmetry, which is a central finding. The role-swap checks in Section 4.2 demonstrate agreement among LLM instruments, not agreement with human judgment, so a shared blind spot remains possible. Because every drift measurement and the 74.7-point span assume that each surviving variation truly preserves the answer, an unvalidated validator confounds wording effects with actual answer changes. The manuscript should include a human-annotation study on a stratified sample of variations (across axes and benchmarks) reporting validator precision and judge directional bias, and should show that the headline drift rates and the asymmetry are robust to a conservative re-analysis restricted to human-confirmed answer-preserving variations.
  3. [Section 4.2] The statement that role swaps 'shift average negative drift by at most 11.9 percentage points and average positive drift by at most 2.9, both within the per-cell confidence intervals of Table 2' appears to be incorrect. The per-cell (or per-configuration) confidence intervals in Table 3 are around ±3.7–4.3 percentage points, while the main configuration reports negative drift of 45.9±4.3 and the fully swapped configuration reports 34.0±3.8; the difference of 11.9 points is outside the reported interval. Since this sentence is the basis for the claim that the measurement is stable across pipeline roles, the authors should correct the CI comparison (e.g., report overlap tests or the standard error of the difference).
minor comments (6)
  1. [Section 2.2] The notation 'Rep.' is used in the identities Best = Rep. + Pos. and Worst = Rep. − Neg. before it is defined; define it at first use (e.g., when drift is introduced).
  2. [Table 3] Confidence intervals are only shown for the main configuration; report CIs for each role-swap row so the reader can assess the overlap claims directly.
  3. [Figure 3] The y-axes are truncated (starting at 8% and 5%) without break symbols; this can exaggerate relative differences—add an axis break or start at 0 while keeping the printed values.
  4. [Section 4.8] The comparison with DSPy and GEPA uses only 15 GSM8K problems and a single target model; the text should clearly state that this is an illustrative case study rather than a general efficiency comparison.
  5. [Section 4.3] The permutation test reports p<0.001 but does not state the test statistic; specify the quantity being permuted (e.g., spread of axis-level drift rates) and the test direction.
  6. [Abstract and Section 4.1] The statement that 'models more often lose correct answers to rephrasing than they gain new ones' is an aggregate over 24 model–benchmark pairs; make explicit that this is an average and that the direction reverses for weaker models below the 60% baseline.

Circularity Check

1 steps flagged · score 6.0 of 10

The sign-change asymmetry claim is an artifact of unconditional drift denominators; conditional recovery/fragility rates are never reported.

  1. self definitional [Section 2.2 (Drift definitions) and Section 4.1 ('The Range Behind a Score')]
    "Pos.= 1/N |{q:C(q)=0,∃q′,C(q′)=1}| ... Neg.= 1/N |{q:C(q)=1,∃q′,C(q′)=0}| ... The direction of the shift depends on how strong the model already is. The asymmetry between the two directions — negative minus positive drift — tracks baseline accuracy almost linearly across our 24 pairs (Pearson r=0.98, slope 1.08)."

    By the definitions, Pos can count only problems the model got wrong on the original phrasing, and Neg only problems it got right, so 0 ≤ Pos ≤ 1−Rep and 0 ≤ Neg ≤ Rep. Writing p = Pos/(1−Rep) and n = Neg/Rep gives the identity Neg−Pos = Rep(n+p) − p. The claimed near-linear dependence of Neg−Pos on Rep, the sign change around a ~60% baseline, and the conclusion that the best models are the most wording-dependent are therefore the shape forced by using the same unconditional denominator N: whenever the conditional recovery and fragility rates are roughly stable, the asymmetry is linear in Rep by construction.

full rationale

The paper's core measurements — the 74.7pp average best-to-worst range, the transformation-level rankings, and the length/confidence checks — are direct empirical measurements on external benchmarks (GSM8K, MMLU, MATH-Hard) and are not circular. There are no load-bearing self-citations or imported uniqueness theorems. However, the headline result that phrasing sensitivity changes sign with model strength and that the best models are the most wording-dependent is built on Pos and Neg defined as unconditional fractions of the same N. Because Pos can only arise from baseline failures and Neg only from baseline successes, the feasible ranges of both rates are tied linearly to Rep, and the observed r=0.98 / slope 1.08 relation is exactly the signature of that definitional structure under roughly constant conditional rates. Appendix A's admission that no human validation was run, and that a directionally permissive judge could inflate or deflate positive versus negative drift asymmetrically, further weakens the asymmetry claim, though that is a validity limitation rather than a circular step. On balance, the magnitude-of-drift findings have independent content, but the sign-change / 'best models are most wording-dependent' conclusion partially reduces to the construction of the drift denominators, so a partial circularity score of 6 is appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method is a measurement framework, not a fitted model. The main hand-chosen parameters are the variation budget proportions and confidence bin thresholds. The load-bearing domain assumptions are that the LLM validator and judge are reliable enough to guarantee answer preservation and scoring; the paper explicitly flags the absence of human validation.

free parameters (3)
  • Axis variation budget proportions = 1.0, 0.75, 0.55, 0.40
    Chosen by hand in Section 2.4 to allocate a problem's variation budget across the four axes ranked by property match. Unequal allocation means axis-level drift rates are not directly comparable; the authors acknowledge this and rely on transformation-level results.
  • Confidence bin thresholds = 0.85 and 0.95
    Chosen by hand in Section 4.6 for Figure 5 to bin mean generated-token probabilities into low, mid, and high confidence. The high-confidence negative drift rate of 18.5% depends on these cutoffs.
  • Length-change bins for Figure 4 = not exactly specified
    Bins such as 'shorter', 'preserved', 'longer', and 'more than double' are not given precise numerical definitions, which affects the claim that negative drift rises in both directions.
assumptions (4)
  • domain assumption The LLM validator's acceptance of a variation implies the ground-truth answer is unchanged.
    Section 2.5 uses an LLM validator to discard candidates that change the answer. If the validator has a shared blind spot, drift rates mix wording sensitivity with actual answer changes. Appendix A states no human validation was run.
  • domain assumption The LLM judge's scoring is correct and has no directional bias between original and rephrased problems.
    Section 3 uses Llama-3.3-70B-Instruct as judge. Appendix A notes a directional permissiveness bias could inflate or deflate the positive versus negative drift asymmetry, a central finding.
  • domain assumption The four-axis taxonomy covers meaning-preserving rephrasing; transformations that pass validation are treated as equivalent questions.
    The paper assumes that validated variations keep the answer fixed by construction. This is the core measurement assumption, and it is only checked with other LLMs, not with humans.
  • standard math Best equals Rep plus Pos, Worst equals Rep minus Neg, and the four-case partition covers all problems exactly.
    These identities follow directly from the definitions in Section 2.2 and are used to compute best and worst case accuracy. They are standard and not in dispute.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance." pith.science (2026). https://pith.science/paper/AOKQ6YYV

@misc{pith2026260811694,
  author       = {Pith},
  title        = {Pith review of: The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AOKQ6YYV}},
  note         = {Machine review of arXiv:2608.11694}
}
read the original abstract

A benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it does not. We show that rephrasing a problem while keeping its meaning and answer fixed routinely flips a model's answer in both directions, so some failures become successes and some successes become failures. We call this drift. BenchDrift generates meaning-preserving variations of benchmark problems along four axes, namely linguistic, referential, pragmatic, and structural, and measures how often, and why, correctness flips under each. Across eight models and three benchmarks (GSM8K, MMLU, MATH-Hard), we observe that drift is large in both directions. Two findings stand out. First, phrasing sensitivity does not fade as models get better. Instead, it changes sign. Weak models gain more from rephrasing than they lose, while strong models lose far more than they gain. We find that the best models on a benchmark are therefore the ones whose scores depend most on the wording they happened to be given. Second, the models largely agree on which rephrasings cost the most correct answers even though they differ in how much they drift, so fragility belongs to the rephrasing and not to the model. Furthermore, rephrasing breaks answers a model was confident about, whether the problem is made shorter or longer. Code and Data: https://github.com/IBM/BenchDrift/tree/demo-ui

Figures

Figures reproduced from arXiv: 2608.11694 by the authors.

Figure 1
Figure 1. A GSM8K problem reworded along four axes, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. BenchDrift takes a benchmark problem, generates meaning-preserving variations along four axes, validates [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. All 32 transformations, pooled over 8 models and 3 benchmarks; colour gives the axis. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Drift by how much the rephrasing changed the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Negative drift by baseline confidence, pooled [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 33 canonical work pages

  1. [1]

    American Psychologist , volume=

    Transfer Learning in Cognitive Science: From Laboratory to Real-World Settings , author=. American Psychologist , volume=. 2002 , publisher=

    2002

  2. [2]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

    Beyond Accuracy: Behavioral Testing of NLP Models with CheckList , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

  3. [4]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages=

    Universal Adversarial Triggers for Attacking and Analyzing NLP , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages=

    2019

  4. [5]

    arXiv preprint arXiv:2108.06314 , year=

    TimeQA: A Benchmark for Temporal Reasoning in Question Answering , author=. arXiv preprint arXiv:2108.06314 , year=

    arXiv

  5. [6]

    arXiv preprint arXiv:2211.09110 , year=

    Holistic Evaluation of Language Models , author=. arXiv preprint arXiv:2211.09110 , year=

    arXiv

  6. [7]

    Psychometrika , volume=

    Cognitive Diagnosis for Adaptive Testing , author=. Psychometrika , volume=. 1983 , publisher=

    1983

  7. [8]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    Attention is not Explanation , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    2019

  8. [9]

    IEEE Access , volume=

    A Systematic Assessment of Different Word-Embedding Models for Better Feature Representation and Better Text Classification , author=. IEEE Access , volume=. 2019 , publisher=

    2019

Show all 88 references
  1. [10]

    arXiv preprint arXiv:2012.06705 , year=

    The Brittleness of AI: Three Sources and a Solution , author=. arXiv preprint arXiv:2012.06705 , year=

    2012 arXiv

  2. [11]

    ACM Transactions on Intelligent Systems and Technology , year=

    A Survey on Evaluation of Large Language Models , author=. ACM Transactions on Intelligent Systems and Technology , year=

  3. [12]

    Proceedings of ACL , pages=

    Beyond Accuracy: Behavioral Testing of NLP Models with CheckList , author=. Proceedings of ACL , pages=

  4. [13]

    arXiv preprint arXiv:2211.09527 , year=

    Ignore Previous Prompt: Attack Techniques For Language Models , author=. arXiv preprint arXiv:2211.09527 , year=

    arXiv

  5. [14]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and Transferable Adversarial Attacks on Aligned Language Models , author=. arXiv preprint arXiv:2307.15043 , year=

    arXiv

  6. [17]

    arXiv preprint arXiv:2305.14739 , year=

    Trusting Your Evidence: Hallucinate Less with Context-aware Decoding , author=. arXiv preprint arXiv:2305.14739 , year=

    arXiv

  7. [20]

    Distill , year=

    Zoom In: An Introduction to Circuits , author=. Distill , year=

  8. [21]

    ACM Computing Surveys , volume=

    Explainability for Large Language Models: A Survey , author=. ACM Computing Surveys , volume=

  9. [22]

    arXiv preprint arXiv:2401.xxxxx , year=

    Explainability in Large Language Models , author=. arXiv preprint arXiv:2401.xxxxx , year=

  10. [24]

    arXiv preprint arXiv:2402.16906 , year=

    LDB: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step , author=. arXiv preprint arXiv:2402.16906 , year=

    arXiv

  11. [25]

    arXiv preprint arXiv:2312.xxxxx , year=

    AutoSD: Automatic and Scalable Debugging for Deep Learning with Large Language Models , author=. arXiv preprint arXiv:2312.xxxxx , year=

  12. [26]

    arXiv preprint arXiv:2304.05128 , year=

    Teaching Large Language Models to Self-Debug , author=. arXiv preprint arXiv:2304.05128 , year=

    arXiv

  13. [30]

    arXiv preprint arXiv:2404.18796 , year=

    Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models , author=. arXiv preprint arXiv:2404.18796 , year=

    arXiv

  14. [31]

    Proceedings of ICLR , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of ICLR , year=

  15. [33]

    Advances in Neural Information Processing Systems , pages=

    SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems , author=. Advances in Neural Information Processing Systems , pages=

  16. [34]

    arXiv preprint arXiv:2205.10625 , year=

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models , author=. arXiv preprint arXiv:2205.10625 , year=

    arXiv

  17. [36]

    arXiv preprint arXiv:2210.03350 , year=

    Measuring and Narrowing the Compositionality Gap in Language Models , author=. arXiv preprint arXiv:2210.03350 , year=

    arXiv

  18. [37]

    Proceedings of ICLR , year=

    Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval , author=. Proceedings of ICLR , year=

  19. [38]

    Proceedings of ICLR , year=

    ReQA: An Evaluation for End-to-End Answer Retrieval Models , author=. Proceedings of ICLR , year=

  20. [39]

    arXiv preprint arXiv:2211.01910 , year=

    Large Language Models Are Human-Level Prompt Engineers , author=. arXiv preprint arXiv:2211.01910 , year=

    arXiv

  21. [41]

    arXiv preprint arXiv:2305.14283 , year=

    Query Rewriting for Retrieval-Augmented Large Language Models , author=. arXiv preprint arXiv:2305.14283 , year=

    arXiv

  22. [42]

    Proceedings of NAACL-HLT , pages=

    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge , author=. Proceedings of NAACL-HLT , pages=

  23. [43]

    arXiv preprint arXiv:2412.xxxxx , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.xxxxx , year=

  24. [45]

    arXiv preprint arXiv:2412.xxxxx , year=

    Phi-4 Technical Report , author=. arXiv preprint arXiv:2412.xxxxx , year=

  25. [46]

    arXiv preprint arXiv:2412.xxxxx , year=

    DeepSeek-V3 Technical Report , author=. arXiv preprint arXiv:2412.xxxxx , year=

  26. [47]

    arXiv preprint arXiv:2405.xxxxx , year=

    Granite 3.0 Language Models , author=. arXiv preprint arXiv:2405.xxxxx , year=

  27. [48]

    2024 , note=

    GPT-OSS-20B: Open Source GPT-scale Model , author=. 2024 , note=

    2024

  28. [49]

    2024 , note=

    Test-of-Time Arithmetic Reasoning Dataset , author=. 2024 , note=

    2024

  29. [51]

    Perkins, D. N. and Salomon, G. , title =. International Encyclopedia of Education , edition =. 1992 , pages =

    1992

  30. [52]

    Detterman, D. K. , title =. Transfer on Trial: Intelligence, Cognition, and Instruction , editor =. 1993 , pages =

    1993

  31. [53]

    Barnett, S. M. and Ceci, S. J. , title =. Psychological Bulletin , year =

  32. [54]

    Gick, M. L. and Holyoak, K. J. , title =. Cognitive Psychology , year =

  33. [55]

    Singley, M. K. and Anderson, J. R. , title =. 1989 , publisher =

    1989

  34. [56]

    , title =

    Lobato, J. , title =. The Journal of the Learning Sciences , year =

  35. [57]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  36. [58]

    How People Learn: Brain, Mind, Experience, and School , year =

  37. [59]

    2025 , eprint=

    gpt-oss-20b Model Card , author=. 2025 , eprint=

    2025

  38. [60]

    2024 , eprint=

    Phi-4 Technical Report , author=. 2024 , eprint=

    2024

  39. [61]

    2024 , eprint=

    Granite Code Models: A Family of Open Foundation Models for Code Intelligence , author=. 2024 , eprint=

    2024

  40. [62]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  41. [63]

    2024 , eprint=

    Qwen2.5 Technical Report , author=. 2024 , eprint=

    2024

  42. [64]

    2024 , institution=

    Granite 3.0 Language Models , author=. 2024 , institution=

    2024

  43. [65]

    2024 , url=

    Large Enough , author=. 2024 , url=

    2024

  44. [66]

    2024 , eprint=

    Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning , author=. 2024 , eprint=

    2024

  45. [67]

    2024 , eprint =

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting , author =. 2024 , eprint =

    2024

  46. [69]

    2021 , eprint =

    Robustness Gym: Unifying the NLP Evaluation Landscape , author =. 2021 , eprint =

    2021

  47. [71]

    2023 , eprint =

    Self-Refine: Iterative Refinement with Self-Feedback , author =. 2023 , eprint =

    2023

  48. [73]

    Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges

    Ma, Bolei and Li, Yuting and Zhou, Wei and Gong, Ziwei and Liu, Yang Janet and Jasinskaja, Katja and Friedrich, Annemarie and Hirschberg, Julia and Kreuter, Frauke and Plank, Barbara. Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportuniti...

    2025 doi

  49. [74]

    Ant \`o nia and Salam \'o , Maria

    Kovatchev, Venelin and Mart \'i , M. Ant \`o nia and Salam \'o , Maria. ETPC - A Paraphrase Identification Corpus Annotated with Extended Paraphrase Typology and Negation. Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018). 2018

    2018

  50. [77]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics: Industry Track , year=

    Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics: Industry Track , year=

  51. [78]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Ol...

    2024 arXiv

  52. [79]

    Sandhini Agarwal, Lama Ahmad, and others. 2025. https://arxiv.org/abs/2508.10925 gpt-oss-20b model card . Preprint, arXiv:2508.10925

    2025 arXiv

  53. [80]

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2025. Gepa: Ref...

    2025 arXiv

  54. [81]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  55. [82]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. https://doi.org/10.1162/tacl_a_00410 Measuring and improving consistency in pretrained language models . Transactions of the Association for Computati...

    2021 doi

  56. [83]

    Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher R \'e . 2021. https://arxiv.org/abs/2101.04840 Robustness gym: Unifying the nlp evaluation landscape . Preprint, arXiv:2101.04840

    2021 arXiv

  57. [84]

    IBM Granite Team. 2024. https://github.com/ibm-granite/granite-3.0-language-models/blob/main/paper.pdf Granite 3.0 language models . Technical report, IBM

    2024

  58. [85]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and others. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  59. [86]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of ICLR

    2021

  60. [87]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and others. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  61. [88]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023 b . Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561

    2023 arXiv

  62. [89]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-...

    2023 arXiv

  63. [90]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406

    2022 arXiv

  64. [91]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  65. [92]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://arxiv.o...

    2023 arXiv

  66. [93]

    Mayank Mishra, Matt Stallone, and others. 2024. https://arxiv.org/abs/2405.04324 Granite code models: A family of open foundation models for code intelligence . Preprint, arXiv:2405.04324

    2024 arXiv

  67. [94]

    Mistral AI . 2024. https://mistral.ai/news/mistral-large-2407/ Large enough . Mistral Large 2

    2024

  68. [95]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. arXiv preprint arXiv:2305.03495

    2023 arXiv

  69. [96]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020 doi

  70. [97]

    Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, and Daniel Dahlmeier. 2025. Optimization before evaluation: Evaluation with unoptimised prompts can be misleading. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics: Industry Track

    2025

  71. [98]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://arxiv.org/abs/2310.11324 Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . Preprint, arXiv:2310.11324

    2024 arXiv

  72. [99]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366

    2023 arXiv

  73. [100]

    Qwen Team. 2024. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2024 arXiv

  74. [101]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, and Maosong Sun. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621

    2024 arXiv

  75. [102]

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. https://arxiv.org/abs/1908.07125 Universal adversarial triggers for attacking and analyzing NLP . Preprint, arXiv:1908.07125

    2019 arXiv

  76. [103]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266--3280

    2019

  77. [104]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  78. [105]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903

    2022 arXiv

  79. [106]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and others. 2025. https://arxiv.org/abs/2505.09388 Qwen3...

    2025 arXiv

  80. [107]

    Xing, and others

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, and others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.