Pith. sign in

REVIEW 3 major objections 8 minor 13 cited by

Do Large Language Model Benchmarks Test Reliability?

T0 review · 3 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current LLM benchmarks are not equipped to test reliability: even after all label errors are cleaned away, frontier models still fail elementary math and logic questions, so benchmark saturation does not mean dependable performance.

desk verdict Useful reliability benchmark suite undercut by an over-strong 'every failure is genuine' claim; the two failure patterns are the strongest contribution. read the letter →

arxiv 2502.03461 v1 pith:RNLXS2AB submitted 2025-02-05 cs.LG cs.CL

Joshua Vendrow , Edward Vendrow , Sara Beery , Aleksander Madry This is my paper · ORCID
classification cs.LGcs.CL
keywords LLMreliabilityplatinumbenchmarksbenchmarklabelnoisefrontierlanguagemodelsfailurepatternssaturationevaluationmethodology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard LLM benchmarks measure peak capability, not reliability: a high score on a 'saturated' benchmark does not mean the model can be trusted on easy tasks. It shows that many apparent model failures on fifteen popular benchmarks are actually due to label errors or ambiguous questions, and that removing those errors reveals a persistent set of genuine failures. The authors propose platinum benchmarks, datasets curated so that every remaining wrong answer is a real model mistake, and use them to locate each model's reliability frontier. The key result is that even frontier models still fail elementary math word problems, basic coreference, and simple logic after cleaning, so the reliability gap is real and largely hidden by noisy benchmarks.

What carries the argument

The platinum benchmark is the mechanism: a dataset is revised by querying several frontier LLMs, manually inspecting every example on which at least one model errs, fixing mislabeled solutions, and removing questions that are logically contradictory, ambiguous, or ill-posed. This makes 100% accuracy an attainable target, so any model error on the cleaned set is a genuine failure rather than an artifact of the benchmark. The same curated set also lets the authors discover failure patterns, which they then confirm by procedurally generating additional examples in the same format.

What would settle it

Independently re-annotate a random sample of the questions that survived cleaning because all models agreed with the stated answer; if a material fraction of those are judged ambiguous or mislabeled by human experts, then some reported model failures on the platinum benchmarks would not be genuine, undercutting the claim that the reliability gap is that large.

Watch

Extended reading notes

Core claim

The central discovery is that benchmark saturation is not evidence of reliability. Across fifteen widely used benchmarks spanning math, logic, table understanding, reading comprehension, commonsense reasoning, and visual question answering, the authors find pervasive label noise: for example, 5% of the GSM8K questions they sample are mislabeled or poorly written, and for most benchmarks more than half of apparent model errors vanish once the benchmark is cleaned. On the resulting platinum benchmarks, however, nearly every model still fails on nearly every task category, including grade-school arithmetic and simple pronoun reference. The failures follow identifiable patterns, such as answering 'first' when asked which event happened 'second' and incorrectly rounding up a whole-number quotient when the answer has few divisors.

Load-bearing premise

The cleaning filter assumes that any question on which no tested model errs is correctly labeled and unambiguous, so a bad question that all models happen to answer the same way can remain in the benchmark and be counted as a genuine model failure later.

Editorial extensions

If this is right

  • Benchmark saturation should no longer be read as evidence of reliability; a model at 95% on GSM8K can still be unreliable on elementary math.
  • Benchmark builders should report label-error rates separately from model-error rates, because on most benchmarks in this study the majority of apparent errors are attributable to the benchmark itself.
  • Reliability is not a single ranking: different models lead on different task families, so deployment choices should be matched to the specific capability required.
  • Systematic failure patterns such as the first-event bias and rounding-up-primes behavior can be elicited with procedurally generated examples, making targeted reliability testing feasible.
  • The platinum benchmark concept gives reliability a concrete operational meaning: pass means 100% accuracy on a cleaned set, similar to counting 'nines' of uptime for deployed software.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension of the paper's method would be to run the same LLM-agreement screening on fresh benchmark releases before they become saturated, catching label errors early.
  • The cleaning filter is vulnerable to systematically shared model blind spots; if all evaluated models misparse a particular question template, the template survives as 'platinum' and later errors on it would be misattributed.
  • The failure-pattern results suggest that reliability may be prompt- and format-dependent, and that a model's reliability frontier might shift when question wording is varied, a possibility the paper leaves open.
  • One could stress-test the reliability gap by building adversarial-but-natural questions tailored to each discovered failure pattern, since the paper shows these patterns are reproducible synthetically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper investigates whether existing LLM benchmarks can meaningfully measure reliability. The authors argue that label noise and ambiguity in 'saturated' benchmarks hide genuine model failures, and they introduce 'platinum benchmarks' obtained by manually cleaning subsets of fifteen popular benchmarks. They evaluate more than twenty models on the cleaned subsets and report that even frontier models still make errors on simple tasks such as elementary math word problems. They also identify two previously undocumented failure patterns—'first event bias' and 'rounding up primes'—and verify them with procedurally generated examples.

Significance. If the conclusions hold, the paper makes a valuable contribution by challenging the common practice of retiring benchmarks once models reach high accuracy, and by proposing a concrete methodology for constructing reliability-oriented benchmarks. The release of code, the transparent discussion of limitations, the comparison with MMLU-Redux annotations, and the synthetic follow-up experiments for the two failure patterns are all strengths that support the paper's empirical core. However, the quantitative reliability claims depend critically on the completeness and accuracy of the manual cleaning process, and the paper contains at least one concrete instance where that process appears to have failed. This issue directly affects the headline finding that frontier LLMs still fail simple tasks after cleaning.

major comments (3)
  1. [Section 1.1 / Table 3] The retained GSM8K example 'A company pays each of its employees $600 in a month... what's her annual salary after three more years of service?' is listed as a model failure, with the benchmark answer '23'. This answer is not derivable from the stated terms: an initial monthly salary of $600 implies an annual salary of $7,200, and any reasonable 10%-per-year increase would produce an answer in the thousands. The listed model outputs (e.g., 10,452, 10,080, 7,920) are attempts at a real computation, while '23' appears to be a label error. Because this example is counted in Table 3 as a genuine model failure, the claim in Section 3.2 that 'every model failure we report is genuine' is contradicted by the paper's own appendix. The authors should either correct this label, remove the example, or justify why '23' is correct, and they should re-audit the other retained failure examples for similar missed label errors.
  2. [Section 2.2 / Section 5.1] The introduction states that 'for most tasks no state-of-the-art model that we evaluate is able to pass (i.e., achieve 100% accuracy).' This claim appears inconsistent with Table 3: for example, the o1-2024-12-17 (high) row shows 0 errors on 10 of the 15 platinum benchmarks, meaning that model achieves 100% on those tasks. Even if one interprets 'state-of-the-art' as a specific subset of models, the statement as written overstates the findings. The authors should either quantify the number of benchmarks for which no evaluated model achieves 100% and show that it is indeed 'most,' or revise the claim to match the data (e.g., 'almost every model makes errors on many benchmarks').
  3. [Appendix D / Section 2.2] The cleaning protocol in Section 2.2 inspects only examples on which at least one LLM errs, with manual review by the authors. The acknowledged limitation 3 (Section 5.1) covers the case where all models agree on a wrong answer, which cannot inflate current-model failure counts. However, the Appendix E case shows that manual inspection itself can be fallible even for flagged examples. Because no inter-annotator agreement or second-pass verification is reported, the paper's guarantee that 'every model failure we report is genuine' is not established. The authors should provide a more rigorous annotation protocol, or at minimum correct the concrete counterexample and soften the claim accordingly.
minor comments (8)
  1. [Section 1.1] The reference for GSM8K is given as [DGS+18], but the correct citation is [CKB+21] (used elsewhere in the paper). [DGS+18] is a different paper ('Training verified learners with learned verifiers'). Please fix the citation.
  2. [Section 2.1, Table 2] The sentence 'follow a consistent pattlife cycledevelopment, progress, and eventual retirement thai Specifically' appears garbled. The intended text is likely 'follow a consistent pattern: development, progress, and eventual retirement.' Please correct this typo.
  3. [Appendix C, Figure C.1] The caption notes that the number of mislabeled examples is missing from VQA v2.0, but the table entry for VQA v2.0 is '—'. Since the table already has a footnote, consider adding the same note directly in the caption or keeping the '—' but explaining it in the footnote for clarity.
  4. [Appendix E] In the first event bias figure, the x-axis labels show accuracies with a percent sign, but the caption reads 'Accuracy (%).' The values such as '6.0% 8.0%' are clear, but consider adding a note that the red dashed line at 50% represents random guessing, as this is mentioned in the text but not in the figure.
  5. [Section 3.3, Appendix C.1] The example 'John couldn't see the stage with Billy in front of him because he is so short' is listed with the correct answer as 'John.' This is a subtle Winograd-style sentence, but the authors might consider adding a brief explanation of the intended reading to help readers understand why the model answers are wrong.
  6. [Section 3.3, Appendix C.2] The synthetic verification of the first event bias uses GPT-4o to generate contexts, which is a reasonable choice, but the paper does not report whether the generated contexts were manually checked for chronological clarity. Adding a sentence about quality control of the synthetic examples would strengthen the claim.
  7. [Section 5.1] For the rounding-up-primes analysis, the paper filters out failures caused by incorrect division to isolate the rounding error. This is sensible, but the filtering criterion is not described in detail. Please clarify how the authors distinguished a division error from an intentional rounding error.
  8. [Section 2.2] The limitation about sample size (Limitation 2) is important, but it is only stated qualitatively. Given that some benchmarks have as few as 100 examples, a single error corresponds to 1% accuracy; the paper's color coding distinguishes 0% from ≤2%, which may not be statistically meaningful. Consider adding a brief discussion of confidence intervals or error bars for the reported error counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cleaned-benchmark reliability finding rests on manual annotation and synthetic verification, not on a fitted prediction or self-citation chain.

full rationale

The paper's central empirical claim—frontier LLMs still fail on simple tasks even after cleaning—is supported by a self-contained pipeline: examples are flagged by LLM disagreement, then adjudicated by manual inspection, and retained errors are exactly those the authors judged genuine after inspection. The two failure patterns are validated on procedurally generated new examples rather than on the annotation set, so they are not equivalent to their discovery inputs. The Table 4 attribution of original-benchmark errors to label noise is a descriptive comparison of error counts before and after removing or relabeling flagged examples; it is a measurement consequence of the cleaning protocol, not an independent prediction, and it is not statistically forced because the decision to remove or relabel rests on manual review. The acknowledged limitation 3 in Section 5.1—that poorly written questions on which all models agree can remain—is a coverage gap that cannot manufacture false failures for the evaluated models and does not reduce the headline derivation to its inputs. Appendix E does raise a correctness concern: the retained GSM8K 'annual salary' example lists a benchmark answer of 23 that is inconsistent with the problem text, suggesting the 'every error is manually verified' claim may be imperfect; this is an evidence-quality issue, not a circularity. Self-citations, such as Tsipras et al. (2020), appear only as background related work and are not load-bearing. No circular step was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on human curation judgments and on the assumption that LLM disagreement is a good detector of bad examples. There are no fitted numerical parameters. The main risk is unexamined examples that are ambiguous but uniformly answered correctly by all annotator models, which would weaken the claim that every remaining error is a genuine model failure.

assumptions (3)
  • domain assumption Frontier LLM disagreement is a reliable filter for surfacing bad benchmark examples.
    Used in Section 2.2; examples are manually inspected only when at least one LLM makes an error. Bad questions answered uniformly by all models are missed.
  • domain assumption The authors' manual review establishes the correct label or badness of every inspected example.
    No second annotator or inter-annotator agreement is reported; the revised labels and the list of valid answers in reading comprehension rest on this judgment.
  • domain assumption A small cleaned subset is representative of the full benchmark category for reliability measurement.
    Several benchmarks are reduced to 100 to 300 examples, so zero-error results have wide sampling uncertainty; acknowledged in Section 5.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Large Language Model Benchmarks Test Reliability?." pith.science (2026). https://pith.science/paper/RNLXS2AB

@misc{pith2026250203461,
  author       = {Pith},
  title        = {Pith review of: Do Large Language Model Benchmarks Test Reliability?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNLXS2AB}},
  note         = {Machine review of arXiv:2502.03461}
}
read the original abstract

When deploying large language models (LLMs), it is important to ensure that these models are not only capable, but also reliable. Many benchmarks have been created to track LLMs' growing capabilities, however there has been no similar focus on measuring their reliability. To understand the potential ramifications of this gap, we investigate how well current benchmarks quantify model reliability. We find that pervasive label errors can compromise these evaluations, obscuring lingering model failures and hiding unreliable behavior. Motivated by this gap in the evaluation of reliability, we then propose the concept of so-called platinum benchmarks, i.e., benchmarks carefully curated to minimize label errors and ambiguity. As a first attempt at constructing such benchmarks, we revise examples from fifteen existing popular benchmarks. We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems. Analyzing these failures further reveals previously unidentified patterns of problems on which frontier models consistently struggle. We provide code at https://github.com/MadryLab/platinum-benchmarks

Figures

Figures reproduced from arXiv: 2502.03461 by the authors.

Figure 1
Figure 1. Examples of errors in current LLM benchmarks. (a) For mislabeled questions, we fix the solutions and include the re-labeled examples in our benchmark. We find three common categories of “bad” questions: (b) there is a logical contradiction in the problem statement, (c) there is ambiguity leading to many plausible solutions, or (d) there is a clear flaw in the construction of the question, such as missing specificati… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fluid Language Model Benchmarking

    cs.CL 2025-09 conditional novelty 8.0 of 10

    Fluid Benchmarking, combining IRT-based ability estimation with Fisher-information-based adaptive item selection, improves LM evaluation across efficiency, validity, variance, and saturation in pretraining settings.

  2. Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility

    cs.CL 2026-05 conditional novelty 7.0 of 10

    SGRE extracts a reasoning skeleton from a teacher trace, coarsens its graph, and verbalizes it densely; the final answer is preserved verbatim, and students distilled on the edited traces show large accuracy drops.

  3. Proof2Hybrid: Automatic Mathematical Benchmark Synthesis for Proof-Centric Problems

    cs.CL 2025-08 conditional novelty 7.0 of 10

    A fully automated pipeline produces proof-centric math benchmarks, demonstrated on algebraic geometry with 456 items, where leading LLMs score near 60 percent.

  4. CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting

    cs.LG 2025-05 accept novelty 7.0 of 10

    Capping achievable accuracy with randomized correct answers turns any model that exceeds the cap into a detectable contamination alarm.

  5. Position: Evaluation of ECG Representations Must Be Fixed

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Current ECG representation benchmarks overstate the benefits of pretraining and produce unstable method rankings; a random encoder with linear probing is competitive on many tasks.

  6. Weight Decay Improves Language Model Plasticity

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Pretrained models trained with larger weight decay fine-tune better on downstream tasks, so the best pretraining checkpoint by loss is not always the best starting point for later training.

  7. Model soups need only one ingredient

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A single checkpoint, edited by splitting each layer's update with SVD and reweighting the high- and low-energy parts, reaches soup-level OOD robustness without multi-model training.

  8. Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

    cs.CL 2025-10 conditional novelty 6.0 of 10

    An IRT-based adaptive testing framework, ATLAS, estimates LLM ability with 30-89 items per benchmark, matching whole-bank ability estimates and re-ranking 23-31% of models relative to accuracy.

  9. From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    KMMLU-Redux and KMMLU-Pro are new Korean benchmark datasets from national technical and professional licensure exams, with LLM evaluations reported against official pass thresholds.

  10. Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A comprehensive reanalysis finds that min-p sampling does not outperform top-p, top-k, or basic sampling once the original data are re-tested and hyperparameter budgets are equalized.

  11. A Sovereign, Open-Source Foundation Model for German and English

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Soofi S 30B-A3B, a hybrid Mamba-MoE model pretrained on ~27T tokens with deliberately up-weighted German, reports the highest English and German aggregate scores among fully open base models in its comparison while ma...

  12. Simple Policy Gradients for Reasoning with Diffusion Language Models

    cs.LG 2025-10 reject novelty 5.0 of 10

    AGRPO makes GRPO-style policy gradients tractable for diffusion LLMs by Monte-Carlo sampling denoising timesteps, but the unbiasedness claim only holds for a step-level objective, not the token-level GRPO objective.

  13. Domain Specific Benchmarks for Evaluating Multimodal Large Language Models

    cs.LG 2025-06 conditional novelty 3.0 of 10

    A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages · cited by 13 Pith papers

  1. [1]

    Vqa: Visual question answering

    [AAL+15] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. “Vqa: Visual question answering”. In: Proceedings of the IEEE interna- tional conference on computer vision. 2015, pp. 2425–2433. [Ant23] Anthropic. Claude 3.5 Sonnet Model Card Addendum

    work page 2015

  2. [3]

    Training verifiers to solve math word problems

    [CKB+21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. “Training verifiers to solve math word problems”. In: arXiv preprint arXiv:2110.14168 (2021). [CW17] Nicholas Carlini and David Wagner. “Towards evaluating the robustness of neural networks”...

    arXiv 2021

  3. [6]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    arXiv: 2412.19437 [cs.CL] . URL: https://arxiv.org/abs/2412. 19437. [DWD+19] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. “DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs”. In: arXiv preprint arXiv:1903.00161 (2019). [GKS+17] Yash Goyal, Tejas Khot, Douglas Summers-Stay, ...

    arXiv 2019

  4. [7]

    Reasoning with language model is planning with world model

    URL: https://twitter. com/PeterHndrsn/status/1831801148795449410. [HGM+23] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. “Reasoning with language model is planning with world model”. In: arXiv preprint arXiv:2305.14992 (2023). [JYW+24] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir...

    work page arXiv 2023

  5. [8]

    Parsing algebraic word problems into equations

    [KHS+15] Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. “Parsing algebraic word problems into equations”. In:Transactions of the Association for Computational Linguistics 3 (2015), pp. 585–597. [LDM12] Hector Levesque, Ernest Davis, and Leora Morgenstern. “The winograd schema challenge”. In: Thirteenth int...

    work page 2015

  6. [10]

    Pervasive label errors in test sets destabilize machine learning benchmarks

    [NAM21] Curtis G Northcutt, Anish Athalye, and Jonas Mueller. “Pervasive label errors in test sets destabilize machine learning benchmarks”. In: arXiv preprint arXiv:2103.14749 (2021). [NCC+24] Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. “Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art...

    arXiv 2021

  7. [12]

    ] The 15th Nepal China’s Tibet Economic and Trade Fair was held on 17-22 November 2015 in Bhrikutimandap, Kathmandu Nepal

    Mislabeled question, DROP Context: [. . . ] The 15th Nepal China’s Tibet Economic and Trade Fair was held on 17-22 November 2015 in Bhrikutimandap, Kathmandu Nepal. Question: How many days 15th Nepal China’s Tibet Economic and Trade Fair was held in 2015? Solution: 5 The fair was held for 6 days, not

    work page 2015

  8. [2012]

    A Chevy for $1? Car dealer chatbots show perils of AI for customer service

    [Mas23] Bryson Masse. “A Chevy for $1? Car dealer chatbots show perils of AI for customer service”. In: VentureBeat (2023). [MMS+18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. “Towards deep learning models resistant to adversarial attacks”. In:International Conference on Learning Representations (ICLR)

    work page 2023

Show all 14 references
  1. [2016]

    Gpqa: A graduate-level google-proof q&a benchmark

    [RHS+23] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. “Gpqa: A graduate-level google-proof q&a benchmark”. In: arXiv preprint arXiv:2311.12022 (2023). [RJL18] Pranav Rajpurkar, Robin Ji...

    2023 arXiv

  2. [2017]

    Tabfact: A large-scale dataset for table-based fact verification

    [CWC+19] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. “Tabfact: A large-scale dataset for table-based fact verification”. In: arXiv preprint arXiv:1909.02164 (2019). [DGS+18] Krishnamurthy Dvijotham, Sven Gowal...

    2019 arXiv

  3. [2018]

    The llama 3 herd of models

    [DJP+24] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. “The llama 3 herd of models”. In: arXiv preprint arXiv:2407.21783 (2024). [DLF+24] DeepSeek-AI, Aixin Liu, Bei Fe...

    2024 arXiv

  4. [2020]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    [WSM+18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. “GLUE: A multi-task benchmark and analysis platform for natural language understanding”. In: arXiv preprint arXiv:1804.07461 (2018). [WWS+22] Jason Wei, Xuezhi Wang, Dale Schuurman...

    2018 arXiv

  5. [2024]

    What will it take to fix benchmarking in natural language understanding?

    URL: https://www.anthropic.com/news/3-5-models-and-computer-use . [BD21] Samuel R Bowman and George E Dahl. “What will it take to fix benchmarking in natural language understanding?” In: arXiv preprint arXiv:2104.02145 (2021). [Civ24] Civil Resolution Tribunal. Moffatt v. Air ...

    2021 arXiv

  6. [2025]

    Are NLP models really able to solve simple math word problems?

    URL: https : / / openai . com / index / o3 - mini - system-card/. [PBG21] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. “Are NLP models really able to solve simple math word problems?” In: arXiv preprint arXiv:2103.07191 (2021). [PMJ+16] Nicolas Papernot, Patrick McDaniel...

    2021 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.