Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Benchmark scores on SocialIQa, ToMi, and FauxPas-EAI overstate LLM reasoning abilities because high scores can reflect alignment with format-specific cues rather than consistent inference, an audit finds.

desk verdict Full-scale audit is new and the released data is useful, but the central claim about format sensitivity needs a human baseline and label-preservation checks. read the letter →

arxiv 2506.23864 v1 pith:46ZMP7C2 submitted 2025-06-30 cs.CL

Seyed Mahed Mousavi , Edoardo Cecchinato , Lucia Hornikova , Giuseppe Riccardi This is my paper
classification cs.CL
keywords benchmarkauditLLMreasoningSocialIQaToMiFauxPas-EAIpromptsensitivityLLM-as-a-judgeevaluationvalidity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that three widely used reasoning benchmarks contain pervasive flaws in both their items and their scoring procedures, and that high model scores on them often reflect sensitivity to surface wording and format rather than consistent inference. A systematic human audit of SocialIQa, ToMi, and FauxPas-EAI finds that 29.5% of SocialIQa development items are structurally, semantically, or pragmatically flawed, with substantial flaws also in ToMi and FauxPas-EAI. Re-evaluating models on cleaned subsets raises scores notably, but showing semantically equivalent rephrasings of the same items causes large performance swings for all tested models. The authors conclude that current benchmark scores overstate LLM reasoning and advocate for evaluation protocols that track reasoning as an inferential process rather than static output selection.

What carries the argument

The argument is carried by a three-category human audit framework that classifies every benchmark item as structurally, semantically, or pragmatically flawed, and by the cleaned subsets and rephrasing perturbations built from those judgments. Flawed items are removed or corrected to define the cleaned benchmarks; the difference between original and cleaned scores quantifies how much data noise contributes to reported performance. Five semantically equivalent rephrasings per cleaned SocialIQa item create a sensitivity test: score variance across semantically identical inputs measures the model's reliance on surface form. A supplementary LLM-as-a-judge scoring step, in which GPT-4o evaluates semantic validity, replaces string-matching on FauxPas-EAI and reaches near-perfect agreement with humans, showing that meaning-aware scoring is feasible.

What would settle it

An independent second annotation of SocialIQa's 1,954 development items, with inter-annotator agreement reported, followed by evaluating models on only the items both teams agree are clean; if agreement is low or the score gap between original and agreed-clean subsets disappears, the claim that data flaws systematically inflate benchmark scores is not supported.

Watch

Extended reading notes

Core claim

The central claim is that static reasoning benchmarks systematically conflate output accuracy with reasoning competence. Using five LLMs as diagnostic probes and human annotation as ground truth, the paper shows that a large share of model failures on SocialIQa and ToMi trace to faulty data, and that removing or correcting those items inflates scores, e.g., GPT-o1 rises from 80% to 92% on SocialIQa. The decisive evidence is format sensitivity: when each cleaned SocialIQa item is rephrased in five semantically equivalent ways, every model's score fluctuates substantially, even though no content changes. This reveals that high benchmark scores can reflect alignment with format-specific cues rather than stable, input-grounded inference, so benchmark-based reasoning claims are unreliable.

Load-bearing premise

The human audit categories for structural, semantic, and pragmatic flaws are treated as ground truth for which items are faulty, and the same judgments define the cleaned subsets that produce higher scores; if the annotation is biased or inconsistent, the reported flaw rates and score gains do not support the paper's central claim.

Editorial extensions

If this is right

  • Published accuracy numbers on SocialIQa, ToMi, and FauxPas-EAI overstate LLM social-reasoning and theory-of-mind ability if the audit's flaw rates are accurate.
  • Score gaps between newer and older models may be driven by differing alignment with benchmark artifacts rather than by genuine reasoning gains.
  • Evaluation protocols that fragment context into isolated questions systematically underestimate models with sequential reasoning demands, so full-context prompting should become the default.
  • String-matching and Levenshtein scoring should be replaced by meaning-aware scoring, since the LLM-as-a-judge check on FauxPas-EAI shows such scoring can track human judgment closely.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same audit-plus-rephrasing protocol can be exported to logical and mathematical reasoning benchmarks, where the paper's own limitations acknowledge the current scope is restricted to social reasoning and theory of mind.
  • Beyond the paper: if score volatility across rephrasings is adopted as a diagnostic, a natural next standard is reporting a rephrasing-robustness interval alongside every benchmark score, since a point estimate on fixed items is uninformative about reasoning stability.
  • Beyond the paper: the high agreement between GPT-4o judge and human annotations suggests a scalable check, but also invites a test the paper does not run: whether multiple judge models agree with each other, since judge bias could replace benchmark bias.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper conducts a systematic audit of three reasoning benchmarks (SocialIQa, FauxPas-EAI, ToMi) by evaluating five LLMs under standard settings, then performing a human annotation of benchmark items into structural, semantic, and pragmatic flaw categories. It also identifies evaluation-level issues such as context fragmentation, semantics-agnostic string matching, and granularity mismatches. The authors re-evaluate models on cleaned subsets, with and without conversational context, on GPT-4o-generated rephrasings of SocialIQa, and with an LLM-as-a-judge for FauxPas-EAI. Their central claim is that benchmark scores are unreliable and that high scores may reflect alignment with format-specific cues rather than consistent reasoning from input content; they advocate for process-oriented evaluation and release audited data and evaluation tools.

Significance. If the central claim holds, the paper provides a broad, human-verified challenge to current uses of reasoning benchmarks, extending earlier isolated critiques to three widely used datasets with a large annotation effort. The release of audited data, prompts, and evaluation tools is a concrete resource for the community, and the LLM-as-a-judge comparison with human annotations (Cohen's κ = 0.89 vs. 0.77 for Levenshtein) is a useful reproducibility point. However, the central claim rests on subjective human judgments whose reliability is not reported, and the clean-subset and rephrasing analyses are defined by those same judgments; the paper's contribution is therefore more conditional than its abstract suggests.

major comments (4)
  1. [§4.1, Table 1] The human audit is the backbone of the reported flaw rates (e.g., 29.5% for SocialIQa), but no inter-annotator agreement is reported. The text states that two researchers reviewed each item under senior supervision, yet it does not describe how disagreements were resolved or whether the categories are reliable. Without such information, the structural/semantic/pragmatic taxonomy cannot be distinguished from annotator subjectivity, and the downstream clean-subset analysis inherits this uncertainty.
  2. [§5A, Table 2] The cleaned subsets are defined by the same human audit that establishes the benchmarks as flawed; for SocialIQa, the clean subset is obtained by removing the 576 items flagged in Table 1. The observed improvements (e.g., GPT-o1 from 80% to 92%) are therefore partly self-fulfilling: the audit flags items it regards as flawed and then shows that scores rise when those items are removed. An independent or pre-registered annotation, or a sensitivity analysis removing items by an objective criterion (e.g., only duplicated items), is needed to support the claim that benchmark scores are inflated by data flaws.
  3. [§5C, Figure 3] The rephrasing experiment is the most direct evidence for the claim that high scores reflect format-specific cues, but it does not verify that the gold answer remains uniquely correct after GPT-4o rewrites each item. Only a random 15% of generated items were manually checked, and the check covered 'semantic equivalence and linguistic correctness', not whether the originally correct option remains the only defensible answer. No human accuracy baseline on the rephrased items is reported, so the large score fluctuations in Figure 3 could be caused by label-preservation failures in generation rather than by model brittleness. The paper should add a label-preservation check on all or a representative sample of rephrased items and report human performance on the same rephrasings.
  4. [§4.1, Table 1] There is a numerical inconsistency for ToMi structural flaws: the text reports '≈0.8%' while Table 1 lists '24% (50 Qs)'; 50 out of 600 questions is 8.3%, not 0.8% or 24%. Because Table 1 is the primary quantitative support for the audit, this discrepancy needs to be corrected and the story-level versus question-level basis of the percentages stated explicitly.
minor comments (6)
  1. [Abstract and Introduction] Several sentences appear garbled, including 'model scores often improve not due to due to erratic surface wording variations' in the abstract and 'current evaluation setups often the reasoning ability' in the Introduction; these need proofreading.
  2. [Figures 1 and 2] The y-axis label 'Erorrs' is a typo for 'Errors'.
  3. [§5C, Figure 3] Figure 3 shows only point estimates across five rephrasings; reporting per-item variance or confidence intervals would make the volatility claim more robust.
  4. [§5C, Limitations] The Limitations section acknowledges the single backbone model for rephrasings, but it does not mention the absence of a label-preservation check or a human baseline, which are the main threats to this experiment.
  5. [Footnote 1] The data and code link is given as 'Repo Link'; a concrete URL or repository identifier should be provided.
  6. [Throughout] The benchmark name is spelled inconsistently as both 'SocialIQA' and 'SocialIQa' across the text and tables; please standardize.

Circularity Check

1 steps flagged · score 5.0 of 10

ToMi cleaned-data gains are partly by construction (auditors' 'semantically valid' answers are rescored as correct), but the central claim also rests on independent manipulations, yielding partial circularity.

  1. fitted input called prediction [Section 4.2 (ToMi, Granularity Mismatch) and Section 5.A (Evaluation on Cleaned Data), Table 2]
    "In ≈30% of responses marked as incorrect, we observed a granularity mismatch between the expected answer and the model’s semantically valid but coarser output. ... While such answers are semantically valid, they are nonetheless penalized. ... We filtered out flawed items from SocialIQA and FauxPas-EAI, corrected labeling and agent-role inconsistencies in ToMi, adjusted scoring procedures to account for granularity mismatches, and manually reviewed all model responses. As shown in Table 2, all models exhibit improved accuracy on the cleaned datasets."

    The ToMi improvement is the same human judgment restated as an outcome. Section 4.2 defines a granularity-mismatch 'flaw' as a model output that the auditors consider semantically valid even though the scorer marks it wrong. Section 5.A then 'adjust[s] scoring procedures' to count those outputs as correct; any accuracy gain from this adjustment is arithmetic reclassification, not a new measurement. Likewise, Section 4.1 identifies ToMi structural flaws as 'incorrect ground truth labels' that make 'valid model responses' wrong, and Section 5.A 'correct[s] labeling'; the resulting score rise is entailed by the correction rather than independent evidence that the original benchmark overstates or understates reasoning.

full rationale

This paper is an empirical audit rather than a formal derivation, so equation-level circularity is limited. The strongest circular component is the ToMi cleaned-data evaluation: the categories of 'granularity mismatch' and 'incorrect ground truth' are defined by the same human judgments that are then used to rescore or relabel items, so the reported accuracy gains on the cleaned ToMi set are partly the arithmetic consequence of counting previously penalized responses as correct. This is a real by-construction component, but it covers only one benchmark and one of four analyses. The central claim also rests on independent manipulations: the context-aware evaluation (Section 5.B) changes prompt structure without relabeling, the LLM-as-a-judge comparison (Section 5.D) is an external agreement measure, and the raw rephrasing volatility (Section 5.C) is an observed empirical fact. The rephrasing experiment has a validity gap—only 15% of rewrites are checked, and label preservation is not verified—so 'no change in content' is assumed from the GPT-4o generation prompt rather than established; this weakens the inference but is not itself a circular reduction. The self-citation to Mousavi et al. (2022) for the human evaluation framework is methodology, not a load-bearing theorem, and the Limitations section transparently acknowledges the single-backbone rephrasing limitation. Overall, partial circularity in the ToMi cleaning analysis, with independent content elsewhere, supports a moderate score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted numerical parameters or new theoretical entities. Its central claims depend on the reliability of human audit judgments, semantic preservation in generated rephrasings, and the interpreted link between score volatility and lack of reasoning. These are domain assumptions rather than free parameters or invented constructs.

assumptions (4)
  • domain assumption Human annotation by two researchers is a valid ground truth for identifying flawed benchmark items.
    Used throughout Section 4 to classify 29.5% of SocialIQa and large fractions of ToMi as flawed; no inter-annotator agreement is reported for these judgments.
  • domain assumption GPT-4o-generated rephrasings preserve semantics; the 15% manual validation is sufficient to guarantee equivalence.
    Section 5.C relies on this for the format-sensitivity analysis; if unvalidated rephrasings alter meaning, the observed score changes are confounded.
  • domain assumption Sensitivity to rephrasing indicates absence of stable reasoning rather than inherent ambiguity of the original items.
    This interpretive link underpins the conclusion in Sections 5.C and 6, but no human baseline on the rephrasings is provided to rule out item ambiguity.
  • domain assumption LLM-as-a-judge with Cohen's kappa 0.89 on FauxPas-EAI is a valid surrogate for meaning-aware scoring.
    Section 5.D tests this only on one small benchmark and with one judge model, limiting generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It." pith.science (2026). https://pith.science/paper/46ZMP7C2

@misc{pith2026250623864,
  author       = {Pith},
  title        = {Pith review of: Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46ZMP7C2}},
  note         = {Machine review of arXiv:2506.23864}
}
read the original abstract

We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi, and uncover pervasive flaws in both benchmark items and evaluation methodology. Using five LLMs (GPT-{3, 3.5, 4, o1}, and LLaMA 3.1) as diagnostic tools, we identify structural, semantic, and pragmatic issues in benchmark design (e.g., duplicated items, ambiguous wording, and implausible answers), as well as scoring procedures that prioritize output form over reasoning process. Through systematic human annotation and re-evaluation on cleaned benchmark subsets, we find that model scores often improve not due to due to erratic surface wording variations and not to improved reasoning. Infact, further analyses show that model performance is highly sensitive to minor input variations such as context availability and phrasing, revealing that high scores may reflect alignment with format-specific cues rather than consistent inference based on the input. These findings challenge the validity of current benchmark-based claims about reasoning in LLMs, and highlight the need for evaluation protocols that assess reasoning as a process of drawing inference from available information, rather than as static output selection. We release audited data and evaluation tools to support more interpretable and diagnostic assessments of model reasoning.

Figures

Figures reproduced from arXiv: 2506.23864 by the authors.

Figure 1
Figure 1. Breakdown of total errors for each model on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Breakdown of total errors for each model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance on SocialIQa (clean subset) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 8 canonical work pages

  1. [1]

    Simon Baron-Cohen, Michelle O'riordan, Valerie Stone, Rosie Jones, and Kate Plaisted. 1999. Recognition of faux pas by normally developing children and children with asperger syndrome or high-functioning autism. Journal of autism and developmental disorders, 29:407--418

    work page 1999

  2. [2]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406.18403

    arXiv 2024

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Xanh Ho, Jiahao Huang, Florian Boudin, and Akiko Aizawa. 2025. Llm-as-a-judge: Reassessing the performance of llms in extractive qa. arXiv preprint arXiv:2504.11972

    arXiv 2025

  5. [5]

    Holliday, Matthew Mandelkern, and Cedegao E

    Wesley H. Holliday, Matthew Mandelkern, and Cedegao E. Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.222 Conditional and modal reasoning in large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3800--3821, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.222 2024

  6. [6]

    Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, and Dan Roth. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.272 A peek into token bias: Large language models are not yet genuine reasoners . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4722--475...

    doi:10.18653/v1/2024.emnlp-main.272 2024

  7. [7]

    Abhinav Joshi, Areeb Ahmad, Divyaksh Shukla, and Ashutosh Modi. 2025. https://aclanthology.org/2025.naacl-long.487/ Towards quantifying commonsense reasoning with mechanistic insights . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long ...

    work page 2025

  8. [8]

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. https://doi.org/10.18653/v1/D19-1598 Revisiting the evaluation of theory of mind through question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872-...

    doi:10.18653/v1/d19-1598 2019

Show all 24 references
  1. [9]

    Levenshtein

    Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8):707--710

    1966

  2. [10]

    Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. 2025. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.15612

    2025 arXiv

  3. [11]

    Ziyi Liu, Soumya Sanyal, Isabelle Lee, Yongkang Du, Rahul Gupta, Yang Liu, and Jieyu Zhao. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.213 Self-contradictory reasoning evaluation and detection . In Findings of the Association for Computational Linguistics: EMNLP 2024...

    2024 doi

  4. [12]

    Philipp Mondorf and Barbara Plank. 2024. https://openreview.net/forum?id=Lmjgl2n11u Beyond accuracy: Evaluating the reasoning behavior of large language models - a survey . In First Conference on Language Modeling

    2024

  5. [13]

    Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2025. Llms as repositories of factual knowledge: Limitations and solutions. arXiv preprint arXiv:2501.12774

    2025

  6. [14]

    Seyed Mahed Mousavi, Gabriel Roccabruna, Michela Lorandi, Simone Caldarella, and Giuseppe Riccardi. 2022. https://doi.org/10.18653/v1/2022.gem-1.12 Evaluation of response generation models: Shouldn`t it be shareable and replicable? In Proceedings of the 2nd Workshop on Natural...

    2022 doi

  7. [15]

    Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.198 Plausibly problematic questions in multiple-choice benchmarks for commonsense reasoning . In Findings of the Associatio...

    2024 doi

  8. [16]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  9. [17]

    Sneheel Sarangi, Maha Elgarf, and Hanan Salam. 2025. https://aclanthology.org/2025.coling-main.682/ Decompose- T o M : Enhancing theory of mind reasoning in large language models through simulation and task decomposition . In Proceedings of the 31st International Conference on...

    2025

  10. [18]

    Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2024. https://aclanthology.org/2024.eacl-long.138/ Clever hans or neural theory of mind? stress testing social reasoning in large language models . In Procee...

    2024

  11. [19]

    Natalie Shapira, Guy Zwirn, and Yoav Goldberg. 2023. https://doi.org/10.18653/v1/2023.findings-acl.663 How well do large language models perform on faux pas tests? In Findings of the Association for Computational Linguistics: ACL 2023, pages 10438--10451, Toronto, Canada. Asso...

    2023 doi

  12. [20]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  13. [21]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    2023

  14. [22]

    Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. 2025. Soft thinking: Unlocking the reasoning potential of llms in continuous concept space. arXiv preprint arXiv:2505.15778

    2025 arXiv

  15. [23]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.