Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a dynamically generated, parameterized benchmark of witness-testimony puzzles built on formal argument attack graphs, and shows that current large language models fail on simple even-length attack chains, with no…

desk verdict Solid formal benchmark with a real NL-validity caveat; worth reviewing. read the letter →

arxiv 2505.01539 v1 pith:EY56VNDD submitted 2025-05-02 cs.AI cs.LG

Cor Steging , Silja Renooij , Bart Verheij This is my paper · ORCID
classification cs.AIcs.LG
keywords largelanguagemodelsreasoningevaluationargumentationabstractargumentattackgraphsbenchmarkswitnesstestimonydatacontaminationreinstatement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to build reasoning benchmarks that are generated fresh each time, so models cannot memorize the answers, and that scale in difficulty by design. The tasks are witness-testimony puzzles whose correct answer is fixed by a formal argument-attack graph, not by the surface wording. Using this benchmark, the paper claims to show that seven current language models reason brittly: they often fail at low complexity, especially on even-length attack chains, and produce inconsistent answers across variations of the same logical structure. A reasoning-specialized model solves all linear puzzles but still fails on selected hard non-linear ones.

What carries the argument

The central object is an abstract argument attack graph: a set of arguments with directed attack edges, evaluated under Dung-style argumentation semantics, which agree on these graphs. In a linear graph of $n$ arguments, the first argument is accepted exactly when $n$ is odd, because each new attack shifts the 'last word' and an attacked attack reinstates the earlier argument. In a non-linear graph, the main argument is accepted if and only if every attacking chain attached to it has even length. These graphs are converted into natural-language witness puzzles using an ontology of 474 names and 90 statements, so every formal graph yields many distinct prompts with the same ground truth.

What would settle it

Run the same witness prompts on a panel of human readers who are given only the rule 'Witnesses should be believed unless there is testimony that they are lying.' If a substantial fraction of readers answer 'yes' for an even-length chain such as Alice–Bob–Charlie–Dan, failing to apply reinstatement, then the natural-language rule does not convey the intended formal semantics, and the benchmark's ground truth is not unambiguously communicated.

Watch

Extended reading notes

Core claim

The paper's central claim is that a benchmark generated from parameterized abstract argument attack graphs, translated into natural language by an ontology, can evaluate whether generative language models genuinely reason rather than recall. On this benchmark, most tested models show a systematic failure: for linear chains with an even number of arguments, where the formal answer is 'no', they tend to accept the first witness and answer 'yes'. Performance drops as chains lengthen, non-linear graphs are harder than linear ones, and shuffling the order of statements helps some models and hurts others, indicating that order itself, not the logical structure alone, drives many answers.

Load-bearing premise

The load-bearing premise is that the single sentence 'Witnesses should be believed unless there is testimony that they are lying' makes readers and models infer the full formal rule, including that a liar's accusation does not count; the paper never tests this rule on human readers.

Editorial extensions

If this is right

  • If the benchmark measures what it claims, the same pipeline can produce fresh, non-reusable tests for any future model, directly addressing data contamination in static legal reasoning benchmarks.
  • The recurring failure on even-length linear chains points to a positional bias: models tend to believe the first-mentioned witness, so prompt order is itself a reasoning confound that the benchmark exposes.
  • Because the graphs have unambiguous formal interpretations, model errors can be traced to specific inference steps such as failing to reinstate an attacked argument, rather than to ambiguous wording.
  • Scaling the graph parameters upward should continue to separate models as they improve, allowing the benchmark to track reasoning gains over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same attack-graph generator could be extended to support relations other than attack, such as support or undermining, and to richer legal fact patterns, making it a general template for argument-based reasoning evaluation.
  • Beyond the paper: because the formal graphs agree across argumentation semantics, the generated puzzles could be used as a training curriculum with checkable targets, teaching models to apply reinstatement rather than simply to answer 'yes' or 'no'.
  • Beyond the paper: the observed order-sensitivity suggests a direct testable extension: presenting the last witness first, as in the shuffled condition, should flip many model answers, and quantifying that flip would give a clean measure of positional bias.
  • Beyond the paper: the authors note the models' non-determinism without controlling for it; repeated sampling of the same prompt would separate random fluctuation from systematic reasoning failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper proposes a method for dynamically generating argumentation-based reasoning benchmarks as a means of evaluating large language models. The approach generates abstract argument attack graphs — both linear and non-linear — of parametrized complexity, automatically translates them into natural-language witness-testimony puzzles via an ontology of names and statements, and computes ground-truth answers from Dung's formal argumentation semantics. The authors evaluate seven commercial LLMs on three benchmark sets (linear, non-linear, and shuffled non-linear) using accuracy, F1, recall, precision, and MCC, reporting that most models perform poorly even at low complexity, show an odd-even pattern of failures on linear graphs, and exhibit inconsistent performance across prompt variations. A reasoning-specialized model, o1-preview, solves all tested linear prompts but fails on selected non-linear prompts. The paper argues that the dynamic generation and ontological variation mitigate data contamination.

Significance. The contribution is significant if the benchmark's validity assumption holds: the formal ground truth is derived independently of model behavior, the code and data are released, and the use of MCC for the heavily skewed non-linear benchmarks is methodologically sound. The demonstration that multiple state-of-the-art models fail at low-complexity, formally clear tasks is a potentially useful data point on LLM reasoning robustness. However, the benchmark's validity rests on an untested assumption that the natural-language rule conveys the intended formal semantics, including reinstatement, to human readers. Since all model scores are measured against formal labels, the paper's central empirical claims could be undermined if the prompts are ambiguous. The absence of a human baseline or any annotator agreement study is therefore a consequential gap, not a minor omission.

major comments (2)
  1. [Section 2.3, Figure 1b] The claim that the generated tasks have 'formally unambiguous interpretations' is only established for the attack graphs, not for the natural-language prompts the models actually receive. The rule 'Witnesses should be believed unless there is testimony that they are lying' does not by itself convey the formal property of reinstatement (that an attack by an attacked witness fails). For a three-node linear chain, a natural reading can support either 'no' (there is direct testimony from the second witness that the first witness lies) or 'yes' (the second witness is not credible because the third witness says the second lies). The paper provides no human-subject validation that readers infer the intended semantics, and does not discuss this ambiguity. Because every accuracy, F1, and MCC result is computed against the formal labels, the central empirical conclusion that LLMs are brittle could reflect a systematic mismatch between the formal ground truth and the natural-language interpretation of the task.
  2. [Section 5.1, Figure 8] The paper's own qualitative example illustrates the ambiguity. GPT-4o-mini's response to the two-argument prompt — that Alice should be believed because there is no evidence Bob is lying — is a plausible reading of the rule, since the prompt does not state that an accusation is itself subject to a credibility check. The authors label this as a reasoning failure, but under an alternative but reasonable interpretation of the rule, the formal answer is not the only one a competent reader might derive. A human baseline or annotator study is needed to establish that the intended formal reading is the one that human readers adopt; without it, the benchmark's validity as a test of reasoning rather than of prompt interpretation is unverified.
minor comments (6)
  1. [Section 3.2] The statement that the correct answer is 'yes' in '52% of the 2,500 (or 48% of the 5,000) prompts' is incorrect for the 5,000-prompt set; with 1 to 50 arguments there are 25 odd and 25 even lengths, so the expected fraction is 50%.
  2. [Section 5.4] The phrase 'the models do not generate reasoning texts that are always correct nor always incorrect' should be rephrased, e.g., 'neither always correct nor always incorrect,' for clarity.
  3. [Figure 4] The y-axis of Figure 4 is scaled from 0 to 50, but the caption does not state this; because the models' MCC values range up to about 49, the scaling compresses visible differences and should be explicitly noted.
  4. [Section 5.1] The claim that o1-preview 'performs every task perfectly' is based on only 75 prompts (three per length); the lack of statistical power should be acknowledged in that sentence.
  5. [Section 2.2] The set notation in the definition of non-linear attack graphs (the indexed union and intersection symbols) renders awkwardly in the PDF; please reformat for readability.
  6. [Section 4] The paper does not report confidence intervals or significance tests for the MCC differences between models or between shuffled and non-shuffled conditions; given the small number of prompt variations per graph (five), some reported differences may be within noise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark ground truth is computed from external formal argumentation semantics and is independent of model outputs.

full rationale

The paper's central claim is that dynamically generated, parametrized prompts with formally unambiguous interpretations can benchmark LLM reasoning. The derivation chain for the benchmark's ground truth is self-contained and external: attack graphs are defined formally in Sections 2.1 and 2.2, the evaluation of arguments follows Dung's abstract argumentation semantics as introduced in reference [4] (an independent, machine-checkable formal theory), and Section 2.3 states that 'When generating the prompts, we also store the correct answer to easily evaluate the models' performance.' The correct answer for a linear graph is 'yes' for odd length and 'no' for even length, and for non-linear graphs it is 'yes' iff every attached directed path has even length; these conditions are computed from the graph structure and the formal semantics, not from any fitted model parameter or from model responses. No parameter is fitted to a subset of data and then used to 'predict' a closely related quantity; no uniqueness theorem from the authors' prior work is invoked to force a choice; and the authors' self-citations ([12], [13]) are motivational or contextual and are not load-bearing for the benchmark construction or for the reported results. The only substantive concern is whether the natural-language rule 'Witnesses should be believed unless there is testimony that they are lying' unambiguously conveys reinstatement to human readers, which is a construct-validity or ambiguity threat, not a circularity threat: the formal ground truth is not defined in terms of the model behavior that the benchmark is meant to measure. Under the stated rules, a paper whose benchmark is self-contained against an external formal standard and whose conclusions are measured against that standard should receive a low circularity score, and no specific reduction from outputs to inputs can be exhibited. Therefore the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not fit any numbers to model outputs; the only modeling choices are the formal semantics (from Dung's work), the mapping from testimony to attack graphs, and the natural-language rendering. The main load-bearing assumptions are that the formal semantics match the intended legal notion of credibility and that the generated text preserves the graph structure unambiguously.

assumptions (4)
  • domain assumption Dung's abstract argumentation frameworks and their grounded, stable, and preferred semantics are the correct formal model for belief in witness testimony under the puzzle rule.
    The benchmark's ground truth is computed using these semantics (Sections 2.1 and 2.2), adopting the 'last word' principle; this mapping from legal testimony to abstract argumentation is assumed, not empirically established.
  • standard math The grounded, stable, and preferred semantics coincide on the generated linear and non-linear attack graphs, giving a unique intended outcome.
    Stated in Section 2.1 for linear graphs and used in Section 2.2 for non-linear graphs; true for these graph classes but unproved in the paper.
  • ad hoc to paper The ontology-based natural-language rendering preserves the identity and direction of each attack, so a reader can recover the graph from the prompt.
    The design assumes the generated sentences 'X says Y is lying' transparently encode edges; this is central to the benchmark's validity and is not tested against human interpretation.
  • domain assumption A forced binary yes/no answer is an appropriate response mode for the task.
    Models are instructed to answer only yes or no, which forbids expressing uncertainty; this may disadvantage models that correctly detect underdetermination (the paper does not allow such responses).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models." pith.science (2026). https://pith.science/paper/EY56VNDD

@misc{pith2026250501539,
  author       = {Pith},
  title        = {Pith review of: Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EY56VNDD}},
  note         = {Machine review of arXiv:2505.01539}
}
read the original abstract

Generative large language models as tools in the legal domain have the potential to improve the justice system. However, the reasoning behavior of current generative models is brittle and poorly understood, hence cannot be responsibly applied in the domains of law and evidence. In this paper, we introduce an approach for creating benchmarks that can be used to evaluate the reasoning capabilities of generative language models. These benchmarks are dynamically varied, scalable in their complexity, and have formally unambiguous interpretations. In this study, we illustrate the approach on the basis of witness testimony, focusing on the underlying argument attack structure. We dynamically generate both linear and non-linear argument attack graphs of varying complexity and translate these into reasoning puzzles about witness testimony expressed in natural language. We show that state-of-the-art large language models often fail in these reasoning puzzles, already at low complexity. Obvious mistakes are made by the models, and their inconsistent performance indicates that their reasoning capabilities are brittle. Furthermore, at higher complexity, even state-of-the-art models specifically presented for reasoning capabilities make mistakes. We show the viability of using a parametrized benchmark with varying complexity to evaluate the reasoning capabilities of generative language models. As such, the findings contribute to a better understanding of the limitations of the reasoning capabilities of generative models, which is essential when designing responsible AI systems in the legal domain.

Figures

Figures reproduced from arXiv: 2505.01539 by the authors.

Figure 1
Figure 1. Example of a linear argument attack graph and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The pipeline of our approach for generating dynamic benchmarks of scaling complexity. In this paper, the graph [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. The mean MCC of models on prompts that are [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Linear attack graphs: percentage of correct answers versus the number of arguments in the prompt [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Non-linear attack graphs: percentage of correct answers versus the number of arguments in the prompt, for prompts [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Non-linear attack graphs: percentage of correct answers versus the number of directed paths in the prompt, based on [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: it accepts the main argument and claims that there is no [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Response of GPT-4o to the prompt in Figure 1b [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Response of GPT-4o to the prompt from Figure 2b [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Acquiescence Bias in Large Language Models

    cs.CL 2025-09 reject novelty 5.0 of 10

    LLMs in this study did not show human-like acquiescence bias; they showed a tendency to answer 'no' in English yes/no rephrasings, though the evidence for a bias independent of question meaning is weak.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Baroni, F

    P. Baroni, F. Toni, and B. Verheij. 2020. Introduction to the special issue ‘On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games: 25 years later’. Argument & Computation, 11, 1–2, 1–14

    work page 2020

  2. [2]

    Chien and M

    C.V. Chien and M. Kim. 2024. How generative AI can help address the access to justice gap through the courts. Loyola of Los Angeles Law Review

    work page 2024

  3. [3]

    M. Dahl, V. Magesh, M. Suzgun, and D. E. Ho. 2024. Large legal fictions: profiling legal hallucinations in large language models. Journal of Legal Analysis, 16, 1, 64–93

    work page 2024

  4. [4]

    P. M. Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence, 77, 321–357

    work page 1995

  5. [5]

    Guha et al

    N. Guha et al. 2024. LegalBench: a collaboratively built benchmark for mea- suring legal reasoning in large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23) Article 1915. Curran Associates, New Orleans, LA, USA, 157 pages

    work page 2024

  6. [6]

    Huang and K.C.C

    J. Huang and K.C.C. Chang. 2023. Towards reasoning in large language models: a survey. In Findings of the Association for Computational Linguistics: ACL 2023 . A. Rogers, J. Boyd-Graber, and N. Okazaki, (Eds.) Association for Computational Linguistics, Toronto, Canada, (July 2023), 1049–1065

    work page 2023

  7. [7]

    Ji et al

    Z. Ji et al. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55, 12, Article 248, 38 pages

    work page 2023

  8. [8]

    M. Kant, M. Kant, M. Nabi, P. Carlson, and M. Ma. 2024. Equitable access to justice: logical LLMs show promise. (2024)

    work page 2024

Show all 16 references
  1. [9]

    Liga and L

    D. Liga and L. Robaldo. 2023. Fine-tuning gpt-3 for legal rule classification. Computer Law & Security Review , 51, 105864

    2023

  2. [10]

    Savelka and K.D

    J. Savelka and K.D. Ashley. 2023. The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6

    2023

  3. [11]

    Steenhuis, B

    Q. Steenhuis, B. Willey, and D. Colarusso. 2023. Beyond readability with Rate- MyPDF: a combined rule-based and machine learning approach to improving court forms. In Proceedings of the Nineteenth International Conference on Artifi- cial Intelligence and Law , 287–296

    2023

  4. [12]

    Steging, S

    C. Steging, S. Renooij, and B. Verheij. 2021. Discovering the rationale of de- cisions: Towards a method for aligning learning and reasoning. In ICAIL ’21: Proceedings of the Eighteenth International Conference for Artificial Intelligence and Law. J. Maranhão and A. Wyner, (Ed...

    2021

  5. [13]

    Steging and L

    C. Steging and L. van Leeuwen. 2024. A hybrid approach to legal textual entail- ment. In JSAI-isAI ’24: Sixteenth JSAI International Symposia on AI . Eighteenth International Workshop on Juris-Informatics (JURISIN 2024) . Hamamatsu, Japan, 154–169

    2024

  6. [14]

    T. Webb, K. Holyoak, and H. Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7, 1–16

    2023

  7. [15]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q.V. Le, D. Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35, 24824–24837

    2022

  8. [16]

    Westermann

    H. Westermann. 2024. Dallma: semi-structured legal reasoning and drafting with large language models. In 2nd Workshop on Generative AI and Law, co- located with the International Conference on Machine Learning (ICML) (Vienna, Austria, 2024)

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.