{"id":"7797d5f0-f412-4822-8a31-a9bbd034abe9","arxiv_id":"2505.01539","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A parametrized benchmark of argumentation-based reasoning puzzles shows that leading LLMs are brittle even at low complexity, with perfect linear performance only from OpenAI's o1-preview.","lead":"The paper introduces a dynamic, scalable benchmark that turns formal argumentation attack graphs into natural-language witness-testimony puzzles, and shows that several commercial language models fail even simple versions. The approach aims to avoid data contamination and to measure reasoning brittleness in models considered for legal applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's natural-language rule may not unambiguously encode Dung reinstatement; without a human baseline, low LLM accuracy could reflect prompt ambiguity rather than brittle reasoning.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the natural-language rule is assumed to convey Dung semantics, including reinstatement, without human validation. I agree this is the most serious issue because it sits between the formally correct graph construction and every empirical result. The formal core of the paper is sound: the attack graphs, the parity criterion for linear graphs, and the all-even-paths criterion for non-linear graphs are correctly derived from Dung's semantics. The dynamic generation and the use of MCC as the headline metric are also appropriate, given the strong class imbalance in the non-linear benchmark. What is not secured is the translation layer. The prompt wording 'Witnesses should be believed unless there is testimony that they are lying' does not transparently express the defeat of attacks by attacked attackers, and the paper gives no evidence that competent human readers recover the intended labels. Qualitative model outputs, such as GPT-4o-mini's refusal to accept Bob's testimony without corroborating evidence, show that some models adopt a different default-credibility reading; without a human baseline, this is not necessarily a reasoning error relative to the natural-language task. The o1-preview comparison is also weakened by selective hard-prompt sampling, and the absence of repeated sampling leaves variance unmeasured, but those are secondary. The human-subject check would resolve the primary concern: if humans agree with the formal labels, the benchmark is validated and the brittleness conclusion stands; if they do not, the benchmark's labels do not represent the intended reasoning task. Since the reader already made the verdict CONDITIONAL and this concern supports that condition, no adjustment is needed.","tokens_in":13624,"tokens_out":4623,"duration_ms":54790,"concrete_test":"Run a human-subject study on a stratified sample of prompts, including linear chains of lengths 1–7 and non-linear graphs that require reinstatement (for example, a chain where the attacker of the main argument is itself attacked). Present the exact prompt and rule text from Section 2.3 to 20–30 lay participants or law students and ask for the yes/no answer. Compute agreement with the formal labels. If agreement is at least 90% overall and at least 85% on reinstatement-critical cases, the ambiguity concern is resolved; if agreement drops substantially on those cases, the benchmark's natural-language ground truth is not established and the model-failure conclusions need to be re-interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the generated benchmarks have formally unambiguous interpretations, but that formal unambiguity holds for the attack graphs, not necessarily for the natural-language prompts the models actually see. In Section 2.3, the rule 'Witnesses should be believed unless there is testimony that they are lying' is used to instantiate Dung semantics, and Figure 1b presents this rule to the model as a reasoning puzzle. Under Dung's grounded semantics, an attack by an attacked argument is itself defeated, so for a three-node linear chain A1←A2←A3 the intended answer is 'yes' because A1 is reinstated. A natural reading of the rule, however, can support 'no': there is testimony from A2 that A1 lies, and unless one recursively applies credibility to the accusers, that testimony defeats A1. The paper reports no human-subject validation that untrained readers infer the formal semantics, including reinstatement, from the prompt wording. Since every accuracy, F1, and MCC result is measured against these formal labels, the central empirical conclusion that models are brittle could instead reflect a systematic mismatch between the formal ground truth and the natural-language interpretation of the task. The two-argument example in Figure 1b is not diagnostic because formal and naive readings coincide there; the ambiguous cases are longer chains and non-linear graphs where attacked attackers should fail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method for dynamically generating argumentation-based reasoning benchmarks as a means of evaluating large language models. The approach generates abstract argument attack graphs — both linear and non-linear — of parametrized complexity, automatically translates them into natural-language witness-testimony puzzles via an ontology of names and statements, and computes ground-truth answers from Dung's formal argumentation semantics. The authors evaluate seven commercial LLMs on three benchmark sets (linear, non-linear, and shuffled non-linear) using accuracy, F1, recall, precision, and MCC, reporting that most models perform poorly even at low complexity, show an odd-even pattern of failures on linear graphs, and exhibit inconsistent performance across prompt variations. A reasoning-specialized model, o1-preview, solves all tested linear prompts but fails on selected non-linear prompts. The paper argues that the dynamic generation and ontological variation mitigate data contamination.","tokens_in":13862,"tokens_out":6998,"duration_ms":62875,"significance":"The contribution is significant if the benchmark's validity assumption holds: the formal ground truth is derived independently of model behavior, the code and data are released, and the use of MCC for the heavily skewed non-linear benchmarks is methodologically sound. The demonstration that multiple state-of-the-art models fail at low-complexity, formally clear tasks is a potentially useful data point on LLM reasoning robustness. However, the benchmark's validity rests on an untested assumption that the natural-language rule conveys the intended formal semantics, including reinstatement, to human readers. Since all model scores are measured against formal labels, the paper's central empirical claims could be undermined if the prompts are ambiguous. The absence of a human baseline or any annotator agreement study is therefore a consequential gap, not a minor omission.","major_comments":[{"comment":"The claim that the generated tasks have 'formally unambiguous interpretations' is only established for the attack graphs, not for the natural-language prompts the models actually receive. The rule 'Witnesses should be believed unless there is testimony that they are lying' does not by itself convey the formal property of reinstatement (that an attack by an attacked witness fails). For a three-node linear chain, a natural reading can support either 'no' (there is direct testimony from the second witness that the first witness lies) or 'yes' (the second witness is not credible because the third witness says the second lies). The paper provides no human-subject validation that readers infer the intended semantics, and does not discuss this ambiguity. Because every accuracy, F1, and MCC result is computed against the formal labels, the central empirical conclusion that LLMs are brittle could reflect a systematic mismatch between the formal ground truth and the natural-language interpretation of the task.","section":"Section 2.3, Figure 1b"},{"comment":"The paper's own qualitative example illustrates the ambiguity. GPT-4o-mini's response to the two-argument prompt — that Alice should be believed because there is no evidence Bob is lying — is a plausible reading of the rule, since the prompt does not state that an accusation is itself subject to a credibility check. The authors label this as a reasoning failure, but under an alternative but reasonable interpretation of the rule, the formal answer is not the only one a competent reader might derive. A human baseline or annotator study is needed to establish that the intended formal reading is the one that human readers adopt; without it, the benchmark's validity as a test of reasoning rather than of prompt interpretation is unverified.","section":"Section 5.1, Figure 8"}],"minor_comments":[{"comment":"The statement that the correct answer is 'yes' in '52% of the 2,500 (or 48% of the 5,000) prompts' is incorrect for the 5,000-prompt set; with 1 to 50 arguments there are 25 odd and 25 even lengths, so the expected fraction is 50%.","section":"Section 3.2"},{"comment":"The phrase 'the models do not generate reasoning texts that are always correct nor always incorrect' should be rephrased, e.g., 'neither always correct nor always incorrect,' for clarity.","section":"Section 5.4"},{"comment":"The y-axis of Figure 4 is scaled from 0 to 50, but the caption does not state this; because the models' MCC values range up to about 49, the scaling compresses visible differences and should be explicitly noted.","section":"Figure 4"},{"comment":"The claim that o1-preview 'performs every task perfectly' is based on only 75 prompts (three per length); the lack of statistical power should be acknowledged in that sentence.","section":"Section 5.1"},{"comment":"The set notation in the definition of non-linear attack graphs (the indexed union and intersection symbols) renders awkwardly in the PDF; please reformat for readability.","section":"Section 2.2"},{"comment":"The paper does not report confidence intervals or significance tests for the MCC differences between models or between shuffled and non-shuffled conditions; given the small number of prompt variations per graph (five), some reported differences may be within noise.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short conference contribution and the requested human validation could be seen as beyond the initial scope, but the validity of the benchmark and the interpretation of all empirical results hinge on it. It would be reasonable to require either a small human-subject study or a substantial revision of the claims and limitations. Additionally, the 'formally unambiguous' phrasing overstates what is demonstrated; the manuscript would be strengthened by carefully distinguishing formal graph semantics from natural-language rendering ambiguity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nQuick take: this paper is genuinely useful, but the central empirical claim—that LLMs are brittle on formally unambiguous tasks—rests on an assumption about the natural-language prompt that is not validated. The formal core is sound; the NL layer is where I'd push.\n\nWhat's new: they generate benchmarks by translating Dung-style abstract argumentation attack graphs into witness-testimony puzzles, using an ontology-based randomizer to make each prompt unique. That combination—parametrized complexity, formal ground truth, dynamic wording—is a real step against data contamination, and they ship the code and data. The evaluation is honest: they report MCC alongside accuracy, acknowledge the class imbalance, and show per-length and per-structure plots. The observation that models often accept the first argument they read—and that shuffling can sometimes help—is well supported by the verbatim outputs.\n\nWhere it gets soft: the prompt rule 'Witnesses should be believed unless there is testimony that they are lying' is doing a lot of work. In the formal graphs, an attack by an attacked argument is defeated (reinstatement). Does a typical reader infer that from that one sentence? For two arguments, no problem. For a three-node chain Alice←Bob←Charlie, the intended answer is 'yes' (Charlie defeats Bob, so Alice is reinstated). But a perfectly natural reading of the rule is 'any accusation of lying defeats the witness', which would give 'no'—and the paper provides no human baseline to show which reading naive readers adopt. If the NL is ambiguous, then the models' low accuracy on even-length chains could reflect that ambiguity rather than faulty reasoning. That's not fatal: the formal benchmark is still valid as a formal task, and the models' inconsistent answers across semantically identical graphs still signal instability. But the claim of 'formally unambiguous interpretations' only holds at the graph level, not the prompt level. A quick crowdsourced baseline would settle it.\n\nTwo smaller things: o1-preview was tested on only 75 linear prompts and on a selected set of 'hard' non-linear prompts, so its failure rate isn't comparable to the other models—the paper says this, but the abstract's phrasing overstates it. And there's no repeated sampling, so within-model variance is unmeasured; the authors acknowledge this, but it confounds the 'ontological variation' effect with sampling noise.\n\nBottom line: worth engaging with. The benchmark design is reusable, the formal semantics are correctly computed, and the paper is transparent about its choices. I'd send it to review. The NL-validity question is what I'd ask the authors to address first.\n\nBest,\n[Your name]","headline":"Solid formal benchmark with a real NL-validity caveat; worth reviewing.","tokens_in":14341,"tokens_out":3119,"would_cite":true,"duration_ms":32580,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a dynamically generated, parameterized benchmark of witness-testimony puzzles built on formal argument attack graphs, and shows that current large language models fail on simple even-length attack chains, with no…","keywords":["large language models","reasoning evaluation","argumentation","abstract argument attack graphs","benchmarks","witness testimony","data contamination","reinstatement"],"falsifier":"Run the same witness prompts on a panel of human readers who are given only the rule 'Witnesses should be believed unless there is testimony that they are lying.' If a substantial fraction of readers answer 'yes' for an even-length chain such as Alice–Bob–Charlie–Dan, failing to apply reinstatement, then the natural-language rule does not convey the intended formal semantics, and the benchmark's ground truth is not unambiguously communicated.","tokens_in":13422,"feed_emoji":"⚖️","tokens_out":4699,"duration_ms":50770,"temperature":0.7,"pith_summary":"The paper proposes a way to build reasoning benchmarks that are generated fresh each time, so models cannot memorize the answers, and that scale in difficulty by design. The tasks are witness-testimony puzzles whose correct answer is fixed by a formal argument-attack graph, not by the surface wording. Using this benchmark, the paper claims to show that seven current language models reason brittly: they often fail at low complexity, especially on even-length attack chains, and produce inconsistent answers across variations of the same logical structure. A reasoning-specialized model solves all linear puzzles but still fails on selected hard non-linear ones.","feed_headline":"LLMs fail on simple witness-chain puzzles","feed_subtitle":"A parameterized benchmark of argument puzzles shows most models guess 'yes' when logic says 'no', even at low complexity.","key_machinery":"The central object is an abstract argument attack graph: a set of arguments with directed attack edges, evaluated under Dung-style argumentation semantics, which agree on these graphs. In a linear graph of $n$ arguments, the first argument is accepted exactly when $n$ is odd, because each new attack shifts the 'last word' and an attacked attack reinstates the earlier argument. In a non-linear graph, the main argument is accepted if and only if every attacking chain attached to it has even length. These graphs are converted into natural-language witness puzzles using an ontology of 474 names and 90 statements, so every formal graph yields many distinct prompts with the same ground truth.","core_discovery":"The paper's central claim is that a benchmark generated from parameterized abstract argument attack graphs, translated into natural language by an ontology, can evaluate whether generative language models genuinely reason rather than recall. On this benchmark, most tested models show a systematic failure: for linear chains with an even number of arguments, where the formal answer is 'no', they tend to accept the first witness and answer 'yes'. Performance drops as chains lengthen, non-linear graphs are harder than linear ones, and shuffling the order of statements helps some models and hurts others, indicating that order itself, not the logical structure alone, drives many answers.","pith_inferences":["Beyond the paper: the same attack-graph generator could be extended to support relations other than attack, such as support or undermining, and to richer legal fact patterns, making it a general template for argument-based reasoning evaluation.","Beyond the paper: because the formal graphs agree across argumentation semantics, the generated puzzles could be used as a training curriculum with checkable targets, teaching models to apply reinstatement rather than simply to answer 'yes' or 'no'.","Beyond the paper: the observed order-sensitivity suggests a direct testable extension: presenting the last witness first, as in the shuffled condition, should flip many model answers, and quantifying that flip would give a clean measure of positional bias.","Beyond the paper: the authors note the models' non-determinism without controlling for it; repeated sampling of the same prompt would separate random fluctuation from systematic reasoning failure."],"forward_implications":["If the benchmark measures what it claims, the same pipeline can produce fresh, non-reusable tests for any future model, directly addressing data contamination in static legal reasoning benchmarks.","The recurring failure on even-length linear chains points to a positional bias: models tend to believe the first-mentioned witness, so prompt order is itself a reasoning confound that the benchmark exposes.","Because the graphs have unambiguous formal interpretations, model errors can be traced to specific inference steps such as failing to reinstate an attacked argument, rather than to ambiguous wording.","Scaling the graph parameters upward should continue to separate models as they improve, allowing the benchmark to track reasoning gains over time."],"supporting_citations":[{"why":"Supplies the formal argumentation semantics that give the attack graphs their unambiguous ground truth.","marker":"[4]"},{"why":"The static legal benchmark the paper's dynamic approach is designed to improve on by reducing memorization and contamination.","marker":"[5]"},{"why":"Provides the high hallucination-rate figure that motivates the need for better evaluation of legal reasoning in language models.","marker":"[3]"},{"why":"Supports the premise that models can produce correct conclusions for wrong reasons, motivating reasoning-focused benchmarks.","marker":"[12]"},{"why":"Frames abstract argumentation as the formal background for the attack graphs used in the benchmark.","marker":"[1]"}],"fun_headline_variants":["LLMs answer 'yes' to logic that says 'no' in witness puzzles","Parameterized witness puzzles reveal brittle LLM reasoning","Even simple argument chains cause LLMs to flip the answer","LLMs fail at even-count witness chains in new reasoning benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single sentence 'Witnesses should be believed unless there is testimony that they are lying' makes readers and models infer the full formal rule, including that a liar's accusation does not count; the paper never tests this rule on human readers.","fun_headline_variants_meta":{"raw":{"variants":["LLMs answer 'yes' to logic that says 'no' in witness puzzles","Parameterized witness puzzles reveal brittle LLM reasoning","Even simple argument chains cause LLMs to flip the answer","LLMs fail at even-count witness chains in new reasoning benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2405,"prompt_tokens":897,"completion_tokens":1508,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1437}},"tokens_in":513,"tokens_out":1508,"duration_ms":11433,"temperature":1.0,"reasoning_tokens":1437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:15:52.822173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same witness prompts on a panel of human readers who are given only the rule 'Witnesses should be believed unless there is testimony that they are lying.' If a substantial fraction of readers answer 'yes' for an even-length chain such as Alice–Bob–Charlie–Dan, failing to apply reinstatement, then the natural-language rule does not convey the intended formal semantics, and the benchmark's ground truth is not unambiguously communicated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the formal argumentation semantics that give the attack graphs their unambiguous ground truth."},{"cited_title":"Guha et al","cited_arxiv_id":null,"evidence_quote":"The static legal benchmark the paper's dynamic approach is designed to improve on by reducing memorization and contamination."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the high hallucination-rate figure that motivates the need for better evaluation of legal reasoning in language models."},{"cited_title":"Steging, S","cited_arxiv_id":null,"evidence_quote":"Supports the premise that models can produce correct conclusions for wrong reasons, motivating reasoning-focused benchmarks."},{"cited_title":"Baroni, F","cited_arxiv_id":null,"evidence_quote":"Frames abstract argumentation as the formal background for the attack graphs used in the benchmark."}],"review_version":1}