{"id":"b052d419-0255-49f8-a5a5-17fa8746e8e1","arxiv_id":"2502.05675","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An error taxonomy and GPT-4o auto-evaluator show that LLMs answering Civil Procedure MCQs have high accuracy but much lower soundness and correctness, with misinterpretation as the dominant step-level error.","lead":"This paper builds a taxonomy of step-by-step reasoning errors and an automated GPT-4o judge that scores how soundly five LLMs reason through 175 US Civil Procedure questions. It finds that models often choose the right answer through flawed reasoning, and feeding the error taxonomy back into prompts improves accuracy by only a few points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 27-point accuracy-to-correctness gap is measured by an auto-evaluator validated only on the annotation set used to build it; the human-annotated subset shows roughly a 12-point gap, so the quantitative claim is not yet pinned down.","rationale":"The paper's intended contribution is not just a taxonomy but a quantitative diagnosis: accuracy overstates reliable legal reasoning by ~27 points, and misinterpretation is the dominant failure. That claim has two pillars: the human annotation study and the auto-evaluator. The human pillar (Table 6, Tables 8/9) is honest but small: 120 chains, four models, 30 questions, no GPT-4o. It supports the qualitative direction—correctness trails accuracy on every model—but its average gap is ~12 points, not ~27. The full-dataset pillar rests on model-generated labels whose agreement with humans is only fair-to-good on the very same chains used to develop the taxonomy. The paper's own Appendix K demonstrates high sensitivity to evaluator backbone: Gemini-1.5-Flash labels 79% of error-free steps as erroneous. Thus the numerical headline is fragile. I do not see a separate internal inconsistency in the metrics; soundness and correctness are clearly defined, and the taxonomy is detailed. The empirical claim is plausible, and the released code and data are a plus. But a conditional verdict that asks for held-out validation of the evaluator is exactly right; the concern is that the 27-point number may be a measurement artifact rather than a finding.","tokens_in":32880,"tokens_out":5538,"duration_ms":56918,"concrete_test":"Sample 100 reasoning chains from the full 875, stratified 20 per LLM including GPT-4o. Have at least two legal-expert annotators apply the paper's 11-step annotation guideline (Appendix G) to these chains without access to auto-evaluator output; resolve disagreements and recompute soundness, correctness, and the accuracy-to-correctness gap from the human labels. Then compare model-wise gaps with Table 4. If the average gap is below roughly 15 percentage points, or if evaluator-human agreement on GPT-4o chains is below the reported 83–93% premise-level range, the 'average roughly 27 percent' headline should be revised or reframed as an upper-bound estimate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim — an average ~27% drop from accuracy to correctness (Section 5.1, Table 4) — depends entirely on GPT-4o auto-evaluator labels for 875 chains. The paper's validity evidence is limited: the auto-evaluator is checked against the same 120 manually annotated chains that generated the taxonomy and calibration examples, and no human labels are reported for GPT-4o-generated chains. Error-detection recall at premise level is 83.9–90.6%, but absence-of-error recall at the conclusion level is as low as 57% (Table 11, GPT-3.5). Since the correctness score requires both all premises error-free and a correct conclusion, evaluator false positives on correct conclusions directly depress C. Appendix K shows that swapping the evaluator backbone to Gemini-1.5-Flash changes recalled error-free steps from ~86% to ~20.6%, so evaluator choice is consequential. Independent human-annotated scores in Table 6 show the same direction but a much smaller average drop (roughly 12 points vs. 27). Unless the full-dataset auto-evaluator labels are checked on a held-out sample, the magnitude of the central claim is not established; only the qualitative direction is.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies step-by-step legal reasoning of LLMs on a Civil Procedure multiple-choice QA dataset (175 questions from the Glannon Guide). It proposes an error taxonomy with premise-level and conclusion-level categories, two metrics (soundness and correctness), and a GPT-4o-based auto-evaluator pipeline that labels errors in reasoning chains. Applying the pipeline to five LLMs, it reports that accuracy overstates the quality of legal reasoning: correctness scores are on average about 27 points lower than accuracy, with 'Misinterpretation' as the dominant premise-level error. It also reports that adding error-taxonomy definitions as feedback to several prompting strategies improves accuracy by up to about 4 percentage points. The paper includes human annotation of 120 reasoning chains and makes data and code publicly available.","tokens_in":33105,"tokens_out":5171,"duration_ms":53669,"significance":"If the findings are reliable, this is a useful contribution: it provides a reusable taxonomy and an automated evaluation framework for legal reasoning chains, and it makes a strong, falsifiable claim that standard accuracy numbers substantially overstate the soundness of LLM legal reasoning. The qualitative direction is independently supported by the human-annotated subset (Table 6), and the release of code and data is a concrete strength. The main unresolved issue is that the headline quantitative claim depends on an auto-evaluator whose agreement with humans is measured only on the same 120 chains used to develop the taxonomy and calibration examples, so the magnitude of the accuracy-correctness gap is not yet pinned down.","major_comments":[{"comment":"The headline result—an average drop of roughly 27 points from accuracy to correctness—is computed by the GPT-4o auto-evaluator on all 875 reasoning chains, but the evaluator is validated only against the same 120 manually annotated chains that were used to develop the taxonomy and to calibrate the prompt examples. The human-annotated subset in Table 6 shows the same direction but a much smaller average gap (about 12.4 points: 13.3, 16.7, 9.7, and 10.0 for Mistral, Llama, GPT-3.5, and GPT-4-turbo, respectively). Because no held-out human evaluation is reported for the remaining chains, or for GPT-4o-generated chains, the quantitative magnitude in Table 4 is not established. The authors should either provide a held-out human validation sample or explicitly re-frame the full-dataset numbers as provisional and place the primary claim on the human-evaluated subset.","section":"Section 5.1, Table 4 and Table 6"},{"comment":"The correctness score is a binary metric that requires both a perfect soundness score and a correct final answer. This makes it highly sensitive to false positives in the evaluator's conclusion-level labels: if the evaluator flags a correct, error-free conclusion as erroneous, the correctness score for that chain becomes zero. Table 11 shows that agreement on error-free conclusions (R(C)) is only 57% for GPT-3.5-turbo and 75% for Mistral and GPT-4-turbo. The paper reports only recall, not precision or a confusion matrix, so the reader cannot determine how many correct conclusions were falsely downgraded. Please report precision and full confusion statistics, or a second human-annotation round on a random sample of the full dataset.","section":"Section 4.2 and Appendix J, Table 11"},{"comment":"The auto-evaluator is not stable across LLM backbones. Replacing GPT-4o with Gemini-1.5-Flash changes recall for error-free premise steps from about 86% to about 20.6%, while recall for erroneous steps remains about 78.1%. This large swing indicates that the evaluator's operating point is highly dependent on the backbone model, and the paper does not provide a principled reason to trust GPT-4o's operating point on the full dataset. The authors should either demonstrate robustness of the main conclusions under evaluator-backbone variation or provide additional evidence, such as calibration against a held-out human-annotated sample, that GPT-4o's labels are reliable for the full dataset.","section":"Appendix K"},{"comment":"The claim that error-taxonomy feedback improves prompting performance is based on point estimates over 175 questions with no confidence intervals or significance tests. Improvements are at most about 4 percentage points, and several conditions (e.g., Gemini with Self-Correct or Self-Discovery) show decreases. Given the small dataset and the number of conditions compared, the reported 'marginal improvements' may reflect noise. Please report per-condition sample sizes, standard errors or confidence intervals, and appropriate multiple-comparison considerations, or soften the claim to an observation of a small and inconsistent effect.","section":"Section 5.3 and Table 5"}],"minor_comments":[{"comment":"Inter-annotator agreement (Cohen's kappa) is reported on only 10 reasoning chains from a single model. Please state whether the full human annotation set was double-annotated, and if so, report per-model kappa values.","section":"Appendix G"},{"comment":"The displayed formula for the correctness score has an unmatched parenthesis: 'C = (1, if (premises + conclusion) correct' is missing a closing parenthesis. Please fix the typo.","section":"Section 4.2"},{"comment":"The table headers say 'Statistics Agreement % (Recall)' but the columns contain raw counts followed by recall percentages. Please relabel the header, for example as 'Counts (Recall)' or 'Agreement statistics (counts; recall)'.","section":"Appendix J, Tables 10 and 11"},{"comment":"In the caption, 'WI denotes EC from Incomplete Premises' appears to be a typo: 'EC' should be 'Wrong Conclusion (WC)'. Please correct.","section":"Figure 6"},{"comment":"The dataset is described as 'college-level,' but the Civil Procedure questions come from a law-school-oriented guide. Please use more precise terminology, such as 'law-school-level' or 'US civil procedure bar-exam style questions.'","section":"Abstract and Section 1"},{"comment":"The validation of the auto-evaluator is limited to the four models used for the initial manual annotation; no human agreement is reported for GPT-4o-generated chains. Please state this limitation explicitly in the main text, not only in the appendix.","section":"Section 3.4 and Appendix J"}],"recommendation":"major_revision","confidential_remarks":"The qualitative finding—that accuracy overstates reasoning quality—is likely correct and is independently supported by the human-annotated subset. The main risk to the paper is quantitative: the headline 27-point gap relies on an auto-evaluator validated only on the same 120 chains used to build the taxonomy, and the evaluator's conclusion-level false-positive rate is not measured. I recommend asking the authors to add a held-out human evaluation (even a few hundred chains) or to substantially weaken the quantitative claims. The prompting-feedback result also needs statistical grounding. If these points are addressed, the paper could be a solid contribution to LLM evaluation for legal reasoning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The qualitative result is real: on the 120 human-annotated chains, accuracy beats correctness by roughly 10–17 points per model, so accuracy does overstate how often the reasoning is sound. The paper's headline number, an average ~27% drop from accuracy to correctness, is produced by a GPT-4o auto-evaluator validated only on the same 120 chains used to build the taxonomy. The human-annotated subset shows an average drop of roughly 12 points. Direction established; magnitude not.\n\nWhat is new: a legal-specific error taxonomy (misinterpretation, irrelevant premise, factual hallucination; plus conclusion-level categories), soundness and correctness metrics, and a multi-call LLM auto-evaluator aimed at hallucination detection. The annotation work is real: 120 chains, 537 steps, kappa 0.819, detailed guidelines released, and data/code are public. The paper is also honest about limits, reporting Gemini backbone sensitivity and an undetectable rare conclusion-level category.\n\nSoft spots, in order. First, the full-dataset Table 4 and the 27% claim rest entirely on the auto-evaluator. Validation is on the same 120 chains used to design the taxonomy and calibrate the prompts; no held-out human set, and no human labels for GPT-4o-generated chains. Conclusion-level absence-of-error recall goes as low as 57% (GPT-3.5), and correctness requires an error-free conclusion, so false positives there directly depress the score. Swapping the evaluator backbone to Gemini-1.5-Flash drops error-free-step recall from ~86% to ~20.6%. The exact size of the gap is therefore fragile. Second, the dataset is 175 MCQs from one civil procedure textbook; single domain and exam style. The paper acknowledges this. Third, the prompting-with-feedback results are marginal (up to ~4%, sometimes negative), and the paper says so. Fine as a minor section.\n\nBottom line: someone working on legal reasoning evaluation or reasoning-chain assessment should read this and can cite the taxonomy and the framework. It deserves a serious referee, but the referee should insist on held-out validation of the auto-evaluator or on a reframed quantitative claim. I would not let the current Table 4 stand as the headline without that.","headline":"Qualitative finding is solid, but the headline 27% accuracy-correctness gap rests on an auto-evaluator validated only on the chains used to build it; direction real, magnitude not.","tokens_in":33675,"tokens_out":3613,"would_cite":true,"duration_ms":35295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Correct legal answers often hide unsound LLM reasoning chains.","keywords":["legal reasoning","LLM evaluation","chain-of-thought","error taxonomy","soundness","correctness","misinterpretation","multiple-choice QA"],"falsifier":"Have legal experts annotate a fresh sample of, say, 200 reasoning chains from the same dataset without seeing the auto-evaluator's labels, and compare the accuracy-to-correctness gap measured by humans with the auto-evaluator's numbers; if the human-measured gap is near zero or the agreement falls well below 87%, the paper's central quantitative claim is undermined.","tokens_in":32668,"feed_emoji":"⚖️","tokens_out":6399,"duration_ms":53836,"temperature":0.7,"pith_summary":"The paper argues that standard accuracy numbers overstate how reliably large language models reason about law. On a Civil Procedure multiple-choice dataset, the authors generated step-by-step chains from five LLMs, built an error taxonomy from 120 human-annotated chains, and used a GPT-4o-based auto-evaluator to label errors in all 875 chains. They report a sharp drop, averaging about 27 percentage points, between accuracy (correct final option) and correctness (a fully error-free chain that also reaches the right answer), with misinterpretation the most common premise-level error. If true, the finding means a large share of correct legal answers are produced by unsound reasoning, and the paper's taxonomy and auto-evaluator offer a way to measure that gap.","feed_headline":"Legal accuracy overstates LLM reasoning by 27 points","feed_subtitle":"Five LLMs' correct legal answers rest on flawed reasoning chains by ~27 points.","key_machinery":"The load-bearing machinery is the error taxonomy paired with the LLM-based auto-evaluator and two scores. The taxonomy separates premise-level errors (Misinterpretation, Irrelevant Premise, Factual Hallucination) from conclusion-level errors (Wrong or Right Conclusion from False or Incomplete Premises, and Right Conclusion with Hallucinated Content). Soundness is the fraction of premises in a chain that are error-free; correctness is a binary score that is 1 only when every premise is sound and the final option matches the expert answer. The auto-evaluator runs four GPT-4o pipelines per premise—three single-call systems for general errors, misinterpretation, and relevance, plus a multi-call system that generates and answers verification questions to catch factual hallucinations—then a summarizer aggregates the labels and a conditional mapping assigns the conclusion-level error.","core_discovery":"On its own terms, the paper establishes that LLMs frequently reach the right legal answer through flawed reasoning chains. Across five models on 175 Civil Procedure questions, soundness scores (the fraction of error-free premises) are high, up to 78.4% for GPT-4o, while correctness scores (a chain that is both fully sound and ends on the correct option) are much lower, at most 44.5% for GPT-4o. The average gap between accuracy and correctness is roughly 27 percentage points, with the largest drop (31.4 points) for Llama-3-8B-Instruct. Misinterpretation of the legal context, question, or options is the dominant premise-level error, and 'Wrong Conclusion from False Premises' dominates at the conclusion level; for the larger proprietary models, 'Correct Conclusion from False Premises' is also common, suggesting pattern matching rather than genuine reasoning. The GPT-4o-based auto-evaluator reproduces human error labels with about 87% recall on the 120-chain validation set.","pith_inferences":["If the ~27-point gap generalizes, then accuracy should be treated as an upper bound on reasoning reliability in high-stakes domains; the missing 27 points are answers that are right for the wrong reasons.","The auto-evaluator's reliance on GPT-4o means the reported full-dataset numbers are provisional; a stronger or ensemble evaluator could shift the exact percentages even if the qualitative trend survives.","Retraining or fine-tuning models with misinterpretation labels as explicit supervision could be a direct test of whether the taxonomy can convert error detection into better reasoning."],"forward_implications":["Reporting only accuracy on legal MCQA will hide that many correct answers rest on unsound chains; reporting correctness alongside accuracy quantifies the hidden gap.","Error definitions added to common prompting techniques improve accuracy by at most about 4%, suggesting that prompting alone will not close the gap.","Larger proprietary models produce more sound chains than open-weight models, but even the best tested model, GPT-4o, has a correctness score below 45%.","The taxonomy and auto-evaluator can be transferred to other logic-intensive tasks where step-by-step rationales need auditing."],"supporting_citations":[{"why":"Provides the Civil Procedure MCQA dataset and expert answers used throughout.","marker":"Bongard et al. (2022)"},{"why":"Supplies the zero-shot chain-of-thought prompting setting used to generate reasoning chains.","marker":"Wei et al. (2022)"},{"why":"Inspires the LLM-as-evaluator approach for aligning automated evaluation with human judgment.","marker":"Liu et al. (2023)"},{"why":"Grounds the auto-evaluator design in meta-evaluation of LLM judges.","marker":"Chern et al. (2024)"},{"why":"Motivates the multi-call verification pipeline for detecting factual hallucinations.","marker":"Varshney et al. (2023)"},{"why":"Provides chain-of-verification, the basis for the hallucination verification questions.","marker":"Dhuliawala et al. (2023)"},{"why":"Supplies the feedback-learning prompting method tested with the error taxonomy.","marker":"Tyagi et al. (2024)"}],"fun_headline_variants":["Legal LLMs: right answers, wrong reasoning chains","LLMs flub legal reasoning yet still pick correct answers","27-point gap between legal accuracy and sound reasoning","LLMs game legal MCQs without genuine step-by-step logic","Legal reasoning flaws hidden behind LLMs' high accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The GPT-4o auto-evaluator's error labels are trustworthy on all 875 reasoning chains, even though it was validated only against the same 120 human-annotated chains used to build the taxonomy and has about 87% recall.","fun_headline_variants_meta":{"raw":{"variants":["Legal LLMs: right answers, wrong reasoning chains","LLMs flub legal reasoning yet still pick correct answers","27-point gap between legal accuracy and sound reasoning","LLMs game legal MCQs without genuine step-by-step logic","Legal reasoning flaws hidden behind LLMs' high accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1456,"prompt_tokens":967,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":583,"tokens_out":489,"duration_ms":5804,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:22:49.760947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have legal experts annotate a fresh sample of, say, 200 reasoning chains from the same dataset without seeing the auto-evaluator's labels, and compare the accuracy-to-correctness gap measured by humans with the auto-evaluator's numbers; if the human-measured gap is near zero or the agreement falls well below 87%, the paper's central quantitative claim is undermined.","supporting_citations":[],"review_version":1}