{"id":"27edaa2d-8f46-4d8b-a8a9-27aa0653ea4b","arxiv_id":"2505.12837","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a CUAD contract QA task, GPT-4.1's exact-match accuracy rises from about 48% to 79% with well-structured Markdown input and a structure-aware system prompt, while GPT-4o stays near 50%.","lead":"What if the way a contract is formatted changes how well an AI can answer questions about it? This paper tests two OpenAI models on contract questions and finds that one model improves from about 48% to 79% accuracy when the contract is well structured and the prompt tells the model to expect structure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline effect may be an artifact of the citation-scoring rule: RegexClean removes line breaks, yet ground-truth citations come from the original text, so exact verbatim matching is ill-defined unless whitespace is normalized; the paper's metric definition is also internally inconsistent.","rationale":"The paper's central direction is plausible and valuable: it empirically probes whether input formatting and system-prompt placement change LLM performance on a legal QA task. The reported gaps are large, and the general phenomenon of format sensitivity has independent support in the literature. However, the exact-match metric is the instrument through which every comparison is read, and it is underspecified in a way that directly interacts with the independent variable. Because the RegexClean condition is created by removing line breaks, and the ground-truth citations are spans in the original formatted text, an exact verbatim match without whitespace normalization would systematically penalize the very condition the paper uses to demonstrate the structure effect. The paper's own wording in Section 3.1.5 is ambiguous about whether the citation is even required for the reported number. The mention of LLM-as-a-judge and answer_correctness, with results deferred to a future report, adds uncertainty rather than resolving it. This is not an ad hominem or an appeal to novelty; it is a request for the minimal evidence needed to distinguish a genuine 20-point formatting effect from a scoring artifact. The authors' qualitative conclusion may well survive such a check, but without the evaluation script or normalization rule the current numbers are not independently verifiable. A secondary inconsistency, the Appendix D counts not matching the 928-question sample, reinforces the need for released artifacts. A conditional verdict is therefore appropriate: the paper should be accepted only after the scoring procedure is specified and at least one head-to-head comparison is re-run under a documented normalization rule.","tokens_in":15239,"tokens_out":5359,"duration_ms":49582,"concrete_test":"Request or reconstruct the evaluation script and apply it to one stored condition, e.g., GPT-4.1 S2 on GPT4oVision_MD, under three scoring variants: (a) exact match with all whitespace normalized by collapsing runs of spaces and newlines to a single space, (b) exact match after removing all whitespace, and (c) Yes/No-only correctness. Then score CUAD_RegexCleanTXT with variant (a). If the 79% figure changes by more than a few points, or if RegexClean accuracy rises from 48% toward 67%, the headline structure effect is at least partly an artifact of citation-string formatting. A complementary check is to publish per-question outputs for 100 sampled RegexClean and OriginalTXT items and manually verify whether RegexClean failures are semantically correct citations that differ only in line breaks.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1.5 defines exact-match accuracy as requiring both parts of the bipartite answer: the Yes/No decision and, for Yes answers, an exact verbatim citation matching the CUAD ground-truth clause. It then reports the metric as 'the percentage of test instances for which the model produced a fully correct Yes/No answer', which would omit the citation requirement. These two definitions can produce different numbers.\n\nThe more serious problem is that the input transformations change the surface text to which verbatim matching applies. CUAD_RegexCleanTXT is created by stripping consecutive line breaks from CUAD_OrigTXT, while the ground-truth citations are spans in the original CUAD text. If the evaluator compares model output strings to original ground-truth strings without whitespace normalization, every RegexClean answer that contains a line break in the cited span will be marked wrong even when the model found the correct clause. The same issue applies, less severely, to Azure OCR and vision-extracted text. The claimed ~20-point gap between RegexClean/OCR (48%) and well-structured inputs (67%) is exactly the comparison most vulnerable to this artifact; it may measure formatting fidelity of the citation string, not comprehension.\n\nThe problem is not resolved by the paper's note that an LLM-as-a-judge and answer_correctness were used for content-level evaluation, because those results are explicitly not included. No evaluation script, normalization procedure, or judge prompt is released. The Appendix D table also lists counts that do not obviously sum to the 928 questions described in Section 3.1.1, so the reader cannot reconstruct which questions were used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how input text structure and prompt-placement design affect GPT-4o and GPT-4.1 performance on a legal question-answering task built from an excerpt of the CUAD dataset. Five input modalities are compared (original structured plain text, regex-cleaned plain text, Azure OCR text, GPT-4o Vision plain text, and GPT-4o Vision Markdown) under two prompting strategies (user-centric task definition and system-centric structure-aware prompting). The central finding is that GPT-4.1 is highly sensitive to input structure, improving by roughly 20 percentage points when given well-structured inputs, and by another 10-13 percentage points under the optimized system prompt, with the best condition (GPT-4o Vision Markdown, Strategy S2) reaching 79% exact-match accuracy. GPT-4o is reported as robust to format variation but lower in absolute accuracy. The paper argues that input formatting and prompt design are first-order variables for legal LLM deployment.","tokens_in":1786,"tokens_out":1944,"duration_ms":44642,"significance":"If the main effects are real, the paper has substantial practical significance for legal technology: it would demonstrate that document preprocessing and system-prompt design can change contract-review accuracy by tens of percentage points, and it would add an important, domain-specific data point to the growing literature on format sensitivity in LLMs. The work is also useful in its explicit focus on a realistic document-processing pipeline (original text, regex cleaning, OCR, vision-based extraction) rather than only synthetic perturbations. The paper is transparent about several limitations and does not overclaim generalizability beyond the tested models and task. However, the quantitative claims are currently supported only by a single point-estimate comparison with no uncertainty quantification, and the evaluation metric is not fully operationalized, so the size and even the direction of the headline effects are not yet securely established.","major_comments":[{"comment":"The exact-match metric is defined inconsistently. The text first states that an answer is correct only if both the Yes/No component and, for Yes answers, the exact verbatim citation match, and that no partial credit is awarded. It then reports 'the percentage of test instances for which the model produced a fully correct Yes/No answer', which omits the citation requirement. Clarify which criterion produced the numbers in Table 1. In addition, the paper does not specify the string-comparison procedure: is whitespace normalized? Are line breaks collapsed in both the model output and the ground-truth span? This matters directly for the RegexClean condition, because those inputs have line breaks removed while the CUAD ground-truth citations are spans from the original text. Without a defined normalization rule, the ~20-point gap between RegexClean/OCR and well-structured inputs may confound citation-string fidelity with comprehension.","section":"§3.1.5, Table 1"},{"comment":"All results are reported as single point estimates with no confidence intervals, standard errors, or information about repeated runs. Because the headline comparisons (e.g., 48% vs. 67% for GPT-4.1 under S1; 66% vs. 79% for Markdown under S2) are the central evidence, the paper should report either multiple runs with different seeds/temperatures or bootstrap confidence intervals over the 928-question sample. Without this, the reader cannot assess whether the observed differences are statistically stable, especially given the strict, partly citation-based scoring rule.","section":"§3.2, Table 1"},{"comment":"The description of the test set is internally inconsistent. The text says that after filtering to four topics ('Competitive Restriction Exception', 'Non-Compete', 'Exclusivity', 'No-Solicit of Customers') 928 questions were used. Appendix D, however, lists 35 question types with counts whose total far exceeds 928, and the named topics do not obviously correspond to those question types. The paper should state precisely how the 928 questions were derived from the CUAD excerpt, whether each question is paired with a uniquely associated contract, and how the four selected topics map to the question templates in Appendix A.","section":"§3.1.1, Appendix D"},{"comment":"There is a generator-evaluator confound for the vision-extracted modalities. GPT-4o Vision produces both GPT4oVision_TXT and GPT4oVision_MD, and GPT-4o is then evaluated on those inputs. This creates an advantage for the GPT-4o condition and, more importantly, bundles input-format effects with the extraction source. To separate the format effect from the source effect, the paper should include a condition in which the same Markdown content is rendered as plain text (or the same text is re-extracted by an independent OCR engine) and evaluated under otherwise identical conditions.","section":"§3.1.3, §4.2"},{"comment":"The paper mentions that an LLM-as-a-judge and the answer_correctness metric were used for content-level evaluation but states the results are not included. To make the exact-match results independently verifiable, the evaluation script, the judge prompt, and the normalization logic should be released or, minimally, described in enough detail to replicate the comparison. As written, the only quantitative evidence is a black-box point estimate.","section":"§3.1.5, §4.3"}],"minor_comments":[{"comment":"Several typos and formatting artifacts appear, such as 'Out goal' in footnote 4, 'theContract Understanding' in the abstract, and the line 'while modern while...' in the introduction. The paper would benefit from a careful proofreading pass.","section":"General"},{"comment":"The caption labels the subfigures as '(a).txt', '(b).txt', and '(c).md', but the subfigures themselves are not referenced clearly in the text. The figure could also state which contract section is shown and whether the same section is displayed across all three panels.","section":"Figure 1"},{"comment":"The system prompt shown contains the literal string 'Markdown // Plain Text' and the footnote explains that the segment was varied. It would be clearer to show the two actual variants (one for Markdown, one for plain text) as separate prompt blocks.","section":"§3.1.4, Strategy 2"},{"comment":"Table 2 reports unique question counts but does not explain the relationship between these counts and the 928 questions used in the study, nor how the four selected topics map to the templates in Appendix A. Adding a reconciliation table would resolve the apparent inconsistency.","section":"Appendix D"},{"comment":"Reference [43] appears both as 'Wang et al. (2023) MAUD' and as 'Wei et al. (2023) Chain-of-thought'; the numbering is inconsistent and should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practically important and timely question, and the qualitative pattern (format and prompt sensitivity) is plausible. However, the current manuscript does not provide enough measurement detail to verify the magnitude of the headline effects. The most serious issue is the ambiguous and unoperationalized exact-match definition, which could change the reported numbers substantially. I recommend major revision, with the expectation that the authors re-run or re-analyze their data under a clear, reproducible evaluation protocol and provide uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper because it makes a big, concrete claim: for legal contract QA, GPT-4.1's exact-match accuracy swings from ~48% to ~79% depending on how the input is formatted and whether the task instructions live in the system prompt. That is a 30-plus-point swing, which would matter to anyone deploying LLMs on contracts. The underlying observation—that some models are much more sensitive to input structure than others—is plausible and worth taking seriously.\n\nWhat's new: the specific GPT-4.1 versus GPT-4o comparison on a CUAD excerpt, with five input modalities including a GPT-4o-vision-to-Markdown pipeline, is not in the cited prior work. He et al. and MDEVAL show formatting effects, but not for this model pair or legal QA. The paper also does something useful: it tests a realistic degradation path (OCR, aggressive cleaning) and reports that GPT-4o is flat across conditions while GPT-4.1 improves sharply with well-structured input. That is a concrete, replicable empirical claim.\n\nThe soft spots are real, and one is load-bearing. Section 3.1.5 first says an answer must include both the Yes/No decision and an exact verbatim citation, then defines exact-match accuracy as 'the percentage of test instances for which the model produced a fully correct Yes/No answer.' Those are not the same thing. If the headline numbers come from the looser definition, they are inflated. The bigger problem is the citation-matching procedure. CUAD_RegexCleanTXT strips line breaks, but ground-truth citations are spans from the original text. If the comparison is a raw string match without whitespace normalization, then any answer containing a line break in the cited span is marked wrong even when the model found the right clause. That would systematically depress the RegexClean and OCR conditions and could account for a chunk of the claimed 20-point gap. The paper neither describes the normalization nor releases the evaluation script, so this cannot be checked.\n\nSmaller issues: no confidence intervals or repeated runs; the Appendix D counts do not obviously sum to the 928 questions stated in Section 3.1.1; and the same model (GPT-4o Vision) generates the Markdown that GPT-4o is then evaluated on, a mild same-model confound. The authors mention LLM-as-a-judge results but exclude them, which is fine if disclosed, but the main metric has to stand alone.\n\nThe direction of the finding is believable, but the magnitude is not established. A serious referee should ask for the evaluation script, a precise definition of citation matching, whitespace-normalized comparison, and a reconciled sample size. That is a substantial revision, not a desk reject—the experiment is well-designed and the question is practically important.\n\nMy recommendation: send it to peer review, but with the clear expectation that the measurement is fixed before it goes forward.","headline":"The 30-point formatting effect on GPT-4.1 is plausible and worth testing, but the exact-match metric is under-specified to the point that the magnitude is not yet established.","tokens_in":16093,"tokens_out":4470,"would_cite":false,"duration_ms":43107,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that explicit input formatting and a structure-aware system prompt can lift GPT-4.1's exact-match accuracy on contract question answering from about 48% to 79%.","keywords":["legal document understanding","contract question answering","input formatting","Markdown structure","prompt engineering","exact-match accuracy","GPT-4.1","CUAD dataset"],"falsifier":"Pull the raw model outputs for the 928 S2 Vision-Markdown questions and apply a strict scorer that requires the cited span to be a character-for-character substring of the CUAD ground-truth clause; if the 79% figure falls toward the ~48% unstructured baseline, the headline result is a scoring artifact rather than a structure effect.","tokens_in":15038,"feed_emoji":"📄","tokens_out":6863,"duration_ms":66533,"temperature":0.7,"pith_summary":"The paper tries to establish that the exact formatting of a legal contract—whether it is cleanly structured, aggressively cleaned, or OCR-degraded—and where the task instructions sit in the prompt are first-order determinants of LLM performance on legal question answering, not cosmetic details. On a 928-question contract-review task derived from CUAD, GPT-4.1 scored around 48% exact-match on poorly structured text and 66–67% on well-structured text, then 77–79% when the system prompt explicitly announced the input structure. GPT-4o, by contrast, stayed nearly flat at 47–52% across all conditions. The authors conclude that document preprocessing and prompt architecture should be managed as core variables in high-stakes legal AI, and that vision-extracted Markdown is the best-performing input when the model is primed for it. This matters because practitioners commonly assume LLMs can infer structure from any text; the data say otherwise for at least one widely used model.","feed_headline":"Contract formatting swings GPT-4.1 accuracy by 20 points","feed_subtitle":"A legal QA study: structured input plus a structure-aware system prompt lifts GPT-4.1 to 79% exact match.","key_machinery":"The load-bearing object is a spectrum of five input formats that vary only in structural explicitness: original CUAD plain text, regex-cleaned text with line breaks removed, Azure OCR text, GPT-4o Vision plain text, and GPT-4o Vision Markdown. The second lever is prompt placement: strategy S1 keeps the system prompt generic and puts all task instructions in the user message, while strategy S2 moves the instructions to the system prompt and states the input type ('structured Markdown' or 'structured Plain Text'). The outcome measure that carries the argument is exact-match accuracy on a bipartite answer—a correct yes/no decision plus an exact, verbatim clause citation—which forces the model both to decide and to locate the evidence.","core_discovery":"The central empirical discovery is a large, model-specific sensitivity to input structure and prompt placement. With a generic system prompt, GPT-4.1's exact-match accuracy on the bipartite yes/no-plus-verbatim-citation task is about 48% when line breaks are removed or raw OCR is used, and rises to about 66–67% for well-structured plain text and for text extracted by GPT-4o Vision, including Markdown. Moving the full task instructions into the system prompt and adding an explicit 'you will receive structured Markdown / plain text' cue adds another 10–13 points, bringing GPT-4.1 to 77% on original CUAD text and 79% on GPT-4o Vision Markdown. GPT-4o is far less sensitive to formatting changes but also lower in absolute accuracy, improving only 2–5 points under the same prompt shift. Under the stronger prompt, Markdown outperforms well-structured plain text, whereas without the prompt the two formats are equivalent.","pith_inferences":["The results imply that legal NLP leaderboards may be inadvertently measuring input-format sensitivity rather than model reasoning; standardizing a canonical document format across models would make comparisons fairer.","A concrete extension would test whether the structure-awareness cue works by activating format-specific priors; varying the cue's specificity could isolate the mechanism.","The 20-point gap for GPT-4.1 suggests many retrieval-augmented pipelines that feed OCR text into modern models may be silently underperforming; re-running RAG evaluations with Markdown inputs is a cheap test.","The same input-format manipulation could be applied to open-weight models to see whether the sensitivity pattern tracks model family or training objective."],"forward_implications":["GPT-4.1-class models can lose roughly 20 points of exact-match accuracy when contracts arrive as cleaned text or raw OCR, so document preprocessing should be treated as a core performance lever.","A structure-aware system prompt ('you will receive structured Markdown') is worth 10–13 points on top of good formatting, so prompt architecture belongs in legal LLM evaluation.","Vision-model PDF-to-Markdown conversion can match or exceed human-formatted plain text, making it a practical path for scanned contracts.","Markdown beats well-structured plain text only when the system prompt primes the model; without priming the formats are equivalent.","Even at its best, the model reaches 79% exact-match, so legal workflows still require human verification rather than autonomous decision-making."],"supporting_citations":[{"why":"Supplies the contracts, question templates, and ground-truth clause annotations for the task.","marker":"[18]"},{"why":"The conversion tool used to generate the Markdown inputs that achieve the best results.","marker":"[4]"},{"why":"Supplies the question-answer pairs used to complement the CUAD excerpt with the needed paired answers.","marker":"[31]"},{"why":"The vendor prompting guidance that motivates the structure-aware system-prompt strategy S2.","marker":"[26]"},{"why":"Azure OCR creates the degraded plain-text condition that anchors the low end of the accuracy range.","marker":"[23]"},{"why":"Prior work showing prompt-formatting changes can swing LLM accuracy, which motivates the formatting hypotheses.","marker":"[15]"},{"why":"Defines the exact-match evaluation metric that the paper adopts for the bipartite answers.","marker":"[9]"}],"fun_headline_variants":["Structure and prompts lift GPT-4.1 legal QA by ~30 points","Markdown plus system prompt: 79% legal QA exact match","GPT-4.1 needs structure: 20-point swing, prompts add 10 more","Well-structured contracts add 20 points to GPT-4.1, prompts 10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains stand or fall on the scorer requiring both halves of each answer—a correct yes/no and an exact verbatim citation—in every condition; the paper's own definition of the metric is ambiguous about whether both halves were actually enforced.","fun_headline_variants_meta":{"raw":{"variants":["Structure and prompts lift GPT-4.1 legal QA by ~30 points","Markdown plus system prompt: 79% legal QA exact match","GPT-4.1 needs structure: 20-point swing, prompts add 10 more","Well-structured contracts add 20 points to GPT-4.1, prompts 10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001808,"raw_usage":{"total_tokens":7187,"prompt_tokens":1083,"completion_tokens":6104,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":6018}},"tokens_in":699,"tokens_out":6104,"duration_ms":42631,"temperature":1.0,"reasoning_tokens":6018,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:10.336788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pull the raw model outputs for the 928 S2 Vision-Markdown questions and apply a strict scorer that requires the cited span to be a character-for-character substring of the CUAD ground-truth clause; if the 79% figure falls toward the ~48% unstructured baseline, the headline result is a scoring artifact rather than a structure effect.","supporting_citations":[{"cited_title":"Vision-parse: Parse pdfs into markdown using vision llms","cited_arxiv_id":null,"evidence_quote":"The conversion tool used to generate the Markdown inputs that achieve the best results."},{"cited_title":"Cuad: An expert-annotated nlp dataset for legal contract review, November 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the contracts, question templates, and ground-truth clause annotations for the task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the question-answer pairs used to complement the CUAD excerpt with the needed paired answers."},{"cited_title":"Gpt-4.1 prompting guide | openai cookbook","cited_arxiv_id":null,"evidence_quote":"The vendor prompting guidance that motivates the structure-aware system-prompt strategy S2."},{"cited_title":"Azure ai vision with ocr and ai","cited_arxiv_id":null,"evidence_quote":"Azure OCR creates the degraded plain-text condition that anchors the low end of the accuracy range."},{"cited_title":"Wang, and Samira Hasan","cited_arxiv_id":null,"evidence_quote":"Prior work showing prompt-formatting changes can swing LLM accuracy, which motivates the formatting hypotheses."},{"cited_title":"Metrics to evaluate a question answering system","cited_arxiv_id":null,"evidence_quote":"Defines the exact-match evaluation metric that the paper adopts for the bipartite answers."}],"review_version":1}