{"id":"d8bb74b0-4ed5-4def-ae21-a09a1692bdba","arxiv_id":"2607.24651","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Attribution hallucination in document VLMs is largely a coordinate-interface artifact; quote-and-retrieve plus region-label-free GRPO roughly doubles evidence recall and cuts hallucination.","lead":"Open vision-language models fail at citing document evidence with bounding boxes even when answers are correct. Quoting evidence in text and retrieving layout regions recovers most of that attribution, and label-free RL improves it further.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The empirical recall gains are solid, but the \"interface artifact, not capability gap\" diagnosis leans on one interpretive move: retrieval from the question alone already triples coordinate recall (22.7–29.1 vs ≤8.1, Table 2), and question+generated-answer retrieval matches or beats quotes on 3 of 6","rationale":"The reader's verdict (CONDITIONAL, high confidence) already conditions acceptance on the credit-assignment wording, and my stress test lands on the same spot, so I agree rather than escalate. Reasons not to move the verdict: (1) The empirical claims are unusually well armored for a systems paper — the recall comparison is judge-free and geometric, replicates under an independent parser with a near-identical ceiling (App. E), uses paired bootstrap/McNemar with intersection-AH controls (App. D), and the coordinate condition uses CiteVQA's prompt verbatim with numbers in the published regime. (2) The verification filter inspects only ground-truth annotations, so it cannot favor either interface, and the dropped-document analysis (App. B) argues against a parser-friendly skew. (3) The GRPO caveats the reader listed (single seed/backbone, precision 37.6→26.5, F1 down, gain purchased via 4.6 vs 2.1 citations) are disclosed in the paper itself (Sec 5.4, App. H) and partially offset by the judge-based strict-SAA improvement (19.1→27.4, p=6e-7). (4) The paper itself surfaces the q+a control and the complementarity analysis; my concern is that the interpretation drawn from them is under-determined, not that data was hidden. The proposed q+gold-answer control is the natural completion of the paper's own ablation matrix: it decouples \"answer content\" from \"model-generated language\" using machinery the authors already have. If it comes out favoring the quotes channel, the paper's framing stands essentially as-is; if not, the fix is reframing Sec 6.1/6.2 rather than retracting any number. That is exactly the CONDITIONAL shape the reader assigned: accept the systems contribution, revise the causal wording, ship code as promised in App. A.","tokens_in":27104,"tokens_out":2866,"duration_ms":117214,"concrete_test":"Add one row to Table 2: retrieval using the question plus the *gold* answer as the query (same encoder, same whole-document candidate set, same per-question citation budget, one-to-one assignment), scored at IoU 0.5 on all 719 questions, and optionally a second row using an answer generated by a *different* backbone. Compare against the quotes row and the quotes∪q+a union (51.9 pooled). If q+gold-answer retrieval reaches ≈ the quotes∪q+a union, then the model's quotes contribute no location signal beyond answer content restated in text, and the \"models already describe locations in language\" claim (Sec 6.2) should be dropped in favor of a retriever-centric account. If q+gold-answer falls clearly short of the quotes row (as question-only does), the interface diagnosis is substantially vindicated. The check reuses the existing pipeline and costs under an hour of embedding/inference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest_claim's empirical core — quotes+retrieval recovering 25.9–46.9 recall where coordinates get ≤8.1, with stable answer quality, and the GRPO gain — is well supported: judge-free geometric scoring, paired significance tests, a cross-parser replication (Docling, App. E), honest budget-matched controls, and disclosed precision/F1 regressions after RL. I find no internal inconsistency there. The load-bearing soft spot is the same one the reader flagged, and it deserves sharpening: Section 5.3's own controls show that the retrieval machinery with *no model evidence expression at all* (question-only query, budget matched) already reaches 22.7–29.1 recall — roughly triple the coordinate ceiling — and question+answer retrieval reaches 32.5–42.8, exceeding the quotes row for Qwen3.5-9B, Gemma-3-12B, and the GRPO model. The paper's defense of the interface diagnosis therefore rests on two interpretive planks rather than on quotes dominating external baselines: (a) reading q+a as \"a second reading of the same model output\" (Sec 6.2), and (b) the 11.1-point oracle-union complementarity over a 2.4-point query-diversity control (App. F, Table 11). Both are reasonable, but neither excludes the alternative decomposition: a strong multimodal retriever over parser blocks does most of the localization work, and *any* fluent text that restates the answer's content — model-generated or not — gives it enough to find the block. If that reading is right, the correct headline is \"coordinate emission is broken in open VLMs and a retriever fixes attribution,\" not \"models already carry location information in their language.\" Notably, the missing discriminating experiment is symmetric to the ones the paper ran: the q+a control uses the model's *own* answer, which entangles \"model language\" with \"answer content.\" A retrieval query built from the gold answer (or another model's answer) would separate the two. This does not threaten the recall numbers, the practical pipeline, or the格拉","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript studies evidence attribution in visual document QA and argues that the widely reported failure of open VLMs to cite correct regions (Attribution Hallucination) is substantially an artifact of the coordinate output interface rather than a missing capability. On a verified 719-question bilingual subset of CiteVQA, the authors hold backbone, inputs, and scoring fixed and contrast the standard coordinate interface with a language interface in which the model quotes evidence verbatim and a multimodal retriever over layout-parser blocks resolves each quote to a region. Across six open VLMs, necessary-evidence recall rises from ≤8.1 to 25.9–46.9 and AH roughly halves, with answer quality largely unchanged. The same quote-and-retrieve pipeline is then used as a GRPO training scaffold whose reward (a VLM judge reading the gold answer and retrieved crops, gated on answer correctness) requires no region labels, raising an 8B backbone's strict attributed accuracy from 22.4 to 33.8. Ablations include coordinate-to-block snapping, question-only and question+answer retrieval controls, lexical/BM25 resolvers, a Docling cross-parser check, and an element-level residual error decomposition.","tokens_in":27599,"tokens_out":3714,"duration_ms":149685,"significance":"If the results hold, this is a useful and well-executed contribution to visual document attribution. The strengths are concrete: a controlled same-backbone/same-input/same-scoring comparison across six models and four families; judge-free geometric scoring (IoU-based recall) independent of the training reward; paired bootstrap and McNemar significance tests; a cross-parser replication (Docling, App. E) that rules out shared segmentation conventions with the annotation tooling; budget-matched retrieval controls; an honest residual decomposition (App. H, Fig. 7) showing where training gains come from; and a promised full release of the verified set, filters, and code. The region-label-free GRPO reward (Eq. 2) is a genuinely practical recipe — it uses only QA-pair data any benchmark already has — and the authors disclose its precision/F1 trade-off rather than hiding it. The interface diagnosis, even in its weaker form (\"coordinates lose page-correct evidence at element granularity; quotes resolved by retrieval recover it\"), reframes Attribution Hallucination in a way the growing citation-VQA literature will need to engage with.","major_comments":[{"comment":"The paper's interpretive headline — that the recovery \"rests on location information that the models express in generated language\" (§1, §5.3) — is supported only under a specific reading of the retrieval controls. The question-only control already reaches 22.7–29.1 recall (vs. ≤8.1 for coordinates), so the parser+retriever machinery alone triples the coordinate ceiling with no model evidence expression at all; and the q+a control exceeds the quotes row for Qwen3.5-9B (32.5 vs 25.2), Gemma-3-12B (41.8 vs 29.9), and the GRPO model (52.0 vs 51.2). The authors' defenses — reading q+a as \"a second reading of the same model output\" (§6.2) and the 11.1 vs 2.4 oracle-union complementarity (App. F, Table 11) — are reasonable but do not exclude the alternative that any fluent text restating the answer's content gives the retriever enough to find the block. Two concrete requests: (1) add a budget-","section":"§5.3, Table 2; §6.1–6.2; App. F, Table 11"},{"comment":"The headline training claim (SAA 22.4→33.8) coexists with a disclosed precision drop (37.6→26.5), a flat-to-negative box F1 (32.2→28.8), and a doubling of citations per response (2.1→4.6). The authors address reward-hacking via the strict SAA variant (Ans≥4∧Rel≥4, 19.1→27.4, McNemar p=6e-7) and the App. H budget-truncation analysis, which is good practice. However, the abstract and §5.4 present SAA as the headline number while the F1 regression appears only in §5.4/App. H; since SAA contains a recall disjunct (Rec≥0.6) that rewards exactly the broader-citation behavior that degrades F1, the claim \"raising strict attributed accuracy from 22.4 to 33.8\" overstates the practical improvement. Please report F1 (and citations/response) alongside SAA in Table 1's discussion and qualify the abstract accordingly. Relatedly, the GRPO result is a single seed on a single backbone (acknowledged in Lim","section":"§5.4, Table 1 (last row); §6.3; App. H"}],"minor_comments":[{"comment":"Stray typographical artifact: 'attribution\"without' — also check line-wrap artifacts in the extracted prompts (App. J).","section":"Abstract"},{"comment":"\"Answer scores are stable across interfaces for five of six backbones\" — but the exception, Qwen3-VL-8B (−6.2 Ans points), is exactly the backbone used for GRPO training. The abstract's \"little change in answer quality\" should be qualified, since the trained model's Ans (60.4) sits on a base that already lost 6.2 points under the language interface.","section":"§5.1"},{"comment":"Qwen3.5-9B cites in only 49.2% of responses under the language interface (Table 4, fmt). Please state explicitly how non-citing responses enter recall (presumably zero) in §4, and note that Table 1's judged-subset n (679 for this model) partially reflects this compliance failure rather than judge parse failures alone.","section":"§6.2, Table 4"},{"comment":"Qwen3.5-27B and Gemma-4-31B agree to one decimal at every IoU threshold in both coordinate and language rows. The footnote states they differ at the second decimal, but given how unusual this coincidence is, a sentence confirming the two runs' outputs were verified as distinct would preempt reader suspicion of a pipeline mix-up.","section":"Table 4"},{"comment":"The language-interface prompt (J.2) requests a 0-based page index per quote, but §3.2 states page predictions are deliberately ignored. Either remove the field from the contract or add one sentence explaining why it is solicited (e.g., kept for future use / ablation); as written, the pipeline discards information it asks the model to produce.","section":"§3.2, App. J.2"},{"comment":"Steps 1–44 of training metrics were lost to a cluster requeue (Fig. 5 caption). Consider noting in §A whether early-training behavior (e.g., initial quote-count drift) is observable from the validation curve or logs; the current figure starts mid-training, which slightly weakens the coverage-driven-gain narrative of §6.3.","section":"Fig. 5"},{"comment":"The inherited precision metric (App. H) is nonstandard — numerator counts hit elements, denominator counts cited boxes, and it can exceed 1. This is disclosed in App. H but §4's one-line description (\"follows the loop semantics\") is easy to miss on first reading; add a forward pointer or one clarifying clause in §4.","section":"§4, App. H"},{"comment":"Figure 1 is effective but the coordinate box example shows a case where IoU=0 despite a correct page; a sentence noting this is representative (per Fig. 3's page-vs-IoU decay) rather than cherry-picked would help.","section":"Fig. 1"}],"recommendation":"minor_revision","confidential_remarks":"The verified evaluation set, filters, and code are promised for release but not yet available at review time; given that the paper's credibility rests on the reproduction of the CiteVQA protocol and the verified-subset construction, I would ask the AE to confirm the release at revision. The evaluation judge (Gemini-3.5-Flash) is itself unreleased/proprietary, so the judged metrics (Rel, Ans, SAA, AH) are not exactly reproducible by third parties even with the authors' code; the geometric recall results are, and they carry the paper. Citation pattern is unremarkable; overlap with CiteVQA is appropriately acknowledged and the work is clearly positioned as a diagnosis of that benchmark's central finding rather than a re-derivation of it."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is simple: on a verified CiteVQA single-doc split, open VLMs that barely hit single-digit box recall under coordinates jump to ~26–47% when they only quote text and a parser+multimodal retriever places the quote. Answer quality barely moves, AH roughly halves, and the flat IoU curves plus Docling replication make the geometric part hard to dismiss. The GRPO add-on without region labels then lifts an 8B from 22.4 to 33.8 SAA. That package is useful for anyone stuck training box heads on long PDFs.\n\nWhat is actually new is not quote-then-cite (text RAG already did that) but the controlled head-to-head on visual documents with an explicit AH rate, the credit-assignment suite (snap, q, q+a, lex/BM25), and a reward that only needs gold answers plus retrieved crops. Design is careful: same backbones/inputs/scoring, six models, paired tests, separate train/eval judges, no train–eval PDF overlap, residual decomposition showing training mostly fixes “never quoted.” They also ship the reproducibility plan.\n\nSoft spot, in proportion: Section 5.3’s own numbers. Question-only retrieval already triples coordinate recall; q+a often matches or beats quotes. Their defense—that q+a is a second reading of model language, plus 11.1 pts oracle-union complementarity—is reasonable, but it is interpretive. A gold-answer (or other-model-answer) retrieval control would separate “model-carried location” from “any fluent restatement + strong retriever.” Without it, the safer headline is “coordinates are broken for open models; quote+retrieve (or even answer-aware retrieve) fixes a lot,” not a pure capability-in-language story. Minor: GRPO is one seed/one backbone, precision/F1 drop with more citations, figures stay hard, filter shrinks the bench, judges everywhere.\n\nFor DocVQA, grounded RAG, and eval design people this is worth reading. Math is ordinary RL; data and citations look solid. I would send it to referees—ask them to tighten the credit-assignment wording and ideally add the gold-answer retrieval control—not desk-reject it. Engage; cite the interface comparison if you work on attribution.","headline":"Clean interface diagnosis with real recall gains; the “models already know locations in language” claim is a bit stronger than the ablations strictly force.","tokens_in":28701,"tokens_out":579,"would_cite":true,"duration_ms":13505,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Open vision-language models already locate document evidence in language; forcing them to emit coordinates is what collapses attribution.","keywords":["visual document understanding","evidence attribution","attribution hallucination","vision-language models","quote-based citation","multimodal retrieval","region-label-free RL","GRPO"],"falsifier":"Re-run the same six backbones on the verified set with ablations that strip model-generated language from the query (question-only retrieval at matched citation budget, and coordinate boxes snapped to parser blocks): if those controls match or beat quote resolution on recall and Attribution Hallucination, the interface-artifact claim fails.","tokens_in":28241,"feed_emoji":"📄","tokens_out":1017,"duration_ms":23975,"temperature":0.7,"pith_summary":"Reliable answers over long visual documents need evidence attribution: the model must point to the page regions that support each claim. Current benchmarks force that step through bounding-box coordinates, and open models then show near-zero evidence recall and extreme Attribution Hallucination even when the answer itself is right. This paper holds models, pages, questions, and scoring fixed and changes only the evidence interface: the model quotes supporting text verbatim, and a layout parser plus multimodal retriever turns each quote into a page region. Across six open models, evidence recall jumps from single digits to the mid-twenties through mid-forties and the hallucination rate roughly halves, with little change in answer quality. The same quote-and-retrieve pipeline then becomes a training scaffold whose reward needs only gold answers and retrieved crops, so an 8B model can be improved without any region-level labels. The practical claim is that better attribution is available without a coordinate interface and without expensive box supervision.","feed_headline":"Quotes recover document evidence that coordinates miss","feed_subtitle":"Same models, same pages: verbatim quotes plus retrieval double-digit recall without box labels","key_machinery":"The language evidence interface: the model outputs only text (answer plus verbatim quotes of sentences, cells, captions, or notes); a layout parser proposes semantic blocks; a multimodal encoder embeds quote text and block crops; Hungarian assignment maps each quote to one block whose page and box become the citation. The same construction supplies a region-label-free GRPO reward gated on answer correctness times judged evidence relevance and coverage of the retrieved crops.","core_discovery":"On a verified bilingual single-document CiteVQA subset, the same open vision-language models that score at most about 8 percent necessary-evidence recall under coordinate emission reach 26 to 47 percent recall when they instead emit verbatim quotes resolved by multimodal retrieval over parser blocks, and Attribution Hallucination roughly halves, while answer quality stays largely stable. Using that pipeline as a GRPO training scaffold with a judge reward on gold answers and retrieved crops—no region labels—raises an 8B backbone’s strict attributed accuracy from 22.4 to 33.8.","pith_inferences":["If q+a retrieval already nearly matches quotes, hybrid systems that emit a short answer plus a few quotes may be more robust than pure quote or pure coordinate policies.","The same judge-on-crops reward could supervise models that name parser block IDs or emit coarse page-plus-block indices, bridging toward end-to-end region output without human boxes.","Caption-only figure handling implies a remaining gap for purely graphical evidence; pairing quotes with a light visual pointer for uncaptioned figures is a natural next test.","Closed models that already score well with coordinates may still benefit from language evidence at training time if region labels remain scarce for long documents."],"forward_implications":["Benchmarks and systems that score evidence only through coordinate tokens will systematically understate open VLMs’ attribution ability on visual documents.","Quote-then-resolve becomes a deployable attribution path that reuses text-RAG citation practice across the page-image boundary without teaching coordinate emission.","Region-level evidence labels are not required to raise attributed accuracy: a judge over gold answers and retrieved crops is enough signal for GRPO-style training.","Parser-proposed blocks set a measurable ceiling; improving layout parsing or allowing block naming instead of free quotes can raise the attainable recall without changing the model.","Citation similarity after assignment is an abstention signal, so systems can trade coverage for precision instead of always returning a box."],"fun_headline_variants":["Quotes beat coordinates for document evidence attribution","Verbatim quotes lift evidence recall without bounding boxes","Language interface halves attribution hallucination vs coordinates","Quote-and-retrieve raises recall from 8 to 26-47 percent","GRPO on quotes lifts attributed accuracy without region labels"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The recovery is mainly location information the model already expresses in generated language, not credit that mostly belongs to the external parser and retriever once any strong query is available.","fun_headline_variants_meta":{"raw":{"variants":["Quotes beat coordinates for document evidence attribution","Verbatim quotes lift evidence recall without bounding boxes","Language interface halves attribution hallucination vs coordinates","Quote-and-retrieve raises recall from 8 to 26-47 percent","GRPO on quotes lifts attributed accuracy without region labels"]},"model":"grok-4.5","effort":"low","cost_usd":0.002671,"raw_usage":{"total_tokens":1103,"prompt_tokens":878,"num_sources_used":0,"completion_tokens":58,"cost_in_usd_ticks":26708000,"prompt_tokens_details":{"text_tokens":878,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":167,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":878,"tokens_out":58,"duration_ms":3770,"temperature":1.0,"reasoning_tokens":167,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T09:01:14.122180+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same six backbones on the verified set with ablations that strip model-generated language from the query (question-only retrieval at matched citation budget, and coordinate boxes snapped to parser blocks): if those controls match or beat quote resolution on recall and Attribution Hallucination, the interface-artifact claim fails.","supporting_citations":[],"review_version":1}