{"id":"525b4a91-003f-4fa4-8753-9907918a68ec","arxiv_id":"2501.02235","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.","lead":"This paper surveys how modern AI models answer questions about documents that mix text, layout, and images, such as PDFs, slides, and scanned forms. It compares more than 40 approaches and identifies open challenges, especially handling multi-page documents.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 6's claim that layout-rich models have the best results is contradicted by its own Table 3 (GPT-4o 92.8, InternLMXComposer2 90.0 vs Arctic-TILT 90.2, ERNIE-Layout 88.4) and by the cross-paper confounds the authors disclaim in Section 7.","rationale":"The reader identified cross-paper ANLS comparability as the weakest assumption. That is correct and is part of the problem, but the more load-bearing issue is the inference drawn from those numbers: Section 6 uses Table 3 to claim that layout features are essential, ignoring both the explicit limitation in Section 7 and the presence of vision-only models at the top of the same table. Even if all scores were produced under identical conditions, the claim would still be confounded because models differ in base architecture, scale, pretraining data, and training protocol; no row in Table 3 isolates the effect of positional features. The paper's own limitation statement is therefore not just a caveat but a direct acknowledgment that the Section 6 conclusion is not warranted. The proper fix is to downgrade the conclusion to a conjecture or to add an explicit caveat that the comparison is observational. Since the reader's verdict is already CONDITIONAL and explicitly calls for weakening the ranking claims, my stress-test does not change the verdict; it sharpens the reason why the condition is needed. I do not see a separate, stronger internal error: the taxonomy of approaches, the tables of model characteristics, and the discussion of multi-page strategies appear informative and broadly consistent with the cited literature. The main risk is that a reader takes away the unsupported claim that layout features are essential, which could mislead architecture choices. Hence the concrete test focuses on making the evidence for that claim visible and checking whether any controlled comparison exists.","tokens_in":19295,"tokens_out":4965,"duration_ms":47183,"concrete_test":"Recompute the Section 6 ranking using only Table 3's DocVQA column, including all rows (T+L+V, vision-only, and commercial), and annotate each score with its source paper, base model, and evaluation condition. Then search for any pair of models that differ only in the presence of explicit layout features (same base LLM, same training corpus, same test protocol). If no such controlled pair exists, or if vision-only rows remain at the top, the Section 6 sentence should be revised to describe a correlation rather than a causal 'essential' conclusion. This is a desk-check requiring no new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim in Section 6 ('models that make extensive use of positional features—such as ERNIE-Layout and Arctic-TILT—have the best results... text and layout information are essential') rests entirely on Table 3's ANLS values. Two problems make this claim load-bearing and unverified. First, internal contradiction: Section 7 explicitly states that because 'these techniques are evaluated in their original experimental setups, which differ in terms of model architecture, training protocols, and datasets,' it is 'challenging to draw definitive conclusions,' yet Section 6 draws a definitive causal conclusion. Second, even taking Table 3 at face value, the ranking does not support the sentence: on the DocVQA column, GPT-4o scores 92.8 (a vision-only LVLM, with no explicit positional encoding as defined in Section 2) and InternLMXComposer2-4KHD scores 90.0, while Arctic-TILT scores 90.2 and ERNIE-Layout 88.4. Thus the two named layout-heavy models are not the best results unless one silently excludes commercial or vision-only rows. The claim that layout information is 'essential' requires a controlled comparison (same base LLM, same data, with and without layout features); observational cross-paper scores cannot support it. The survey's value as a structured map survives, but the headline architectural guidance is a hypothesis, not an established finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey of question answering over visually rich documents. It organizes recent work into three encoding families: structured multi-modal encoders that combine text, layout bounding boxes, and visual features (Section 2); vision-only large vision-language models that treat pages as images (Section 3); and multi-page strategies based on retrieval, per-page compression/query tokens, or sparse/recurrent attention (Section 4). Section 5 compares self-attention and cross-attention mechanisms for injecting visual features into an LLM decoder. The survey includes comparative tables of models and datasets (Tables 1-4) and concludes in Section 6 that models making extensive use of positional features, such as ERNIE-Layout and Arctic-TILT, achieve the best results and that text and layout are essential, while Section 7 acknowledges that cross-paper evaluation conditions differ.","tokens_in":19564,"tokens_out":6427,"duration_ms":55907,"significance":"The paper is a useful structured map of a fast-moving area, with broad coverage of encoder designs, multi-page strategies, and dataset characteristics. Its principal value is taxonomic: it collects a large set of recent models into a clear two-step pipeline and provides an appendix of VQA datasets. To the authors' credit, the limitations of the cross-paper comparison are explicitly acknowledged in Section 7. However, the central comparative conclusion in Section 6 is not established by the evidence in Table 3, and as written the conclusion is internally inconsistent with the caveat in Section 7. The significance of the paper as a guide for architecture choice therefore depends on a revision that either presents the conclusion as a hypothesis or supplies a controlled comparison.","major_comments":[{"comment":"Section 6 states that 'models that make extensive use of positional features—such as ERNIE-Layout and Arctic-TILT—have the best results,' but the DocVQA column of Table 3 shows GPT-4o at 92.8 and InternLMXComposer2-4KHD at 90.0, both above ERNIE-Layout's 88.4 and with InternLMXComposer2-4KHD essentially tied with Arctic-TILT's 90.2. GPT-4o is a vision-only commercial LVLM in the survey's own taxonomy, and InternLMXComposer2-4KHD is also a vision-only model; the claimed ranking is therefore contradicted by the table's own numbers.","section":"Section 6, Table 3"},{"comment":"The causal conclusion in Section 6 ('This indicates that text and layout information are essential') directly conflicts with the limitation stated in Section 7: because methods are 'evaluated in their original experimental setups, which differ in terms of model architecture, training protocols, and datasets,' the authors themselves say it is 'challenging to draw definitive conclusions.' Observational cross-paper ANLS values cannot establish that layout information is essential; that would require a controlled ablation in which the same base model, training data, and protocol are evaluated with and without layout features.","section":"Section 6 vs Section 7"},{"comment":"Table 3 mixes incomparable conditions: rows marked * use retrievers (e.g., InternLMXComposer2-4KHD with PDF-Wukong, Pix2Struct with Naidu et al., QwenVL and Idefics2 with M3DocRAG), rows marked ² concatenate page representations rather than performing true multi-page reasoning, and several cells are empty. Rankings based on such heterogeneous scores are not robust. The top-3 bold marking should be disclosed per column and restricted to comparable settings, or the table should be relabeled as a compilation of reported scores without ranking claims.","section":"Table 3"}],"minor_comments":[{"comment":"References Huang et al. 2024a and 2024b are identical ('From detection to application...'), and Xu et al. 2024a and 2024b are identical (LLaVA-UHD); please merge or disambiguate them.","section":"References"},{"comment":"Table 3 model names are inconsistent: 'mPLUGDoc', 'mPLUGDoc1.5', and 'ILMXC24KHD' should be spelled as in the main text (mPLUG-DocOwl, mPLUG-DocOwl1.5, InternLMXComposer2-4KHD).","section":"Table 3"},{"comment":"Table 4 header says '#Pages' per document but BoundingDocs reports 237k, which appears to be a total rather than a per-document average; please clarify the units in that column.","section":"Table 4"},{"comment":"The notation \\hat{V} = V ∪ [BBOX] should use a set of special tokens, e.g., V ∪ {[BBOX]}, and clarify how the marker is tokenized and inserted into the sequence.","section":"Section 2.1"},{"comment":"The bold top-3 markers are not visible in the text and the basis for choosing top-3 across a heterogeneous score matrix should be stated explicitly in the caption.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: this is a borderline case. The taxonomy and coverage are solid, but the headline conclusion in Section 6 is not supported by the paper's own Table 3 and contradicts Section 7. If the authors revise Section 6 to present the layout-essentiality claim as an open hypothesis and overhaul Table 3's presentation, the paper would be acceptable for publication as a survey; rejection is not needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short and candid: this survey is worth having on the shelf. The taxonomy — structured encoding (text+layout+vision) vs vision-only LVLMs, and then fusion via self- or cross-attention, plus the multi-page strategies section — is genuinely useful. The tables collecting models, resolutions, encoders, and datasets will save people time. And the Section 7 limitations are honest: they admit the cross-paper scores are not comparable.\n\nThe problem is that Section 6 draws a conclusion the paper's own evidence contradicts. The sentence 'models that make extensive use of positional features, such as ERNIE-Layout and Arctic-TILT, have the best results' is not what Table 3 shows. On DocVQA, GPT-4o scores 92.8, InternLMXComposer2-4KHD scores 90.0, Arctic-TILT 90.2, ERNIE-Layout 88.4. On DUDE, GPT-4V gets 75.1 vs Arctic-TILT's 57.0. Those are vision-only or commercial models with no explicit layout encoding as defined in Section 2. So the claim only works if you ignore the rows that break it. And Section 7 explicitly says it is 'challenging to draw definitive conclusions' because setups differ. That is an internal contradiction. The conclusion about layout being 'essential' is a hypothesis, not a finding, and the paper should say so.\n\nThere are also some minor table inconsistencies (duplicated citations, odd entries in the dataset table), but they don't undermine the survey's descriptive value.\n\nWho is this for? Newcomers and people who need a structured map of the VRD QA literature. It is not a scientific contribution in the sense of new methods or data, but it is a competent, opinionated synthesis. I would send it to peer review, but the authors need to rewrite Section 6 so the claims match the evidence, and ideally add a sentence right there acknowledging the comparability limits instead of leaving the caveat only in Section 7.","headline":"A genuinely useful survey map of VRD question answering, but the Section 6 ranking claim contradicts its own Table 3 and should be softened.","tokens_in":20031,"tokens_out":2329,"would_cite":false,"duration_ms":20947,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey of visually rich document question answering concludes that explicit 2D layout encoding, not image resolution, separates the top-scoring models.","keywords":["visually rich document understanding","document visual question answering","layout-aware language models","large vision-language models","multi-page document understanding","positional encoding","ANLS evaluation","retrieval-augmented generation"],"falsifier":"Run a matched experiment: take one backbone, train a vision-only variant that sees page images at high resolution and a layout-aware variant that consumes text tokens with 2D positional biases, on the same VQA datasets, and compare ANLS on DocVQA, DUDE, and MMLongBench-Doc; if the vision-only variant matches or beats the layout-aware one on multi-page questions, the survey's central claim that text and layout are essential would be falsified.","tokens_in":19098,"feed_emoji":"📄","tokens_out":11782,"duration_ms":100983,"temperature":0.7,"pith_summary":"This survey maps the design space of question answering over visually rich documents—scanned or born-digital pages whose meaning lives in text, tables, layout, and figures—and draws a comparative conclusion from the benchmark numbers it collects. The authors argue that models that explicitly encode 2D layout, such as ERNIE-Layout and Arctic-TILT, achieve the strongest results, and that this shows text and layout information are essential for document question answering, even when the question concerns a chart or figure. They also contend that vision-only large vision-language models, which treat the page as a single image, are ill-suited to multi-page documents unless paired with a retriever or heavy page compression, and they recommend multimodal fusion guided by cross-attention as the way forward. The survey matters because it gives practitioners a structured comparison of encoding strategies, fusion mechanisms, and multi-page techniques, while explicitly cautioning that its cross-paper score comparisons are not controlled experiments.","feed_headline":"Layout-aware models top document-QA scores","feed_subtitle":"Text and 2D position beat high-resolution vision on multi-page document questions, a survey of current systems concludes.","key_machinery":"The object that carries the argument is the document representation itself, decomposed into three modalities: text tokens, bounding boxes (layout), and the page image. The survey's comparison is organised around how models fuse these modalities: absolute 2D positional embeddings, relative 2D attention biases, disentangled attention that separates a token's semantic meaning from its horizontal and vertical distance to other tokens, cross-attention between visual and textual tokens, and page-level compression tokens. The evaluation machinery is ANLS (Average Normalized Levenshtein Similarity), the metric used in Table 3 to rank single- and multi-page VQA systems. The taxonomy's pivot is the contrast between structured encoders that consume layout explicitly and vision-only LVLMs that see the whole page as an image; the authors argue that this distinction, not resolution, separates the top performers.","core_discovery":"The central claim is that, in current document VQA, how a model represents the spatial layout of a page matters more than how many pixels it can see. The authors read Table 3 as showing that models making extensive use of positional features—ERNIE-Layout's disentangled attention over sequential, horizontal, and vertical relative distances, and Arctic-TILT's blockwise attention with a role bias for text tokens—have the best results, and they infer from this that text and layout information are essential for answering questions, even in complex charts and figures. They argue that structured multimodal approaches combining text, layout, and vision are more efficient for multi-page understanding than vision-only LVLMs, which either compress each page so heavily that performance degrades or must depend on a retriever to select relevant pages. They therefore recommend that the community prioritise layout handling and explicit 2D position encoding, and that visual features be injected through cross-attention with text tokens as queries rather than through self-attention over concatenated visual and textual tokens.","pith_inferences":["If the correlation between positional-feature use and top scores is causal, a matched ablation should show a larger layout advantage on table-heavy and multi-page benchmarks than on plain-text documents; the survey's cross-paper table cannot demonstrate this directly.","The paper's evidence implies that for text-dense pages the visual modality is largely redundant; a concrete extension would route text-heavy pages through layout-aware text encoders and reserve vision encoders for figures, cutting compute with little accuracy loss.","The recommended text-guided cross-attention fusion could be made adaptive by switching the query modality (text vs. vision) based on detected document type, generalising the paper's binary suggestion into a testable mechanism.","Because the survey restricts itself to transformers, its conclusion that layout encoding is essential is scoped; graph-based layout models would be the natural comparison to see whether the finding survives outside attention architectures."],"forward_implications":["Architecture choices should favour explicit 2D position encoding, whether absolute embeddings, relative biases, or 2D rotary positions, over treating document pages as generic images.","For multi-page QA, sparse-attention designs such as global-local or blockwise attention are the paper's recommended direction, ahead of page-by-page compression and retrieval-dependent pipelines.","Pretraining on document parsing tasks that turn page screenshots into structured text (HTML, Markdown, CSV/JSON) should be a standard step, since it aligns text, layout, and vision and makes visual features partially redundant.","Cross-attention visual injection with text tokens as queries is preferred over self-attention over concatenated token lists, because it keeps visual features separate while letting the LLM interrogate them.","Because the paper's score table is not a controlled comparison, the rankings should be re-validated with uniform training protocols before architectural conclusions are treated as settled."],"supporting_citations":[{"why":"This paper supplies ERNIE-Layout, one of the two top-scoring layout-aware models; its disentangled attention over horizontal and vertical relative distances is the survey's main evidence that 2D layout information drives performance.","marker":"Peng et al., 2022"},{"why":"This paper supplies Arctic-TILT, the other top scorer; its blockwise attention and role bias for text tokens anchor the conclusion that layout-aware multimodal models handle multi-page documents efficiently.","marker":"Borchmann et al., 2024"},{"why":"This paper supplies GRAM, the global-local attention model whose strong MPDocVQA score supports the survey's claim that sparse attention is the promising direction for multi-page QA.","marker":"Blau et al., 2024"},{"why":"This paper supplies MMLongBench-Doc, the long-context multi-page benchmark with cross-page questions that the survey uses to compare multi-page strategies.","marker":"Ma et al., 2024c"},{"why":"This paper supplies DUDE, a multi-page benchmark used in Table 3 where layout-aware models score highest.","marker":"Landeghem et al., 2023"},{"why":"This paper supplies DocVQA, the single-page benchmark at the centre of the survey's model comparison.","marker":"Mathew et al., 2021b"},{"why":"This paper shows that adding visual features brings little to no improvement on datasets without visual components, supporting the survey's claim that visual features are partially redundant with text.","marker":"Tang et al., 2023"},{"why":"This paper provides mPLUG-DocOwl2, the vision-only multi-page model whose page compression degrades performance, supporting the survey's claim that vision-only approaches are ill-suited to multi-page documents.","marker":"Hu et al., 2024b"},{"why":"This paper provides Pix2Struct and the screenshot-parsing pretraining strategy, which the survey identifies as the effective way to align visual and textual features before injection into an LLM.","marker":"Lee et al., 2023"}],"fun_headline_variants":["Layout beats pixels in document QA, survey finds","Spatial position drives document question answering","Text layout trumps vision in document VQA","For document QA, how text is placed matters most","Survey: Layout encoding, not pixels, tops document QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ANLS scores in Table 3 can be compared across models even though each model was trained and evaluated by a different team under different protocols; the authors themselves flag this in Section 7, noting that it is challenging to draw definitive conclusions.","fun_headline_variants_meta":{"raw":{"variants":["Layout beats pixels in document QA, survey finds","Spatial position drives document question answering","Text layout trumps vision in document VQA","For document QA, how text is placed matters most","Survey: Layout encoding, not pixels, tops document QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000443,"raw_usage":{"total_tokens":2170,"prompt_tokens":798,"completion_tokens":1372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":1299}},"tokens_in":414,"tokens_out":1372,"duration_ms":10361,"temperature":1.0,"reasoning_tokens":1299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:14.704754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched experiment: take one backbone, train a vision-only variant that sees page images at high resolution and a layout-aware variant that consumes text tokens with 2D positional biases, on the same VQA datasets, and compare ANLS on DocVQA, DUDE, and MMLongBench-Doc; if the vision-only variant matches or beats the layout-aware one on multi-page questions, the survey's central claim that text and layout are essential would be falsified.","supporting_citations":[],"review_version":1}