{"id":"e2d0774f-fcf9-40d1-bd9d-353456e457ec","arxiv_id":"2412.08746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DocVLM compresses OCR text and layout into 64 learned queries that are injected into frozen VLMs, boosting DocVQA accuracy at low image-token budgets across three model families.","lead":"This paper introduces DocVLM, a method that adds a compressed OCR text and layout stream to frozen vision-language models, improving document question answering under low token budgets. The approach could make document understanding cheaper and enable multi-page document processing without retraining the base model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim is contingent on an undisclosed OCR engine; the reported gains may not survive with open-source OCR.","rationale":"I read the paper in good faith. The method is clearly described at the architecture level, the two-stage training is reasonable, and the ablations (Table 3, Figure 5, Table 8) provide internal evidence that the compressed OCR queries carry useful information beyond raw OCR words. The central claim is that integrating DocVLM improves document understanding under token constraints, and the equal-token comparisons for Qwen2-VL (320 vs 320) and LLaVA-OneVision (1.5K vs 1.5K) support this claim independent of token-count mismatches. I considered the token-budget mismatch in the headline InternVL2 number (256 baseline vs 320 DocVLM) but judged it secondary: the abstract frames the comparison at equal image resolution (448x448), and the token-allocation argument is separately supported by Figure 5. The truly load-bearing assumption is the quality and format of the OCR input. The paper leaves the OCR engine unidentified, and the OCR encoder is initialized from DocFormerV2, whose pretraining on IDL used a specific OCR representation. Without knowing the engine, the reader cannot tell whether the reported gains are a property of DocVLM or of an unusually strong OCR extractor. This directly threatens reproducibility and generalizability, and the absence of any OCR-robustness ablation makes the concern concrete. The reader's weakest_assumption identifies exactly this issue, and I agree. The conditional verdict is appropriate: the central result appears sound, but the paper must disclose the OCR engine and ideally include an OCR-robustness study before the claim is fully accepted.","tokens_in":18456,"tokens_out":13148,"duration_ms":131924,"concrete_test":"Re-run the published training recipe (or, if compute is prohibitive, the fine-tuned checkpoint) on the DocVQA test set using two open-source OCR engines, e.g., PaddleOCR and Tesseract, to generate the text and bounding boxes, keeping all other settings identical. Report ANLS for the Qwen2-VL 256-visual-token + 64-query configuration. If the score drops by more than 2 points relative to the reported 91.2 (or the InternVL2 86.6), the claimed improvements are coupled to the undisclosed OCR system and the paper must specify it and provide an error-rate analysis before the central claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper never identifies the OCR engine that produces the text tokens and 2D boxes consumed by the DocFormerV2 encoder. This is not a cosmetic omission: DocFormerV2 was pretrained on IDL, whose OCR annotations come from a specific system and bounding-box format. At evaluation time, the same unspecified OCR pipeline is run on DocVQA, InfoVQA, and the other benchmarks. If that pipeline is a high-accuracy proprietary service (e.g., AWS Textract), the observed gains may largely reflect oracle-quality text extraction rather than a general property of the method; with a noisy open-source OCR, the 64-query compression cannot recover missing or misrecognized words. No ablation varies OCR quality or engine, so the claim that DocVLM 'enhances VLMs' reading ability' is conditional on an unstated input distribution. The paper should disclose the OCR engine, its word-level error rate on each benchmark, and the box-normalization scheme used (matching DocFormerV2's pretraining).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DocVLM, a model-agnostic adapter that injects OCR-derived text and layout information into frozen vision-language models. A DocFormerV2 encoder processes OCR tokens and 2D boxes, and a set of learned queries compresses this signal into a compact representation (typically 64 tokens) that is concatenated with visual tokens before the LLM. The authors train only the OCR encoder, queries, and projection on single-page document datasets, then evaluate on DocVQA, InfoVQA, TextVQA, ST-VQA, TextCaps, and on multipage benchmarks MP-DocVQA and DUDE. They report large gains in low-visual-token regimes for LLaVA-OneVision, InternVL2, and Qwen2-VL, and claim state-of-the-art results on MP-DocVQA. The paper includes ablations on compression level, OCR encoding strategy, token allocation, and training stages.","tokens_in":18605,"tokens_out":5802,"duration_ms":57543,"significance":"If the results are reproducible, the paper makes a useful practical contribution: it is a simple, model-agnostic way to improve document understanding under tight token budgets while keeping base VLM weights frozen. The experimental coverage is broad, spanning three VLM families and multiple benchmarks, and the ablations are informative, especially the compression-level analysis and the two-stage training study. The promise of processing multipage inputs with a fixed 64-token OCR budget is also attractive. However, the paper lacks code or model release, never identifies the OCR engine, and frames MP-DocVQA as state-of-the-art despite acknowledged overlap with DocVQA training data. These issues currently limit the strength of the central claim.","major_comments":[{"comment":"The OCR engine that produces the text tokens and 2D bounding boxes is never identified. The DocFormerV2 encoder used in the paper was pretrained on IDL with a specific OCR annotation format; if the evaluation OCR pipeline differs, the box normalization and tokenization may not match the pretraining distribution, and the reported gains would not be reproducible. This is load-bearing for the central claim, because the method's input is entirely OCR-derived. The authors should disclose the OCR engine, the exact box encoding and normalization scheme, report word-level OCR error rates on the evaluation benchmarks, and add an ablation that varies OCR quality or uses an open-source OCR engine.","section":"Sec. 3.1, Sec. 4.1, and Supplementary Sec. A"},{"comment":"The abstract and Section 1 claim 'state-of-the-art performance on MP-DocVQA (86.3% vs. 80.3%)' while Supplementary B.2 correctly notes that MP-DocVQA is an extension of DocVQA, whose training split is used in DocVLM's training. The claim 'without multipage training data' is technically true but misleading, because the benchmark is not independent of the training set. The paper should either restrict the zero-shot claim to DUDE or clearly report the overlap and present MP-DocVQA as a partially supervised benchmark.","section":"Abstract, Sec. 1, Sec. 4.4, Supplementary Sec. B.2"},{"comment":"All results in Table 1 and the multipage comparisons are single runs with no error bars, seeds, or significance tests. Some of the reported gaps are small, for example 92.8 versus 91.5 on DocVQA for the Qwen2-VL 576-token configuration, so without variance information the claim that DocVLM 'consistently' improves performance is not fully supported. The authors should report at least three seeds for the main configurations and state whether the differences are stable.","section":"Sec. 4.2, Table 1"}],"minor_comments":[{"comment":"The model name 'LlaV A-OneVision' should be 'LLaVA-OneVision', and 'LlaV A-1.5' in Sec. 2 should be 'LLaVA-1.5'; also, Section 6 contains 'than than' which should be 'than'.","section":"Throughout"},{"comment":"The token counts in Table 1 are inconsistent with the prose: the text says Qwen2-VL was evaluated with 256 and 512 visual tokens, but the table lists 320 and 576 total tokens. Please clarify whether the '# Tok.' column includes OCR tokens and define the relationship for every row.","section":"Sec. 4.2 and Table 1"},{"comment":"The labels '2,134' and '1,342' in Figure 3 are not defined in the caption; please state whether they are inference token counts and add a legend.","section":"Figure 3"},{"comment":"The phrase 'ANLS' is used without spelling out 'Average Normalized Levenshtein Similarity' at first use; please define it in the evaluation section.","section":"Sec. 4.1 and Table 5"},{"comment":"The optimization details list learning rates and step counts but omit batch sizes and the exact image resolutions used for each VLM during training; these should be added for reproducibility.","section":"Supplementary Sec. A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems/application submission, but the reproducibility expectations of the journal are not met without identifying the OCR pipeline and providing variance information. The MP-DocVQA comparison should also be re-framed given the acknowledged DocVQA training overlap. I would not reject the paper on these grounds, as the issues are addressable in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's core result is real. Injecting 64 OCR queries from a DocFormerV2 encoder into frozen VLMs consistently lifts document-understanding scores at tight token budgets, across three different VLM architectures. The ablation story is convincing: compressed encodings retain most of the benefit of full OCR encodings, and even 16 queries help. This is a useful, model-agnostic trick, and the evaluation is unusually systematic for an arXiv preprint.\n\nWhat's actually new: the specific combination of a pretrained document encoder, omitted visual branch, and instruction-aware learned queries to compress OCR text+layout into a tiny token budget. Each piece is borrowed, but the integration and the cross-architecture demonstration are new. The paper does not oversell the mechanism; it's an engineering contribution with a clear cost-benefit story.\n\nSoft spots, in proportion:\n\n1. The OCR engine is never named. That matters more than usual here, because DocFormerV2 was pretrained on IDL with a specific OCR format, and the evaluation runs the same undeclared pipeline on all benchmarks. If that's a high-accuracy proprietary service, the gains may not survive with noisy open-source OCR. There is no ablation varying OCR quality/engine. This is the biggest reproducibility gap; a referee should push for disclosure and at least one robustness experiment.\n\n2. The MP-DocVQA \"state-of-the-art\" framing is generous. The supplementary correctly notes that MP-DocVQA is not true zero-shot, since it inherits DocVQA pages and DocVQA is in the training set. The abstract and main text don't make that caveat, so a reader will take away more than the evidence supports. DUDE is the honest zero-shot result, and it's strong enough on its own.\n\n3. Minor: no code, no error bars, and the learnable-query count variants are a single run each.\n\nNone of this sinks the paper. The central claim—compression via OCR queries improves low-token document understanding—is supported. The remaining issues are addressable in a camera-ready.\n\nWho it's for: anyone working on efficient document VLM inference, multi-page document QA, or token-budget optimization. Worth reading and worth citing. I'd accept it for peer review with requests for OCR disclosure and a softened MP-DocVQA claim.","headline":"Useful, honestly-executed empirical paper: OCR compression into 64 queries genuinely helps frozen VLMs at low token budgets; referee should demand OCR-engine disclosure and a less spun MP-DocVQA claim.","tokens_in":19164,"tokens_out":3380,"would_cite":true,"duration_ms":30701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DocVLM claims that a frozen vision-language model can read documents at low resolution as well as or better than at high resolution when a compact 64-token OCR stream is added to its visual tokens.","keywords":["document understanding","vision-language models","OCR encoding","token compression","layout-aware representation","multipage document VQA","low-resolution efficiency","frozen VLM integration"],"falsifier":"Take DocVQA pages whose OCR words are correct, perturb the 2D bounding boxes by random jitter or by swapping boxes between adjacent lines, and run the pipeline; if accuracy stays roughly constant, layout is not load-bearing, whereas a large drop would show the 64-token representation depends on the exact OCR box format it was trained on.","tokens_in":18266,"feed_emoji":"📄","tokens_out":8990,"duration_ms":80875,"temperature":0.7,"pith_summary":"Reading documents with vision-language models normally forces a trade-off: high-resolution images are accurate but expensive in tokens, while low-resolution vision misses fine text, and pasting raw OCR words into the prompt loses layout and visual context. DocVLM claims that a compact OCR channel can break this trade-off without modifying the VLM. It compresses OCR text together with each word's 2D bounding box into 64 learned queries, then feeds those queries to the frozen LLM alongside the visual tokens. Across three different VLMs (InternVL2, Qwen2-VL, LLaVA-OneVision), the method improves document-QA scores in low-token regimes, often beating the high-resolution baseline, and it transfers to multipage documents with no multipage training. The reason to care is practical: fixed token budgets can now be spent mostly on images while a 64-token OCR stream carries the reading load.","feed_headline":"64 OCR tokens lift document QA from 56.0 to 86.6","feed_subtitle":"DocVLM gives frozen vision-language models a compact reading channel that beats high-resolution vision on fewer tokens.","key_machinery":"The load-bearing machinery is instruction-aware query compression over OCR encodings. The OCR encoder (DocFormerV2's encoder, 344M parameters, visual branch removed) processes three concatenated inputs: embeddings of OCR tokens plus their 2D bounding boxes, embeddings of the question or instruction, and M=64 randomly initialized learnable queries. Only the encoder outputs at the query positions are kept, projected to the VLM's hidden dimension, and inserted before the LLM along with the visual tokens, so an arbitrarily long OCR stream becomes a fixed 64-token representation that is also conditioned on what the user asked. A two-stage training procedure (first text-only OCR-LLM alignment, then vision alignment with image features added) teaches the queries to complement, not duplicate, the visual tokens while keeping the whole VLM frozen.","core_discovery":"DocVLM's central claim is that OCR-derived text and layout can become a privileged reading channel for an already-trained VLM, with no change to the VLM's weights. An OCR encoder (the DocFormerV2 encoder without its visual branch) ingests OCR tokens, their 2D bounding boxes, and the user's instruction; a set of 64 learnable queries compresses this signal, and the query outputs are projected and concatenated with the visual tokens entering the LLM. On DocVQA this raises InternVL2 from 56.0 to 86.6 at 256 visual tokens and Qwen2-VL from 84.4 to 91.2 at 320 tokens, while LLaVA-OneVision improves while using 80% fewer image tokens. With either a single 64-token global encoding for a whole document or 64 tokens per page, the same pipeline reaches 86.3 ANLS on MP-DocVQA without multipage training, surpassing the previous 80.3 result, and 47.4 zero-shot on DUDE.","pith_inferences":["An untested implication is that the 64-query bottleneck acts as task-guided evidence selection: the queries are instruction-aware, so the same page compresses differently for different questions, which suggests the query attention could be used to localize the textual evidence that answers the question.","The design implies portability beyond the three tested VLMs: any VLM with a token-concatenation interface could receive the same 64 OCR tokens, with only a projection layer retrained, but the paper does not demonstrate this.","The unspecified OCR engine is the main hidden variable, so a natural extension is to train the OCR encoder on output from several OCR engines and bounding-box formats and measure how much of the DocVQA gain survives format changes.","The 64-token compression could in principle carry other dense modalities such as tables, math, or handwriting, but the paper only evaluates text-plus-layout documents, so that extension is speculative."],"forward_implications":["At a fixed token budget near 1.5k, DocVLM improves DocVQA, TextVQA, ST-VQA, InfoVQA, and TextCaps over all three base VLMs, and the gains persist even in looser token regimes.","With only 64 OCR tokens, a model can reallocate budget to image tokens: Qwen2-VL with 768 image tokens and 64 OCR tokens reaches 93.0 DocVQA, above the 90.1 obtained with 800 uncompressed OCR tokens.","Multipage documents become tractable without multipage training: global encoding uses 64 tokens for the entire document, page-wise encoding uses 64 per page, and both beat direct OCR word insertion while using fewer tokens, reaching 86.3 ANLS on MP-DocVQA.","Compressed 64-token OCR encodings outperform inserting up to 800 raw OCR words into the prompt across all three VLMs, so the benefit comes from the learned layout-aware representation, not just from adding text.","Because the VLM stays frozen, the OCR channel layers uniformly onto different visual token-reduction strategies (resizing, tiling, downsampling), making the method model-agnostic by construction."],"supporting_citations":[{"why":"Supplies the DocFormerV2 encoder whose text-plus-layout representation DocVLM reuses without its visual branch.","marker":"[7]"},{"why":"The OCR-IDL dataset on which the DocFormerV2 encoder was pretrained, defining the OCR text and bounding-box format DocVLM inherits.","marker":"[13]"},{"why":"InternVL2 is one of the three base VLMs; its tile-based visual token reduction is the setting where DocVQA jumps from 56.0 to 86.6.","marker":"[18]"},{"why":"LLaVA-OneVision is the feature-downsampling base VLM where DocVLM improves results while using 80% fewer image tokens.","marker":"[31]"},{"why":"Qwen2-VL is the resolution-controlled base VLM that yields DocVLM's best results, including 92.8 DocVQA with 576 tokens.","marker":"[52]"},{"why":"GRAM is the prior OCR-enhanced multipage method whose MP-DocVQA result of 80.3 DocVLM surpasses with 86.3.","marker":"[14]"},{"why":"MP-DocVQA is the multipage benchmark used to demonstrate zero-shot extension without multipage training.","marker":"[49]"},{"why":"DUDE is the held-out multipage benchmark on which DocVLM reports 47.4 zero-shot accuracy.","marker":"[50]"},{"why":"DocVQA is the main single-page document-QA benchmark and training set that measures the headline gains.","marker":"[39]"}],"fun_headline_variants":["DocVLM: OCR channel lifts DocVQA 56.0→86.6 on 64 queries","Frozen VLMs gain 30 points in DocVQA with 64 OCR queries","DocVLM: 64 OCR queries beat high-res vision on document QA","OCR queries cut image tokens 80% while boosting DocVQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains depend on the OCR engine's text and bounding-box format matching what the OCR encoder was pretrained on; the paper never names the OCR engine used at inference, so if that format drifts, the 64 learned queries cannot recover text the encoder was not built to read.","fun_headline_variants_meta":{"raw":{"variants":["DocVLM: OCR channel lifts DocVQA 56.0→86.6 on 64 queries","Frozen VLMs gain 30 points in DocVQA with 64 OCR queries","DocVLM: 64 OCR queries beat high-res vision on document QA","OCR queries cut image tokens 80% while boosting DocVQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3099,"prompt_tokens":1028,"completion_tokens":2071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1981}},"tokens_in":644,"tokens_out":2071,"duration_ms":14311,"temperature":1.0,"reasoning_tokens":1981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:35:48.609636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take DocVQA pages whose OCR words are correct, perturb the 2D bounding boxes by random jitter or by swapping boxes between adjacent lines, and run the pipeline; if accuracy stays roughly constant, layout is not load-bearing, whereas a large drop would show the 64-token representation depends on the exact OCR box format it was trained on.","supporting_citations":[{"cited_title":"Docformerv2: Local features for document understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the DocFormerV2 encoder whose text-plus-layout representation DocVLM reuses without its visual branch."},{"cited_title":"Hi- erarchical multimodal transformers for multipage docvqa","cited_arxiv_id":null,"evidence_quote":"MP-DocVQA is the multipage benchmark used to demonstrate zero-shot extension without multipage training."},{"cited_title":"Document understanding dataset and evaluation (dude)","cited_arxiv_id":null,"evidence_quote":"DUDE is the held-out multipage benchmark on which DocVLM reports 47.4 zero-shot accuracy."},{"cited_title":"Docvqa: A dataset for vqa on document images","cited_arxiv_id":null,"evidence_quote":"DocVQA is the main single-page document-QA benchmark and training set that measures the headline gains."}],"review_version":1}