Pith. sign in

REVIEW 4 major objections 5 minor 57 references

VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that separating layout detection from content extraction lets a document model cut image tokens about 3.6x while improving zero-shot key-information extraction by 5.5 F1 points over the previous state of the art.

desk verdict Strong architecture, plausible efficiency win, but zero-shot SOTA needs a dedup audit and error bars before I'd trust it. read the letter →

arxiv 2507.09531 v1 pith:62MP56XO submitted 2025-07-13 cs.CV cs.AIcs.LG

Son Nguyen , Giang Nguyen , Hung Dao , Thao Do , Daeyoung Kim This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords keyinformationextractionvisualdocumentunderstandingmultimodallargelanguagemodelscontent-awaretokenizationzero-shotgeneralizationlayoutmodelingregionofinterestdetectioninstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a document-reading multimodal language model does not have to pay a per-pixel token bill. It claims that by detecting text and figure regions first and reading only those regions, an instruction-tuned model can hold key-information extraction accuracy at or above the level of a leading token-heavy model while using roughly 3.6 times fewer image tokens per page. The same design is said to generalize better to unseen document types, beating the strong baseline by 5.5 F1 points in a zero-shot setting. If this holds, it points to a cheaper and more robust way to deploy document AI on receipts, contracts, and forms.

What carries the argument

The load-bearing mechanism is the dual vision encoder. A spatial encoder—Swin Transformer v2 with a feature pyramid network feeding a Faster R-CNN detection head—proposes text and vision regions of interest, and a learned linear layer turns each bounding box into a spatial token. A semantic encoder pools multi-scale features inside those same boxes (one token per text region, sixteen per visual region, plus sixty-four global cross-modality tokens), so the total token count is a function of detected content, not page size. Three training stages build the components in order: detection pretraining on a newly assembled parsing dataset, feature learning on a document-structure dataset, then instruction tuning on InstructDoc.

What would settle it

Search the Stage-1 parsing dataset and the Stage-2 DocStruct4M collection for near-duplicate images of the FUNSD and CORD test pages (for example, by perceptual hash or retrieval). If any test pages or near-copies were seen during pretraining, the claimed +5.5 zero-shot F1 advantage would be inflated, and the comparison would need to be rerun on genuinely unseen documents.

Watch

Extended reading notes

Core claim

The paper's central claim is that content-aware tokenization—allocating image tokens according to detected document regions rather than image resolution—preserves enough layout and semantic detail to make a 7.2-billion-parameter model state of the art at key information extraction. On its unified KIE benchmark, VDInstruct gets an overall F1 of 71.4 and an out-of-domain F1 of 57.2, versus DocOwl 1.5's 71.3 and 51.7, while emitting about 485 image tokens per document compared with 1,768. The authors attribute the gains to the dual vision encoder: spatial tokens carry geometry, semantic tokens carry content, and the two are interleaved in reading order before the language decoder.

Load-bearing premise

The zero-shot comparison rests on the assumption that the held-out FUNSD and CORD pages, or near-duplicates of them, did not appear in the large pretraining corpora used before instruction tuning; the paper reports no overlap check.

Editorial extensions

If this is right

  • If the central claim holds, token-hungry adaptive cropping is not required for strong KIE accuracy: a content-driven token budget of about 500 tokens per page suffices.
  • The measured +5.5 F1 advantage on held-out FUNSD and CORD indicates that layout-aware, content-driven tokens generalize to unseen document formats better than resolution-scaled tokens.
  • Because spatial-token removal costs 2.0 F1 points overall and 1.5 points more on out-of-domain data, explicit geometry tokens carry real signal rather than acting as placeholders.
  • The modality ablation implies that text-region tokens are the largest single contributor to in-domain accuracy, with vision tokens mainly boosting zero-shot performance.
  • With a stronger detection backbone, both region detection and final KIE accuracy improve, so detector quality is a direct lever on the whole pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader might infer that the approach transfers beyond documents: any domain where a detector can identify informative regions—such as charts, dense screenshots, or mixed scene-text images—could use the same divide-and-conquer token budget.
  • The zero-shot gap could be reinterpreted: DocOwl 1.5 merges patches horizontally, so funnel-shaped receipts or narrow columns may suffer disproportionately; a patch-merge variant that preserves vertical layout would isolate how much of the +5.5 points comes from layout retention alone.
  • A direct prediction follows: the advantage over DocOwl 1.5 should be smallest on simple single-column pages (few spatial cues) and largest on multi-column forms and receipts, which any follow-up evaluation could test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VDInstruct, a multimodal LLM for key information extraction (KIE) that separates spatial region detection from semantic feature extraction via a dual vision encoder. A Faster R-CNN based spatial encoder detects text and vision ROIs and produces spatial tokens, while a semantic encoder pools multi-scale features from these ROIs into semantic tokens. Training proceeds in three stages: layout pretraining on a newly assembled VDInstruct-Parsing dataset, feature learning on DocStruct4M from DocOwl 1.5, and instruction tuning on InstructDoc. On six in-domain KIE datasets and two held-out zero-shot datasets (FUNSD, CORD), the paper reports an overall F1 of 71.4, a zero-shot F1 of 57.2 (+5.5 over DocOwl 1.5), and an average of 485 image tokens per document (3.6x fewer than DocOwl 1.5). The central claims are that content-aware tokenization preserves accuracy while drastically reducing token count, and that the architecture improves zero-shot generalization.

Significance. If the empirical claims hold, the dual-encoder design with content-aware tokenization is a useful step toward efficient document MLLMs, and the token-budget analysis in Appendix C.1 is a concrete, reproducible design recipe. The paper also provides a clear three-stage training scheme and performs several ablations (backbone, spatial tokens, token modalities). However, the significance is heavily contingent on the zero-shot result, which is the only material accuracy advantage over the strongest baseline: in-domain F1 is 1.6 points lower than DocOwl 1.5 and the overall average is a 0.1-point tie. The paper does not currently provide the promised data/code, so independent verification is not yet possible. The most important risk is that the zero-shot margin is inflated by pretraining overlap with the held-out datasets.

major comments (4)
  1. [Section 3.2, Appendix B] The zero-shot conclusion rests on the assumption that FUNSD and CORD do not overlap with the Stage 1 corpus (VDInstruct-Parsing, built from AI2D, DocLayNet, DocBank, SciCap, ScienceQA, KLC, PWC) or the Stage 2 corpus (DocStruct4M from DocOwl 1.5). The paper reports no deduplication, hash matching, or near-duplicate analysis between these pretraining corpora and the held-out test images. Since FUNSD scanned forms and CORD receipts are widely reused in public document collections, a leakage audit (e.g., perceptual hashing, layout-level overlap, and text-sequence overlap) is required before the +5.5 F1 zero-shot gain can be credited. This concern is independent of the architectural soundness.
  2. [Table 1, Section 4.2] All F1 scores are reported from a single evaluation run with no variance estimates, multiple seeds, or significance tests. The overall SOTA claim rests on a 0.1 F1 difference (71.4 vs 71.3), which is likely within run-to-run noise, and the in-domain average is 1.6 F1 below DocOwl 1.5. The authors should report means and standard deviations over at least three runs, or bootstrap confidence intervals, to establish whether VDInstruct matches or exceeds DocOwl 1.5 overall and whether the zero-shot margin is statistically meaningful.
  3. [Table 2, Section 4.3] The ablation of semantic token modalities is performed only at inference time by dropping tokens, without retraining the model for each combination. Because the model was trained with all token types, this does not isolate the contribution of each modality to feature learning, and the conclusion that 'Cross+Text+Vision' is the best configuration may reflect the training setup rather than a genuine advantage. Similarly, the pooling sizes (st=1, sv=4, sg=8) are chosen analytically from the token-budget inequality in Eq. (4) to match LLaVA 1.5's token count, and are not validated against KIE performance. The paper should explicitly state that these hyperparameters were not tuned on the reported benchmarks and, ideally, report a sensitivity analysis over sv and sg.
  4. [Section 1, Section 3.2] The introduction claims that the authors 'curate the first comprehensive, instruction-based KIE benchmark' by unifying eight datasets, but Section 3.2 states that Stage 3 uses the existing InstructDoc dataset, which already provides the six training datasets and the FUNSD/CORD held-out split. The benchmark is therefore not newly curated by this paper. This does not affect the experimental numbers, but the contribution claim should be corrected or clarified to avoid overstatement.
minor comments (5)
  1. [Abstract] The abstract contains a typo: 'poorly ondense documents' should read 'poorly on dense documents'.
  2. [Table 1 caption] The caption states 'VDInstruct shows the SOTA results', but the in-domain average (76.2) is below DocOwl 1.5 (77.8); the SOTA statement is only true for the overall average, so the caption should be more precise, e.g., 'SOTA overall average'.
  3. [Section 4.1] The choice of 'similar number of image tokens as LLaVA 1.5' is reasonable, but the reported average of 485 tokens is lower than LLaVA's 576; the paper should acknowledge this discrepancy, even if it is small and content-dependent.
  4. [Section 3.0] The problem formulation says the model must handle 'novel entity labels Eunseen', but in the actual prompts (Fig. 4) the full category list is provided to the model. The zero-shot setting is therefore better described as generalization to unseen datasets rather than to unseen label sets; the wording should be aligned with the experimental setup.
  5. [Appendix C.1] The derivation of token counts would be clearer if Eq. (4) explicitly substituted st=1 and sg=2*sv with the numeric values, so the reader can verify the ranges (476, 676] directly; currently the jump from Eq. (3) to Eq. (4) requires implicit algebra.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported results are evaluated on external benchmarks with token pooling sizes chosen from a token-budget inequality, not from the reported F1 scores.

full rationale

The derivation chain is self-contained and does not reduce to its own inputs. The key design choices are not fitted to the evaluation metrics: pooling sizes st=1, sv=4, and sg=8 are selected from the analytic token-budget inequality (Eq. 4) using measured ROI statistics (Appendix C.1), so the 3.6x token-efficiency claim is a consequence of the chosen architecture rather than a renamed fit. The accuracy claims rest on external benchmarks: six in-domain KIE datasets plus held-out FUNSD and CORD, with DocOwl 1.5, LLaVA 1.5, and Qwen2VL re-fine-tuned on the same InstructDoc data to make comparisons controlled. No load-bearing self-citation appears in the paper: the authors cite third-party works for DocStruct4M (DocOwl 1.5), InstructDoc (Tanaka et al.), and the detection backbone (Faster R-CNN); none of these citations is by the present authors. The remaining concerns are not circularity: (i) the paper does not report a deduplication audit between Stage 1/2 pretraining corpora and FUNSD/CORD, so the zero-shot +5.5 F1 advantage could be inflated by data leakage; (ii) Table 2 ablations remove semantic token modalities at inference time without retraining, so the 'complementary modalities' conclusion is an interpretation caveat rather than a circular derivation; (iii) the 'Rigorous Benchmark' presented as a contribution is the existing InstructDoc dataset, which is a novelty-attribution matter, not a circular reduction. Because no prediction is equivalent by construction to its input, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The model is a supervised system, so the central claim depends on the usual assumptions that the pretraining datasets are representative and correctly labeled, that Faster R-CNN can produce useful region proposals, and that the pooled features preserve enough semantics. The distinctive assumption is that the held-out zero-shot sets do not overlap with the large pretraining corpora; this is asserted by the evaluation design but not verified. The pooling sizes are hand-chosen to satisfy a token budget and are not fitted to accuracy, so they are design parameters rather than evidence of circularity.

free parameters (3)
  • Text ROI pooling size (s_t) = 1
    One semantic token per text region, chosen to match word-level granularity and keep the token budget near LLaVA 1.5's 576 tokens.
  • Vision ROI pooling size (s_v) = 4
    Selected as the largest value satisfying inequality (4) in Appendix C.1, which keeps total image tokens in [476, 676]; not tuned on KIE F1.
  • Cross-modality pooling size (s_g) = 8
    Set to twice s_v because the average vision ROI is about half the image size (Appendix C.1); no accuracy fitting reported.
assumptions (5)
  • domain assumption Faster R-CNN with Swin Transformer v2 and FPN produces usable region proposals after Stage 1 pretraining.
    The entire tokenization pipeline depends on the spatial encoder finding text and vision regions; the paper reports mAP 0.611 but no downstream robustness analysis against detection failures.
  • domain assumption Azure OCR outputs used to create text ROI annotations in VDInstruct-Parsing are sufficiently accurate.
    Stage 1 training relies on OCR outputs from Azure for datasets without existing text annotations (Appendix B), and no noise analysis is provided.
  • domain assumption InstructDoc ground-truth labels are correct and the six in-domain datasets are unified consistently.
    Stage 3 uses InstructDoc without qualification, and all in-domain F1 scores depend on its label quality.
  • ad hoc to paper FUNSD and CORD do not overlap with VDInstruct-Parsing or DocStruct4M pretraining corpora.
    The zero-shot claim requires this non-overlap, but no deduplication check is reported. If overlap exists, the +5.5 F1 zero-shot advantage is inflated.
  • domain assumption Pooled fixed-size ROI features preserve enough semantic detail for the language decoder.
    Text ROIs are reduced to one token each and vision ROIs to 16 tokens; the ablation in Table 2 tests modality combinations but not the sufficiency of these pooling sizes across document types.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization." pith.science (2026). https://pith.science/paper/62MP56XO

@misc{pith2026250709531,
  author       = {Pith},
  title        = {Pith review of: VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/62MP56XO}},
  note         = {Machine review of arXiv:2507.09531}
}
read the original abstract

Key Information Extraction (KIE) underpins the understanding of visual documents (e.g., receipts and contracts) by extracting precise semantic content and accurately capturing spatial structure. Yet existing multimodal large language models (MLLMs) often perform poorly on dense documents and rely on vision tokenization approaches that scale with image size, leading to redundant computation and memory inefficiency. To address these challenges, we introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction. Central to our model is a content-aware tokenization strategy: rather than fragmenting the entire image uniformly, it generates tokens in proportion to document complexity, preserving critical structure while eliminating wasted tokens. Leveraging a three-stage training paradigm, our model achieves state-of-the-art (SOTA) results on KIE benchmarks, matching or exceeding the accuracy of leading approaches while reducing the number of image tokens by roughly 3.6x. In zero-shot evaluations, VDInstruct surpasses strong baselines-such as DocOwl 1.5-by +5.5 F1 points, highlighting its robustness to unseen documents. These findings show that content-aware tokenization combined with explicit layout modeling offers a promising direction forward for document understanding. Data, source code, and model weights will be made publicly available.

Figures

Figures reproduced from arXiv: 2507.09531 by the authors.

Figure 1
Figure 1. Image-token consumption (efficiency) and F1 scores on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of VDINSTRUCT. (a) The model architecture. The core module is the dual vision encoder: a spatial encoder to detect text/vision regions (named Multimodal Region of Interest - RoI) and encode them into spatial tokens; a semantic encoder to extract corresponding semantic tokens based on detected RoIs. The details of the semantic feature pooling scheme are shown in (b), (c), and (d). with a fraction of the para… view at source ↗
Figure 3
Figure 3. Our proposed 3-stage training paradigm. In [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative results on the out-of-domain datasets. The [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The number of image tokens encoded by VDI [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on multimodal RoI detection. The red boxes indicate the text regions, while the blue boxes indicate the vision [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 36 canonical work pages

  1. [1]

    Docformer: End-to-end transformer for document understanding

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003, 2021. 1, 2

    work page 2021

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization. Text Reading, and Beyond , 2,

    work page

  3. [3]

    Due: End-to-end document understand- ing benchmark

    Łukasz Borchmann, Michał Pietruszka, Tomasz Stanis- lawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Grali´nski. Due: End-to-end document understand- ing benchmark. In Thirty-fifth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 4, 5

    work page 2021

  4. [4]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 6

    arXiv

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 5

    work page 2023

  6. [6]

    Instructblip: towards general- purpose vision-language models with instruction tuning

    W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: towards general- purpose vision-language models with instruction tuning. arxiv. Preprint posted online on June, 15:2023, 2023. 6

    work page 2023

  7. [7]

    Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding

    Hao Feng, Qi Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Science China Infor- mation Sciences, 67(12):1–14, 2024. 2

    work page 2024

  8. [8]

    Unidoc: Unified pretraining framework for document understanding

    Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39–50, 2021. 1

    work page 2021

Show all 57 references
  1. [9]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5, 7, 8

    2016

  2. [10]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 5

    2016 arXiv

  3. [11]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281– 1...

    2024

  4. [12]

    SciCap: Generating captions for scientific figures

    Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 3258–3264, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. 4

    2021

  5. [13]

    mPLUG-DocOwl 1.5: Unified structure learning for OCR- free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mPLUG-DocOwl 1.5: Unified structure learning for OCR- free document understanding. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 309...

    2024

  6. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 2

    2022

  7. [15]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM international conference on multimedia , pages 4083–4091,

  8. [16]

    Icdar2019 compe- tition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 compe- tition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. 5

    2019

  9. [17]

    Funsd: A dataset for form understanding in noisy scanned documents

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), pages 1–6. IEEE, 2019. 3, 5

    2019

  10. [18]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  11. [19]

    Donut: Docu- ment understanding transformer without ocr

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Docu- ment understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7(15):2, 2021. 1

    2021 arXiv

  12. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 5, 6

    2023

  13. [21]

    DocBank: A bench- mark dataset for document layout analysis

    Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. DocBank: A bench- mark dataset for document layout analysis. In Proceedings of the 28th International Conference on Computational Lin- guistics, pages 949–960, Barcelona, Spain (Online), 2020. ...

    2020

  14. [22]

    Selfdoc: Self-supervised document representation learning

    Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Han- dong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5660, ...

    2021

  15. [23]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recog...

    2024

  16. [24]

    Doclayllm: An efficient and effective multi-modal extension of large language mod- els for text-rich document understanding

    Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. Doclayllm: An efficient and effective multi-modal extension of large language mod- els for text-rich document understanding. arXiv preprint arXiv:2408.15045, 2024. 6

    2024 arXiv

  17. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  18. [26]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 4

    2017

  19. [27]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 2, 3

    2023

  20. [28]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 5, 6, 2

    2024

  21. [29]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 6

    2024 arXiv

  22. [30]

    Swin transformer v2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12009–12019,...

    2022

  23. [31]

    Sgdr: Stochas- tic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 2

    2016 arXiv

  24. [32]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 2

    2017 arXiv

  25. [33]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  26. [34]

    Docvqa: a dataset for vqa on docu- ment images

    Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Docvqa: a dataset for vqa on docu- ment images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398, 2(3), 2020. 2

    2020 arXiv

  27. [35]

    Azure Cognitive Services: Optical Char- acter Recognition (OCR)

    Microsoft. Azure Cognitive Services: Optical Char- acter Recognition (OCR). https : / / learn . microsoft . com / en - us / azure / cognitive - 9 services / computer - vision/, 2023. Accessed: 2025-05-05. 4

    2023

  28. [36]

    Rectified linear units im- prove restricted boltzmann machines

    Vinod Nair and Geoffrey E Hinton. Rectified linear units im- prove restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML- 10), pages 807–814, 2010. 4

    2010

  29. [37]

    Cord: a con- solidated receipt dataset for post-ocr parsing

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jae- heung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: a con- solidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019, 2019. 5

    2019

  30. [38]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. 2

    2017

  31. [39]

    Doclaynet: A large human- annotated dataset for document-layout segmentation

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human- annotated dataset for document-layout segmentation. InPro- ceedings of the 28th ACM SIGKDD conference on knowl- edge discovery and data mining, pages 3743–3751, 2022. 4

    2022

  32. [40]

    Going full-tilt boogie on document understanding with text-image-layout transformer

    Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Going full-tilt boogie on document understanding with text-image-layout transformer. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausan...

    2021

  33. [41]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international con- ference on knowledge discovery & data mining, pag...

    2020

  34. [42]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information process- ing systems, 28, 2015. 4

    2015

  35. [43]

    Deep learning based key information extraction from business documents: Sys- tematic literature review

    Alexander Rombach and Peter Fettke. Deep learning based key information extraction from business documents: Sys- tematic literature review. arXiv preprint arXiv:2408.06345,

    arXiv

  36. [44]

    Docile benchmark for document information localization and extraction

    ˇStˇep´an ˇSimsa, Milan ˇSulc, Michal U ˇriˇc´aˇr, Yash Patel, Ahmed Hamdi, Mat ˇej Koci´an, Maty´aˇs Skalick `y, Jiˇr´ı Matas, Antoine Doucet, Micka¨el Coustaty, et al. Docile benchmark for document information localization and extraction. In In- ternational Conference on Doc...

    2023

  37. [45]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 2

    2019

  38. [46]

    Spatial dual-modality graph rea- soning for key information extraction

    Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang. Spatial dual-modality graph rea- soning for key information extraction. arXiv preprint arXiv:2103.14470, 2021. 5

    2021 arXiv

  39. [47]

    Instructdoc: A dataset for zero-shot general- ization of visual document understanding with instructions

    Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Instructdoc: A dataset for zero-shot general- ization of visual document understanding with instructions. In Proceedings of the AAAI conference on artificial intelli- gence, pages 19071–19079, 2024. 2, 3, 5, 6

    2024

  40. [48]

    Docllm: A layout-aware gener- ative language model for multimodal document understand- ing

    Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nour- bakhsh, and Xiaomo Liu. Docllm: A layout-aware gener- ative language model for multimodal document understand- ing. Proceedings of the 62nd Annual Meeting of the Ass...

    2024

  41. [49]

    Vision-enhanced semantic entity recognition in document images via visually-asymmetric consistency learning

    Hao Wang, Xiahua Chen, Rui Wang, and Chenhui Chu. Vision-enhanced semantic entity recognition in document images via visually-asymmetric consistency learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15718–15731, 2023. 3, 5

    2023

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 5, 6

    2024 arXiv

  43. [51]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200, 2020. 1, 2

    2020

  44. [52]

    Layoutlmv2: Multi-modal pre-training for visually-rich document understanding

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Compu- t...

    2021

  45. [53]

    mPLUG- DocOwl: Modularized multimodal LLM for document un- derstanding

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Jun- feng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG- DocOwl: Modularized multimodal LLM for document un- derstanding. arXiv preprint arXiv:2307.02499, 2023. 2

    2023 arXiv

  46. [54]

    UReader: Universal OCR-free visually-situated language understand- ing with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, and Fei Huang. UReader: Universal OCR-free visually-situated language understand- ing with multimodal large language model. InFindings of...

    2023

  47. [55]

    By my eyes: Grounding multimodal large language models with sensor data via vi- sual prompting

    Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, and Sung-Ju Lee. By my eyes: Grounding multimodal large language models with sensor data via vi- sual prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 22...

    2024

  48. [56]

    Structextv2: Masked visual- textual prediction for document image pre-training

    Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Structextv2: Masked visual- textual prediction for document image pre-training. arXiv preprint arXiv:2303.00289, 2023. 2 10

    2023 arXiv

  49. [57]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 6 11 VDInstruct: Zero-Shot Key Information Extraction via Content-Awar...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.