Pith. sign in

REVIEW 2 major objections 6 minor 54 references

DoPTA: Improving Document Layout Analysis using Patch-Text Alignment

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By aligning image patches with OCR text during pre-training, a compact 85M-parameter document encoder outperforms larger OCR-dependent models on layout analysis, text detection, and classification while needing no OCR at inference.

desk verdict A sensible IoU-guided patch-text alignment loss for document pre-training, with honest ablations; the CLIP-initialization confound and a PubLayNet tie blunt the efficiency and outperformance claims, but the contribution is worth refereeing. read the letter →

arxiv 2412.12902 v2 pith:73JYAKKE submitted 2024-12-17 cs.CV

Nikitha SR , Tarun Ram Menta , Mausoom Sarkar This is my paper · ORCID
classification cs.CV
keywords documentlayoutanalysispatch-textalignmentcontrastivepretrainingOCR-freeunderstandingvisiontransformermaskedimagereconstructionIoU-guidedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that document layout understanding improves when a vision encoder is pre-trained to align its image patches with the words written on the page, even though no OCR is needed when the model is later used. It introduces DoPTA, an 85M-parameter ViT-style encoder trained on IIT-CDIP with two losses: an IoU-guided patch-text contrastive loss that connects each text token to the image patches its bounding box covers, and a masked reconstruction loss for non-textual visual content. With only 250k pre-training steps, DoPTA reports state-of-the-art results on D4LA (70.72 mAP) and FUNSD (94.73 F1), matches a much larger DiT-L on PubLayNet, and runs about 12 times faster than OCR-based VGT at inference because OCR is not invoked. The significance is that textual semantics can be injected into a purely visual encoder at pre-training time, avoiding OCR latency and OCR error propagation downstream.

What carries the argument

The load-bearing object is the TextToPatch contrastive loss. For each text token embedding $X_i^T$ and each image patch embedding $X_j^I$, the loss is an asymmetric cross-entropy $$L_i = -\sum_{j=1}^N Y(T_i,I_j)\,\log \frac{\exp(\$\lambda$ s_{i,j})}{\sum_k \exp(\$\lambda$ s_{i,k})},$$ with similarity $s_{i,j}=X_i^T\cdot X_j^I$ and ground-truth weight $Y(T_i,I_j)=|\mathrm{bbox}(I_j)\cap \mathrm{bbox}(T_i)|/|\mathrm{bbox}(T_i)|$, the fraction of the text token's bounding box covered by that patch. It is averaged over all text tokens and combined with a MAE-style masked reconstruction loss. This loss converts OCR text geometry into a per-patch learning signal, and only the image encoder is kept for downstream tasks.

What would settle it

Train DoPTA from random initialization for the same 250k steps on IIT-CDIP and evaluate on D4LA and FUNSD; if its score drops to the DiT level or below, or if CLIP-initialized DiT matches DoPTA, then initialization rather than patch-text alignment is carrying the result.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a fine-grained, spatially grounded contrastive alignment between text tokens and image patches is a strong pre-training signal for document images. The ground truth for alignment is not a caption but the geometry of the text: for each OCR token, the target distribution over image patches is the fraction of that token's bounding box falling in each patch, and the model is trained to make its patch similarities match that distribution. Adding a masked-pixel reconstruction head on top further helps regions such as figures and diagrams that carry no text. The trained image encoder alone, without OCR at inference, reaches 94.12% on RVL-CDIP, 94.9 mAP on PubLayNet, 70.72 mAP on D4LA, 69.5 mAP on M6Doc, and 94.73 F1 on FUNSD, outperforming the much larger DiT-L and the OCR-dependent VGT on several benchmarks while using fewer parameters and a shorter pre-training schedule.

Load-bearing premise

The argument assumes the improvement is caused by the new alignment loss, not by the fact that DoPTA starts from pre-trained CLIP weights while the main baselines start from scratch; the reported training budget also leaves out the cost of training those CLIP weights.

Editorial extensions

If this is right

  • Document layout detectors can get the semantic benefit of reading text without paying OCR cost at inference, since the text is consumed once during pre-training.
  • Smaller encoders pre-trained this way can match or beat much larger document transformers, shifting the driver of document AI performance from model size to pre-training objective.
  • The roughly 12x inference speedup over OCR-based VGT on D4LA makes high-throughput document parsing more practical.
  • Classes whose identity is carried by their wording (titles, questions, dates, authors) improve most, consistent with the loss aligning patches to words.
  • Combining the alignment loss with masked reconstruction extends gains to non-textual layout elements like figures and tables.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison to DiT and VGT conflates the alignment loss with CLIP initialization; the paper's own CLIP baseline shows the loss adds value over CLIP, but a from-scratch DoPTA is needed to know how much of the SOTA gap over from-scratch baselines comes from the loss versus the initialization.
  • The reported pre-training budget excludes the cost of training the CLIP weights both encoders start from, so including that cost would shrink the claimed compute advantage.
  • If the gains survive training from random initialization, the same IoU-weighted soft alignment could be ported to other dense visual tasks where weak spatial text labels exist, such as scene-text detection or chart and figure parsing.
  • A natural next experiment is to generate text descriptions of non-textual regions so that the alignment loss also covers figures and tables, not only OCR words.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes DoPTA, a ViT-B/16 document image encoder pre-trained on IIT-CDIP with two objectives: a fine-grained TextToPatch contrastive loss in which each OCR token's distribution over image patches is supervised by the IoU between the token bounding box and each patch, and a masked autoencoding reconstruction loss. The text encoder and image decoder are used only during pre-training, so downstream tasks use only the image encoder. The authors evaluate on RVL-CDIP, PubLayNet, D4LA, M6Doc, and FUNSD, reporting state-of-the-art results on D4LA, M6Doc, and FUNSD and on-par PubLayNet performance with DiT-L, while using about 250k pre-training steps.

Significance. If the reported gains are attributable to the proposed objective, DoPTA is a useful OCR-free document encoder: it improves over the CLIP-initialized baseline in the 224/160k ablation of Table 5, outperforms DiT-B and DiT-L on several benchmarks with fewer parameters, and avoids OCR latency at inference. The paper's strengths are the simple and clearly specified loss, the controlled ablation in Table 5 showing that each loss component adds value, and the use of public benchmarks with the same detection and fine-tuning pipeline as DiT/VGT. The main caveat is attribution: because both the image and text encoders are initialized from CLIP-ViT-B/16 while DiT and VGT are trained from scratch, the claimed efficiency advantage over those baselines is not cleanly separated from CLIP initialization.

major comments (2)
  1. [§4.1, Table 5] The claim that DoPTA 'consistently outperforms larger models, while using significantly lesser pre-training compute' is confounded by the CLIP initialization. Section 4.1 states that both the DoPTA encoder and the text encoder are initialized from CLIP-ViT-B/16 weights, whereas DiT and VGT are trained from scratch. Table 5 shows that the CLIP baseline alone already achieves 93.3 mAP on PubLayNet and 64.5 mAP on D4LA at 224 resolution and 160k steps, against 94.62 and 67.92 for the full DoPTA at the same setting. Since the headline results are reported at 512 resolution and 250k steps, and no CLIP baseline is reported at that setting, the 94.9/70.72 numbers on PubLayNet/D4LA may be substantially inherited from the pretrained CLIP weights. To support the efficiency and outperformance claims, the authors should report a CLIP-initialized baseline at 512 resolution with the same detection setup, and ideally also train DoPTA from scratch or initialize DiT/VGT with CLIP weights, so that the isolated contribution of the patch-text alignment loss is demonstrated. The comparison of pre-training steps alone also excludes the compute used to train CLIP on hundreds of millions of image-text pairs.
  2. [Abstract and §4.3.1, Table 2] The headline statement 'consistently outperforms larger models' is contradicted by the PubLayNet results in Table 2, where DoPTA achieves 94.9 mAP and DiT-L also achieves 94.9 mAP, and the text in §4.3.1 explicitly calls this performance 'on-par with DiT-L.' Since 'consistently outperforms' appears in the abstract and in the introduction, it should be qualified to a claim of competitive or improved performance, with the exact benchmark-by-benchmark comparison stated clearly.
minor comments (6)
  1. [Table 2] The baseline 'UDoc []' in Table 2 has an empty citation and no parameter count; please add the missing reference and clarify the experimental setup for that row.
  2. [§3.2, Eq. (2)] The notation bbox(I_j) in Eq. (2) is never defined for image patches; please state explicitly that it denotes the pixel region covered by patch j and explain how partial overlaps between text bounding boxes and patch regions are computed.
  3. [Table 5] The row labels in Table 5 are easy to misread; since all rows use CLIP initialization, consider renaming the rows to 'CLIP init only', 'Reconstruction only', 'Patch-text alignment only', and 'Both losses', and state in the caption that all rows share the same CLIP initialization.
  4. [§4.5] The inference-time comparison adds EasyOCR parsing time to VGT's FPS but not to DoPTA's; please state explicitly in the text that OCR time is included only for OCR-based methods and note that the reported speedup depends on the specific OCR engine and hardware.
  5. [Appendix C] The statement that the Equation category has 'only 2/3 samples in total' is ambiguous and appears inconsistent with the category-wise mAP reported in Table 3; please clarify whether this means two or three ground-truth instances, and if so, consider reporting the confidence intervals or variance for that category.
  6. [Throughout] There are several formatting and typographical issues, including inconsistent spacing in the model name ('DOPTA' vs 'D OPTA') and the misspelling 'signifcantly' in Section 5; please proofread the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pre-training objective is evaluated on held-out benchmarks and is not definitionally tied to the downstream metrics.

full rationale

DoPTA's results are empirical transfer-learning measurements rather than derivations from the method's own definitions. The patch-text alignment loss (Eqs. 1-3) is supervised by OCR text-bbox IoU overlap, and the reconstruction loss is masked-patch MSE; neither equals the downstream evaluation metrics (classification accuracy, layout mAP@[0.50:0.95], or text-detection F1). The paper fine-tunes on each benchmark and reports held-out test/validation performance, and Table 5 includes a CLIP-initialized baseline without DoPTA pre-training, so the reported gains are not forced by construction. The CLIP-ViT-B/16 initialization and the exclusion of CLIP pre-training compute from the efficiency claim are legitimate comparison-fairness concerns, but they do not make any step circular. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. Accordingly no circularity is identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger shows no invented entities; the load-bearing assumptions are the validity of IoU soft targets, the transferability of CLIP initialization, and the reliability of EasyOCR supervision.

free parameters (3)
  • Masking ratio M = 0.6
    Set based on an ablation (Table 6) showing 0.6 gives the best downstream performance among {0.2, 0.4, 0.6}; it is a tuned hyperparameter.
  • Reconstruction loss weight λ = 1 (when used)
    The combined loss uses λ ∈ {0, 1} (Eq. 4); no intermediate values are tested, so its selection is coarse.
  • Learned scaling factor λ in Eq. 1 = learned (value not reported)
    This temperature scales the dot-product similarities in the contrastive loss; its converged value is never reported, so it is an unobserved model parameter.
assumptions (3)
  • domain assumption The IoU-based soft label Y(T_i, I_j) is an effective supervision signal for visual layout understanding.
    The paper assumes that the fraction of a text box overlapping an image patch is a meaningful target for learning text-aware image representations (Eq. 2).
  • domain assumption CLIP-ViT-B/16 can be fine-tuned on document OCR text while preserving cross-modal alignment.
    The method relies on CLIP weights for both encoders; if fine-tuning destroys the CLIP semantic alignment, the patch-text loss might not transfer.
  • domain assumption EasyOCR word boxes on IIT-CDIP are accurate enough for pre-training.
    The paper uses EasyOCR outputs without quantifying their error rate; a few persistent OCR errors are acknowledged but not assessed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DoPTA: Improving Document Layout Analysis using Patch-Text Alignment." pith.science (2026). https://pith.science/paper/73JYAKKE

@misc{pith2026241212902,
  author       = {Pith},
  title        = {Pith review of: DoPTA: Improving Document Layout Analysis using Patch-Text Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/73JYAKKE}},
  note         = {Machine review of arXiv:2412.12902}
}
read the original abstract

The advent of multimodal learning has brought a significant improvement in document AI. Documents are now treated as multimodal entities, incorporating both textual and visual information for downstream analysis. However, works in this space are often focused on the textual aspect, using the visual space as auxiliary information. While some works have explored pure vision based techniques for document image understanding, they require OCR identified text as input during inference, or do not align with text in their learning procedure. Therefore, we present a novel image-text alignment technique specially designed for leveraging the textual information in document images to improve performance on visual tasks. Our document encoder model DoPTA - trained with this technique demonstrates strong performance on a wide range of document image understanding tasks, without requiring OCR during inference. Combined with an auxiliary reconstruction objective, DoPTA consistently outperforms larger models, while using significantly lesser pre-training compute. DoPTA also sets new state-of-the art results on D4LA, and FUNSD, two challenging document visual analysis benchmarks.

Figures

Figures reproduced from arXiv: 2412.12902 by the authors.

Figure 1
Figure 1. Our method achieves superior FPS due to the OCR free [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pre-training of DOPTA. Only the image encoder is required for downstream usage. Refer section Sec. 3 for details. image. The textual and image features are later combined through early or late fusion, using a joint transformer encoder to produce the final representations. Different variants of objectives such as masked image modeling (MIM), masked language modeling (MLM), and image￾language alignment are proposed in… view at source ↗
Figure 3
Figure 3. Heatmap visualisation of the normalised dot product [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Results of DOPTA and existing SOTA document en￾coder models. DOPTA outperforms other methods on multiple benchmarks, despite having less parameters, and a significantly shorter pre-training schedule. Refer to Sec. 4 for more details of individual benchmarks and use ran…
Figure 5
Figure 5. Figure 5: Failure case of DOPTAon layout analysis on D4LA benchmark. Left is DOPTA. Right is VGT. DOPTAincorrectly marks the central region as RegionKV, which was found to be a common error mode [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Plot explaining FPS and Publaynet accuracy of various [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Prediction from DOPTAon layout analysis on Equation class. Left is DOPTA. Right is VGT. DOPTAidentifies equation objects that were not identified by VGT but also encloses extraneous regions leading to poor performance [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prediction from DOPTAon layout analysis on Equation class. Left is DOPTA. Right is VGT. DOPTAidentifies a chemical equation objects that were not identified by VGT [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Heatmap visualisation of the normalised dot product similarity of image patch embeddings and text embeddings taken from [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 32 canonical work pages

  1. [1]

    Docformer: End-to-end transformer for document understanding

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003, 2021. 1, 2

    2021

  2. [2]

    Visual and textual deep feature fusion for document image classification

    Souhail Bakkali, Zuheng Ming, Micka ¨el Coustaty, and Marc ¸al Rusi˜nol. Visual and textual deep feature fusion for document image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 562–563, 2020. 6

    work page 2020

  3. [3]

    Eaml: Ensemble self-attention-based mu- tual learning network for document image classification,

    Souhail Bakkali, Ziheng Ming, Mickael Coustaty, and Marc ¸al Rusi˜nol. Eaml: Ensemble self-attention-based mu- tual learning network for document image classification,

    work page

  4. [4]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 2, 6, 7

    arXiv 2021

  5. [5]

    Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, and Jovana Mitrovi´c

    Ioana Bica, Anastasija Ili ´c, Matthias Bauer, Goker Erdogan, Matko Bo ˇsnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, and Jovana Mitrovi´c. Improving fine-grained understanding in image-text pre-training, 2024. 2

    work page 2024

  6. [6]

    Cascade r-cnn: High quality object detection and instance segmentation

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence , 43(5):1483–1498, 2019. 6

    work page 2019

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2

    2021

  8. [8]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 2

    work page 2021

Show all 54 references
  1. [9]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 2

    2020

  2. [10]

    Big self-supervised mod- els are strong semi-supervised learners

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised mod- els are strong semi-supervised learners. Advances in neural information processing systems, 33:22243–22255, 2020. 2

    2020

  3. [11]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer,

  4. [12]

    Vision grid transformer for document layout analysis

    Cheng Da, Chuwei Luo, Qi Zheng, and Cong Yao. Vision grid transformer for document layout analysis. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 19462–19472, 2023. 1, 2, 5, 6, 7

    2023

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  7. [15]

    Evaluation of deep convolutional nets for document image classification and retrieval

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR), 2015. 1

    2015

  8. [16]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 2

    2020

  9. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 2, 4, 6, 7

    2022

  10. [18]

    Bros: A pre-trained lan- guage model focusing on text and layout for better key infor- mation extraction from documents

    Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained lan- guage model focusing on text and layout for better key infor- mation extraction from documents. In Proceedings of the AAAI Conference on Artificial Intelligence , pages...

    2022

  11. [19]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091,

  12. [20]

    Icdar2019 compe- tition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 compe- tition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. 1

    2019

  13. [21]

    JadedAI. Easyocr. https : / / github . com / JaidedAI/EasyOCR, 2023. 4, 8

    2023

  14. [22]

    Funsd: A dataset for form understanding in noisy scanned documents, 2019

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents, 2019. 8

    2019

  15. [23]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  16. [24]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Confer- ence on Computer Vision, pages 498–517. Springer, 2022. 2, 3, 5, 6

    2022

  17. [25]

    Dit: Self-supervised pre-training for docu- ment image transformer

    Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for docu- ment image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3530–3539,

  18. [26]

    Grounded language-image pre-training, 2022

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training, 2022. 2

    2022

  19. [27]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  20. [28]

    Docbank: A bench- mark dataset for document layout analysis

    Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A bench- mark dataset for document layout analysis. arXiv preprint arXiv:2006.01038, 2020. 1

    2006 arXiv

  21. [29]

    Bi-vldoc: Bidirec- tional vision-language modeling for visually-rich document understanding

    Chuwei Luo, Guozhi Tang, Qi Zheng, Cong Yao, Lianwen Jin, Chenliang Li, Yang Xue, and Luo Si. Bi-vldoc: Bidirec- tional vision-language modeling for visually-rich document understanding. arXiv preprint arXiv:2206.13155, 2022. 2

    2022 arXiv

  22. [30]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 1, 2

    2021

  23. [31]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 1

    2022

  24. [32]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  25. [33]

    {CORD}: A consolidated receipt dataset for post-{ocr} parsing

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jae- heung Surh, Minjoon Seo, and Hwalsuk Lee. {CORD}: A consolidated receipt dataset for post-{ocr} parsing. In Work- shop on Document Intelligence at NeurIPS 2019, 2019. 1

    2019

  26. [34]

    Doclaynet: A large human- annotated dataset for document-layout segmentation

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human- annotated dataset for document-layout segmentation. InPro- ceedings of the 28th ACM SIGKDD conference on knowl- edge discovery and data mining, pages 3743–3751, 2022. 1

    2022

  27. [35]

    Going full-tilt boogie on document understanding with text-image-layout transformer

    Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Going full-tilt boogie on document understanding with text-image-layout transformer. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausan...

    2021

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 2

    2015

  30. [38]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  31. [39]

    Complex document information processing (cdip) dataset, 2022

    Ian Soboroff. Complex document information processing (cdip) dataset, 2022. Accessed: 2024-09-09. 4

    2022

  32. [40]

    Kleister: key in- formation extraction datasets involving long documents with complex layouts

    Tomasz Stanisławek, Filip Grali ´nski, Anna Wr ´oblewska, Dawid Lipi´nski, Agnieszka Kaliska, Paulina Rosalska, Bar- tosz Topolski, and Przemysław Biecek. Kleister: key in- formation extraction datasets involving long documents with complex layouts. In International Conference...

    2021

  33. [41]

    Vl-bert: Pre-training of generic visual- linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- linguistic representations. In International Conference on Learning Representations, 2020. 2

    2020

  34. [42]

    Revisiting unreasonable effectiveness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017. 2

    2017

  35. [43]

    Unifying vision, text, and layout for universal document processing

    Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mo- hit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , ...

    2023

  36. [44]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 2

    2016

  37. [45]

    Training data-efficient image transformers & distillation through at- tention, 2021

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention, 2021. 6, 7

    2021

  38. [46]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019. 6

    2019

  39. [47]

    Aggregated residual transformations for deep neural networks, 2017

    Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, 2017. 6, 7

    2017

  40. [48]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200, 2020. 1, 2

    2020

  41. [49]

    Layoutlmv2: Multi-modal pre-training for visually-rich document understanding

    Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740, 2020. 1, 2

    2012 arXiv

  42. [50]

    FILIP: Fine-grained interactive language- image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language- image pre-training. In International Conference on Learning Representations, 2022. 2, 4

    2022

  43. [51]

    Structextv2: Masked visual- textual prediction for document image pre-training

    Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Structextv2: Masked visual- textual prediction for document image pre-training. arXiv preprint arXiv:2303.00289, 2023. 3, 6

    2023 arXiv

  44. [52]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training,

  45. [53]

    Zhang, H

    N. Zhang, H. Cheng, J. Chen, Z. Jiang, J. Huang, Y . Xue, and L. Jin. M2doc: A multi-modal fusion approach for document layout analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7233–7241, 2024. 2, 6, 7

    2024

  46. [54]

    Pub- laynet: largest dataset ever for document layout analysis

    Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Pub- laynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE, 2019. 1, 6 DOPTA: Improving Document Layout Analysis using Pa...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.