Pith. sign in

REVIEW 4 major objections 5 minor 61 references

\'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documents

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims ECLAIR is the first model to simultaneously extract a document page's text in reading order, bounding boxes, and semantic class labels in one autoregressive pass, and it reports higher accuracy than prior systems on a…

desk verdict Genuinely new single-pass document extraction with a clever TeX-instrumentation pipeline, but reading-order claims rest on an unvalidated labeling convention and a self-created benchmark. read the letter →

arxiv 2502.04223 v1 pith:5ZFWM6P7 submitted 2025-02-06 cs.CV

Ilia Karmanov , Amala Sanjay Deshmukh , Lukas Voegtle , Philipp Fischer , Kateryna Chumachenko , Timo Roman , Jarno Seppänen , Jupinder Parmar
show 3 more authors
Joseph Jennings Andrew Tao Karan Sapra
This is my paper · ORCID
classification cs.CV
keywords documentOCRreadingordersemanticsegmentationboundingboxpredictionmultimodalLLMLaTeXdatagenerationlayoutanalysisend-to-end
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ECLAIR is an end-to-end document extraction model that takes a page image and outputs structured text in reading order together with bounding boxes and semantic class labels, all from a single auto-regressive decoder. The authors claim this is the first time all three output types are produced simultaneously, selectable through a prompt with eight valid combinations. To make that possible, they generate a 5-million-page training set by instrumenting a LaTeX compiler to emit reading-order-annotated ground truth directly from source, and they release DROBS, a human-annotated benchmark of 789 layout-diverse pages. On DROBS they report higher accuracy than Kosmos-2.5 and GOT, and on established benchmarks they report competitive results for layout detection, formula/table extraction, and LLM pretraining data quality. If the claim holds, document understanding no longer needs a brittle chain of separate OCR, detection, and reordering models.

What carries the argument

The machinery is a prompt-conditioned output serialization together with a source-to-label data pipeline. The output side: a prompt tuple chooses among 8 valid combinations of structured/plain/no text, bounding boxes on/off, and class labels on/off, and each semantic block is emitted as a discrete-coordinate token sequence <x y> text <x y> <class> in a canonical reading order, with coordinates tokenized as special tokens. The data side: ECLAIR's arXiv-5M dataset is produced by a modified TeX Live distribution with hooks inside the TeX compiler and an embedded Python interpreter that tracks node, character, and hbox/vbox allocations from source to PDF page, yielding a nested hierarchy of semantic blocks with character-level alignment. This pipeline is what makes joint text+box+class supervision available at 5M-page scale; the architecture itself is a RADIO-initialized ViT-H vision encoder, a horizontal 1x4 convolution neck, and an mBART auto-regressive decoder trained with teacher forcing and, for the speed variant, n parallel linear heads predicting n tokens per step.

What would settle it

Select 200 multi-column magazine and textbook pages outside the training distribution, have two independent annotators record the reading order of every text block, and keep only pages where the annotators fully agree. Run ECLAIR on those pages with the maximal-information prompt; if the model's block order disagrees with the agreed human order on a consistent class of layouts (for example, a sidebar read before the main column, or a footnote placed before the paragraph that cites it), the claim that ECLAIR preserves human reading order is falsified regardless of the paper's benchmark numbers.

Watch

Extended reading notes

Core claim

This paper claims ECLAIR is the first model to extract, from a single document image, formatted text in reading order together with bounding boxes and semantic class labels at the same time, using one auto-regressive decoder rather than a pipeline of separate OCR, layout-analysis, and reordering components. The claim is operationalized by a prompt space of eight valid output combinations, from plain text only up to the maximal-information prompt <structured_text><bbox><classes> in which each block appears as its markdown/LaTeX text wrapped in discrete-coordinate box tokens and a class token. The authors substantiate the claim with a new human-annotated benchmark, DROBS (789 pages from magazines, books, and web corpora), on which ECLAIR reports higher word-level F1, lower edit distance, and better counting-F1 than Kosmos-2.5 and GOT in both their OCR and markdown modes; and they report competitive or superior scores on the GOT/Fox benchmark, DocLayNet object detection, formula/table extraction on arXiv validation pages, and an LLM-pretraining comparison where Nemotron-8B trained on ECLAIR-extracted text scores higher on MMLU than when trained on PyMuPDF4LLM-extracted text.

Load-bearing premise

The load-bearing assumption is that the modified LaTeX compiler correctly converts LaTeX source into reading-order-annotated bounding boxes for about five million arXiv pages; if it mis-orders floats, footnotes, or multi-column content, the model's learned reading order is wrong no matter how well it scores on downstream benchmarks.

Editorial extensions

If this is right

  • Reading-order-accurate text can be produced alongside layout geometry in a single autoregressive pass, eliminating the need to stitch separate OCR, layout-detection, and ordering stages.
  • A single checkpoint covers the full prompt space: plain text-only output, markdown with formulas, or maximal output with boxed and classified blocks, so one model can serve OCR, detection, and data-curation workloads.
  • Because boxes carry semantic classes, downstream document assembly can drop page furniture such as headers and footers, and relocate floats such as footnotes, captions, and pictures, rather than leaving them inline.
  • The multi-token decoding variant (2-3 tokens per step) keeps or improves OCR metrics while speeding up inference, making high-quality extraction feasible at web scale.
  • LLM pretraining on text extracted by ECLAIR yields higher MMLU and average benchmark scores than text extracted by a traditional PDF-to-markdown tool, under the paper's controlled comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The TeX-instrumentation recipe for arXiv-5M could be ported to other source-format corpora (HTML, Markdown, or Office files) to synthesize aligned image-text-layout data at scale, which would let the same prompt-conditioned design cover slide decks, web articles, and print magazines without new human annotation.
  • Because the model emits class-labeled boxes, downstream retrieval systems could cite a specific table, formula, or footnote by location, turning page-level OCR into a structured document database; this is a use the paper motivates but does not implement.
  • The multi-token decoding results hint that the auxiliary heads learn useful prediction targets even when only the first token is kept, so architecture-level changes to how the n heads share context may extend the speed-accuracy frontier beyond the paper's 2-3 token finding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ÉCLAIR, an end-to-end multimodal document-extraction model that predicts, from a page image, formatted text (markdown/LaTeX), bounding boxes, and semantic classes, optionally in any of eight prompt configurations. The authors train ÉCLAIR on a new large-scale dataset, arXiv-5M, whose labels are produced by a modified TeX compiler that records character-level boxes, semantic classes, and a rule-based reading order. They also release DROBS, a 789-page human-annotated benchmark with reading order, boxes, and classes, and report SOTA or competitive results on DROBS, the GOT benchmark, DocLayNet, a Nougat-style arXiv validation set, and an LLM-pretraining comparison against PyMuPDF4LLM. Additional contributions include a multi-token inference scheme that speeds up decoding by about 2x without degrading accuracy.

Significance. If the claims are supported, ÉCLAIR is a useful practical contribution: it combines content extraction, layout detection, and semantic classification in a single pass, and the multi-token inference result is valuable for throughput. The paper deserves credit for evaluating on external benchmarks (GOT, DocLayNet, Nougat validation) and for measuring downstream LLM-pretraining quality, not just OCR metrics. The DROBS benchmark, if released with careful protocols, could be a useful community resource. However, the flagship SOTA claim rests on a benchmark created by the authors, and the paper's central novelty—reading-order-preserving structured extraction—is trained on rule-based labels whose agreement with human reading order is not validated. Those two issues are load-bearing for the paper's main claims and require stronger evidence.

major comments (4)
  1. [§2.5, §S3.1] The reading-order labels that train the model's core capability come from a rule-based system inside a modified TeX compiler (§2.5), and §S3.1 states that Page-Header elements are moved to the start while Footnote, Page-Footer, Picture, Table, and Caption elements are moved to the end. The manuscript reports no validation of this ordering against human annotations, and it does not state whether the DROBS annotation guidelines follow this same convention or a natural human reading order. This matters because if the training order diverges from the benchmark's notion of reading order, the DROBS reading-order results cannot validate the capability; the paper should either validate arXiv-5M labels on a human-annotated subset or report DROBS reading-order agreement under both conventions.
  2. [§3.1, Table 2] The DROBS comparison applies asymmetric preprocessing: tables and equations are masked for all models, headers and footers are additionally masked for GOT(md), and TeX commands are filtered from GOT outputs, while §S4 describes a hallucination-filtering post-processing step for ÉCLAIR's predictions whose use during the DROBS evaluation is not stated explicitly. These choices make the 'state-of-the-art' claim hard to interpret. The authors should report results under a single preprocessing protocol for all models, report the sensitivity of Table 2 to each preprocessing step, and provide confidence intervals or significance tests across the 789 pages, ideally together with inter-annotator agreement for the reading-order labels.
  3. [§3.2, Table 4] The formula and table extraction results are reported on a validation set generated by the same modified-TeX pipeline that produced the training labels, and the Nougat numbers come from Nougat's own validation set, so the two columns are not directly comparable. The paper is appropriately cautious in saying it does not aim for a direct comparison, but the table still invites the reader to compare columns; a direct comparison on a shared, independently annotated set of formulas and tables is needed to substantiate the implicit quality claim.
  4. [§3.3, Table 6, §S5] The DocLayNet mAP comparison relies on sequence augmentation and top-k class-token sampling (§3.3), and §S5 explains that mAP is not meaningful for ÉCLAIR because the model produces no box score and only a single recall level. The disclosed caveats are appreciated, but the headline 'competitive' claim in Table 6 should be reported together with the matched-precision/recall comparison (Table S1) in the main text, or with the caveat stated there rather than only in the supplement.
minor comments (5)
  1. [§2.3] There are typographical spacing errors in the output-format description, including 'the the first group' and 'thethird group'; these should be fixed.
  2. [§3.4] The benchmark name 'OpenBooxQA' appears to be a typo for 'OpenBookQA'.
  3. [§3.1] The Counting F1 metric is defined over a fixed set of words with no explanation of how that set is derived or why it is appropriate for DROBS; please clarify.
  4. [§3.1, Table 2] The table shows ÉCLAIR-MIP with both 'Mask out' and no-mask-out rows, but the text does not clearly state which row is the primary comparison; please label the rows and the masking protocol more explicitly.
  5. [§S5] There is a typo 'predictior' near the end of the section, and the phrase 'the GOT benchmark proposed in Fox [27]' appears to attribute a benchmark to the Fox paper when the benchmark is introduced in the GOT paper; the citation should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are tested against external and independently human-annotated benchmarks; the unvalidated TeX-label pipeline is an input-assumption risk, not an equation-level self-reduction.

full rationale

The paper has no derivation chain in which a predicted quantity is constructed from the very quantity it is meant to predict. ECLAIR's headline SOTA on DROBS (Table 2) is measured against 789 human-annotated pages that are not produced by the modified TeX pipeline and are not used to fit the model; the same capability is also checked on the external GOT benchmark (Table 3), DocLayNet (Table 6), and the LLM/MMLU comparison (Table 7), so the central results have independent content. The clearly self-referential parts are correctly hedged: formula/table quality in Section 3.2 is reported on the arXiv-5M validation split generated by the paper's own TeX instrumentation, and the authors explicitly state they 'do not aim to provide a direct comparison between Nougat and ECLAIR here' because of output-formatting differences, so this is a benchmark limitation rather than a fitted parameter renamed as a prediction. The reading-order labels in Section 2.5 / S3.1 are produced by a rule-based TeX compiler hook and S3.1 imposes a specific non-natural convention (Page-Header placed at the start; Footnote, Page-Footer, Picture, Table, and Caption placed at the end); this is an unvalidated input assumption and a genuine correctness risk for the 'reading order' capability, but the labels are not defined as the model's own outputs and the DROBS ground truth is human-annotated, so the capability does not reduce to its input by construction. Minor self-citations to NVIDIA Nemotron technical reports [34,38] support only the generic statement that LLM training needs high-quality data; they are not load-bearing for ECLAIR's architecture, training, or evaluation. The unusual footnote placement in S5.1 ('Pre- ... eval/issues/46') is a formatting artifact of a footnote interrupting the word 'Precision', not a missing citation or circular step.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central capabilities rest on the correctness of the TeX instrumentation labels and the human annotations, neither of which is independently validated. The only hand-tuned parameter that influences the headline numbers is the repetition penalty.

free parameters (1)
  • repetition penalty = 1.1
    Inference-time hyperparameter; Table 5 shows it raises Counting F1 on DROBS from 0.925 to 0.934, so the reported SOTA results depend on this manually chosen value.
assumptions (3)
  • domain assumption The modified TeX compiler (Section 2.5) correctly converts LaTeX source into reading-order-annotated bounding boxes for 5M arXiv pages.
    The entire arXiv-5M training set is generated by this pipeline; if it mis-orders or mislabels boxes for floats, footnotes, or multi-column layouts, the model learns incorrect structure. No validation of the pipeline's accuracy is provided.
  • domain assumption DROBS human annotations are correct and represent the intended reading order.
    DROBS is the primary evidence for the SOTA claim; no inter-annotator agreement or error analysis is reported.
  • ad hoc to paper Masking out tables and equations in DROBS evaluation images is a fair comparison condition for all models.
    Because the benchmark lacks labels for these elements, the authors remove them from the input; this preprocessing is uniform but its effect on each model is unmeasured, and additional per-model masking is applied to GOT.

how reviews work

0 comments
Cite this review

Pith. "Pith review of \'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documents." pith.science (2026). https://pith.science/paper/5ZFWM6P7

@misc{pith2026250204223,
  author       = {Pith},
  title        = {Pith review of: \'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZFWM6P7}},
  note         = {Machine review of arXiv:2502.04223}
}
read the original abstract

Optical Character Recognition (OCR) technology is widely used to extract text from images of documents, facilitating efficient digitization and data retrieval. However, merely extracting text is insufficient when dealing with complex documents. Fully comprehending such documents requires an understanding of their structure -- including formatting, formulas, tables, and the reading order of multiple blocks and columns across multiple pages -- as well as semantic information for detecting elements like footnotes and image captions. This comprehensive understanding is crucial for downstream tasks such as retrieval, document question answering, and data curation for training Large Language Models (LLMs) and Vision Language Models (VLMs). To address this, we introduce \'Eclair, a general-purpose text-extraction tool specifically designed to process a wide range of document types. Given an image, \'Eclair is able to extract formatted text in reading order, along with bounding boxes and their corresponding semantic classes. To thoroughly evaluate these novel capabilities, we introduce our diverse human-annotated benchmark for document-level OCR and semantic classification. \'Eclair achieves state-of-the-art accuracy on this benchmark, outperforming other methods across key metrics. Additionally, we evaluate \'Eclair on established benchmarks, demonstrating its versatility and strength across several evaluation standards.

Figures

Figures reproduced from arXiv: 2502.04223 by the authors.

Figure 1
Figure 1. ECLAIR outperforms other methods on complex documents: (a) tables, formulas, figure, page header and multiple columns; ´ (b) uneven columns, styling, figure; (c) non-obvious reading order and visual elements like background coloring. semantic classes, and any combination of these simultane￾ously, while preserving the reading order. Training such a versatile model necessitates a dataset that encompasses all these ann… view at source ↗
Figure 2
Figure 2. Meta architecture for ECLAIR showcasing the usage with two different (out of eight valid) prompts: Example a) uses the ´ maximal information prompt to return bounding boxes along with their semantic class, markdown text, and tables and formulas. In b) we ask the model to return only markdown text without boxes or classes. All supported semantic classes are listed on the right. with 8 possible valid combinations (ign… view at source ↗
Figure 3
Figure 3. Example pages from DROBS, our visually diverse doc [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages

  1. [1]

    Claude 3: A new era in ai with advanced rea- soning and intelligence

    Anthropic. Claude 3: A new era in ai with advanced rea- soning and intelligence. https://www.anthropic. com/claude, March 2024. Claude 3 is a family of large language models including Haiku, Sonnet, and Opus vari- ants, designed for advanced reasoning, analysis, and com- plex problem-solving. 8

    work page 2024

  2. [2]

    Scenescript: Reconstructing scenes with an autoregressive structured language model, 2024

    Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model, 2024. 7, 14

    work page 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 6, 8

    arXiv 2023

  4. [4]

    SwinDocSegmenter: An end-to-end unified domain adaptive transformer for document instance segmentation

    Ayan Banerjee, Sanket Biswas, Josep Llad ´os, and Umapada Pal. SwinDocSegmenter: An end-to-end unified domain adaptive transformer for document instance segmentation. In International Conference on Document Analysis and Recog- nition, pages 307–325. Springer, 2023. 7, 8, 16

    work page 2023

  5. [5]

    Piqa: Reasoning about physical commonsense in nat- ural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in nat- ural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020. 7

    2020

  6. [6]

    Nougat: Neural optical understanding for academic documents

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical understanding for academic documents. In International Conference on Learn- ing Representations, 2024. 1, 3, 4, 6, 8, 13

    work page 2024

  7. [7]

    Medusa: Simple llm inference acceleration framework with multiple decod- ing heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Ja- son D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decod- ing heads. arXiv preprint arXiv:2401.10774, 2024. 8

    arXiv 2024

  8. [8]

    Pix2seq: A language modeling framework for object detection

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Ge- offrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021. 7, 16

    arXiv 2021

Show all 61 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lukas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024

  4. [12]

    https : //commoncrawl.org/

    The Common Crawl Foundation. https : //commoncrawl.org/. 4

  5. [13]

    Better and faster large language models via multi-token prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better and faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024. 8

    2024 arXiv

  6. [14]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 7, 8

    2017

  7. [15]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 7

    2009 arXiv

  8. [16]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895,

    arXiv

  9. [17]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jin- gren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024. 13

    2024

  10. [18]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091,

  11. [19]

    Pymupdf4llm: A python package for extracting pdf content in markdown format, 2024

    Artifex Software Inc. Pymupdf4llm: A python package for extracting pdf content in markdown format, 2024. Version 0.0.17. 7

    2024

  12. [20]

    Ultralytics yolov5, 2020

    Glenn Jocher. Ultralytics yolov5, 2020. 8

    2020

  13. [21]

    Weld, and Luke Zettle- moyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettle- moyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017...

    2017

  14. [22]

    Ctrl: A condi- tional transformer language model for controllable genera- tion

    Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A condi- tional transformer language model for controllable genera- tion. arXiv preprint arXiv:1909.05858, 2019. 5, 13

    1909 arXiv

  15. [23]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Confer- ence on Computer Vision, pages 498–517. Springer, 2022. 1, 8

    2022

  16. [24]

    The stack: 3 tb of permissively licensed source code

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Mu ˜noz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code. Preprint,

  17. [25]

    Race: Large-scale reading com- prehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading com- prehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. 7

    2017 arXiv

  18. [26]

    Focus anywhere for fine- grained multi-page document understanding

    Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chun- rui Han, and Xiangyu Zhang. Focus anywhere for fine- grained multi-page document understanding. arXiv preprint arXiv:2405.14295, 2024. 6

    2024 arXiv

  19. [27]

    Focus anywhere for fine-grained multi-page document understanding, 2024

    Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Focus anywhere for fine-grained multi-page document understanding, 2024. 5, 6

    2024

  20. [28]

    Multilingual denoising pre-training for neu- ral machine translation

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neu- ral machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020. 2, 13

    2020

  21. [29]

    Textmonkey: An ocr-free large multimodal model for understanding document

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024. 6

    2024 arXiv

  22. [30]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 13

    2017 arXiv

  23. [31]

    Kosmos-2.5: A multimodal literate model

    Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shum- ing Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419, 2023. 1, 3, 4, 8

    2023 arXiv

  24. [32]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sab- harwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. 7

    2018 arXiv

  25. [33]

    Tableformer: Table structure understanding with transformers

    Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. Tableformer: Table structure understanding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4614– 4623, 2022. 4, 13

    2022

  26. [34]

    Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sir- shak Das, Ayush Dattagupta, Olivier Delalleau, Leon Der- czynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksand...

  27. [35]

    String normalization

    Ocropus. String normalization. https://github. com / ocropus / ocropus4 - eval / blob / main / ocroeval/eval.py. 5

  28. [36]

    GPT-4o: Large language model

    OpenAI. GPT-4o: Large language model. https:// openai.com/index/hello- gpt- 4o , 2024. Ac- cessed: 2024-11-12. 8

    2024

  29. [37]

    A universal document converter

    Pandoc. A universal document converter. https:// pandoc.org/index.html. 4

  30. [38]

    Nemotron-4 15b technical report,

    Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabalesh- warkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, J...

  31. [39]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786 , 11

    arXiv

  32. [40]

    DocLayNet: A large human- annotated dataset for document-layout segmentation

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. DocLayNet: A large human- annotated dataset for document-layout segmentation. InPro- ceedings of the 28th ACM SIGKDD conference on knowl- edge discovery and data mining, pages 3743–3751, 2022. 2, 4

    2022

  33. [41]

    Source code for nltk.metrics.scores

    NLTK Project. Source code for nltk.metrics.scores. https: / / www . nltk . org / _modules / nltk / metrics / scores.html. 5

  34. [42]

    AM-RADIO: Agglomerative vision founda- tion model reduce all domains into one

    Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision founda- tion model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024. 2, 13

    2024

  35. [43]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016. 8

    2016

  36. [44]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99–106, 2021. 7

    2021

  37. [45]

    Statistics of the Common Crawl Corpus 2012, 2013

    Sebastian Spiegler. Statistics of the Common Crawl Corpus 2012, 2013. 4

    2012

  38. [46]

    An overview of the tesseract ocr engine

    Ray Smith. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recogni- tion (ICDAR 2007), volume 2, pages 629–633. IEEE, 2007. 13

    2007

  39. [47]

    Galactica: A large language model for science, 2022

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poul- ton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022. 3

    2022

  40. [48]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le...

    2024

  41. [49]

    L. Vincent. Google book search: Document understand- ing on a massive scale. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), vol- ume 2, pages 819–823, 2007. 4

    2007

  42. [50]

    Vary: Scaling up the vision vocabulary for large vision-language models, 2023

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models, 2023. 6

    2023

  43. [51]

    Small language model meets with reinforced vision vocabu- lary, 2024

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Small language model meets with reinforced vision vocabu- lary, 2024. 6

    2024

  44. [52]

    General OCR theory: Towards OCR-2.0 via a unified end-to-end model

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jian- jian Sun, Yuang Peng, et al. General OCR theory: Towards OCR-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704, 2024. 1, 2, 4, 6, 8

    2024 arXiv

  45. [53]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. 7 12 Supplementary Material S1. Architecture Details The entire architecture has a total of 937M parameters. Vis...

    1905 arXiv

  46. [54]

    AP is the area under the PR-Curve, which degenerates to a single point without the possibility to rank predictions, making the calculation of the area not meaningful

  47. [55]

    Identical scores (as in our case) lead to incorrect PR- Curves and inconsistent results

    The COCO implementation assumes scores are unique. Identical scores (as in our case) lead to incorrect PR- Curves and inconsistent results. 4. 4See https : / / github . com / MiXaiLL76 / faster _ coco _

  48. [56]

    first separate classes, then match boxes)

    COCO mAP is computed per class independently (i.e. first separate classes, then match boxes). We propose to first match boxes over all classes and then compute the per-class precision/recall, which allows us to plot a con- fusion matrix, to better visualize problematic cases. ...

  49. [57]

    To manage paragraphs that span across pages, we need to carry open paragraphs over to next pages

    Process Pages Individually: Each page is processed separately. To manage paragraphs that span across pages, we need to carry open paragraphs over to next pages

  50. [58]

    Reassign Floating Objects: Floating objects (e.g., im- ages, tables, captions) are removed and captions are reassigned to their respective objects using Hungarian matching based on the Manhattan distance of the bound- ing boxes

  51. [59]

    Floating text blocks (e.g., Text and List-item) are merged based on specific rules, such as not ending with punctuation

    Concatenate Pages: Pages are concatenated while skip- ping sections like Table of Contents, Bibliography, and Indexes by detecting typical headings. Floating text blocks (e.g., Text and List-item) are merged based on specific rules, such as not ending with punctuation

  52. [60]

    Remove Markdown Formatting: All markdown for- matting is removed from the inner text to ensure consis- tency

  53. [61]

    Flush Floating Objects: After processing each page, floating objects that are not part of the floating text are flushed to the output blocks. S7. Examples of predictions In this section, we present examples of predictions from ´ECLAIR on samples from the Common Crawl dataset. ...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.