Pith. sign in

REVIEW 13 cited by

OCR-free Document Understanding Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2111.15664 v5 pith:VXS2PN65 submitted 2021-11-30 cs.LG cs.AI

Geewook Kim , Teakgyu Hong , Moonbin Yim , Jeongyeon Nam , Jinyoung Park , Jinyeong Yim , Wonseok Hwang , Sangdoo Yun
show 2 more authors
Dongyoon Han Seunghyun Park
This is my paper
classification cs.LGcs.AI
keywords documentunderstandingdonutmodelocr-freesimpletasktransformer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains. The code, trained model and synthetic data are available at https://github.com/clovaai/donut.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Across 675 paired API calls, image input-token reductions are 86.5% (Anthropic), 80.6% (OpenAI), and 75.8% (Gemini) under token-volume weighting, with Gemini images costing far more than text below 200 lines.

  2. XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    XL-DocBench is a 1,519-question, fully human-verified extra-long document QA benchmark across six professional domains; state-of-the-art systems still score at most 44% on it.

  3. Scalable Visual Pretraining for Language Intelligence

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    Visual pretraining on rendered document pages beats text-only continued pretraining on scientific reasoning benchmarks across multiple model backbones, using about a quarter of the tokens.

  4. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  5. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.

  6. Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Granite Vision is a ~3B parameter open-weights vision-language model that reaches state-of-the-art scores on document understanding benchmarks despite its small size.

  7. Reverse Browser: Vector-Image-to-Code Generator

    cs.SE 2025-09 conditional novelty 5.0 of 10

    An open-weights system that turns vector images of web designs into HTML/CSS, with new datasets and a multi-scale pixel metric, though accuracy remains below production quality.

  8. CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fine-tuned Llama 3 model with a trained document critic and program-based reasoning beats GPT-4o and other baselines on a new carbon footprint QA benchmark.

  9. FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Fine-tuning small language models on synthetic financial errors yields high detection and editing scores, but the evaluation is limited to synthetic data from the same pipeline.

  10. ExpliCIT-QA: Explainable Code-Based Image Table Question Answering

    cs.CL 2025-07 conditional novelty 5.0 of 10

    ExpliCIT-QA turns table-image question answering into a traceable sequence of table extraction, natural-language reasoning, code generation, execution, and explanation.

  11. Multimodal Tabular Reasoning with Privileged Structured Information

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An 8B multimodal LLM trained on 9k reasoning traces distilled from structured tables reaches state-of-the-art open-source accuracy on table-image question answering and fact verification.

  12. E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition

    cs.CL 2025-09 reject novelty 4.0 of 10

    Their custom PaddleOCR-based system achieves the best F1 (0.46), fastest latency (0.17 s/image), and lowest cost ($0.006/1k images) among seven OCR systems on a private 54-language benchmark.

  13. Digitization of Document and Information Extraction using OCR

    cs.CV 2025-06 reject novelty 2.0 of 10

    The paper describes a standard OCR-plus-LLM extraction pipeline and reports unsupported accuracy percentages with no code, data, or baselines.

Pith tools