Pith. sign in

REVIEW 11 cited by

mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.03420 v2 pith:3NYXXJTO submitted 2024-09-05 cs.CV

Anwen Hu , Haiyang Xu , Liang Zhang , Jiabo Ye , Ming Yan , Ji Zhang , Qin Jin , Fei Huang
show 1 more author
Jingren Zhou
This is my paper · ORCID
classification cs.CV
keywords documentmulti-pageunderstandingdocowl2high-resolutionperformancetokensvisual
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  2. Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A fine-tuning paradigm that prompts MLLMs to self-generate OCR text before translating document images improves DIMT quality and reduces catastrophic forgetting of OCR.

  3. OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OCR-Reasoning, a 1,069-question benchmark with reasoning-chain annotations for text-rich images, finds that no evaluated multimodal model surpasses 50% accuracy.

  4. Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A two-stage model that analyzes page layout first, then parses text, tables, and formulas in parallel, reports state-of-the-art accuracy and speed on the benchmarks it evaluates.

  5. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  6. Spatially Grounded Explanations in Vision Language Models for Document Visual Question Answering

    cs.CV 2025-07 conditional novelty 5.0 of 10

    EaGERS uses a vision language model's own explanation to mask a document image to relevant grid regions, then re-asks the question, improving DocVQA accuracy while grounding answers spatially.

  7. ExpliCIT-QA: Explainable Code-Based Image Table Question Answering

    cs.CL 2025-07 conditional novelty 5.0 of 10

    ExpliCIT-QA turns table-image question answering into a traceable sequence of table extraction, natural-language reasoning, code generation, execution, and explanation.

  8. DRISHTIKON: Visual Grounding at Multiple Granularities in Documents

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A combined OCR, LLM, and fuzzy-matching pipeline locates answer spans in document images at block, line, word, and point granularity, with line-level grounding F1 of 69.10 on a new 70-document benchmark.

  9. CHAOS: Chart Analysis with Outlier Samples

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A chart perturbation robustness benchmark with five textual and ten visual distortion types, three human-calibrated severity levels, and evaluations of 13 MLLMs on ChartQA and chart summarization.

  10. EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...

  11. Survey of Specialized Large Language Model

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey of 24 specialized LLMs (2022-2025) claims a shift from domain fine-tuning to native architectures, but the synthesis is undermined by citation errors and selection bias.

Pith tools