Pith. sign in

REVIEW 13 cited by

mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.02499 v1 pith:2XAMK6IR submitted 2023-07-04 cs.CL cs.AI

Jiabo Ye , Anwen Hu , Haiyang Xu , Qinghao Ye , Ming Yan , Yuhao Dan , Chenlin Zhao , Guohai Xu
show 5 more authors
Chenliang Li Junfeng Tian Qian Qi Ji Zhang Fei Huang
This is my paper
classification cs.CLcs.AI
keywords documentunderstandingocr-freemodelsinstructionmplug-docowllargemodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Document understanding refers to automatically extract, analyze and comprehend information from various types of digital documents, such as a web page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl, have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition, indicating their potential for OCR-free document understanding. Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding. In this paper, we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding. Specifically, we first construct a instruction tuning dataset featuring a wide range of visual-text understanding tasks. Then, we strengthen the OCR-free document understanding ability by jointly train the model on language-only, general vision-and-language, and document instruction tuning dataset with our unified instruction tuning strategy. We also build an OCR-free document instruction understanding evaluation set LLMDoc to better compare models' capabilities on instruct compliance and document understanding. Experimental results show that our model outperforms existing multi-modal models, demonstrating its strong ability of document understanding. Besides, without specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream tasks. Our code, models, training data and evaluation set are available at https://github.com/X-PLUG/mPLUG-DocOwl.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 17 citations worldwide. Full citation record

  1. LLM Agents Can See Code Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    Visual graphs of repository structure added to text inputs for multimodal LLM agents reduce token consumption by up to 26% while maintaining or improving issue-resolution accuracy.

  2. Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion

    cs.CV 2025-05 conditional novelty 7.0 of 10

    PPAD injects MLLM semantic feedback into diffusion denoising via lookahead sketches and ping-pong-ahead resampling, improving text-to-image alignment.

  3. DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Decoupling semantic anchoring from high-res cropping fixes VLM visual attention sinks in document Graph RAG, gaining up to 12.5 SPR points and 40.8% fewer prompt tokens.

  4. XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    XL-DocBench is a 1,519-question, fully human-verified extra-long document QA benchmark across six professional domains; state-of-the-art systems still score at most 44% on it.

  5. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  6. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    M4Doc distills the multimodal representations of a frozen MLLM into an image-only encoder, improving document image translation quality and generalization without requiring the MLLM at inference.

  7. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  8. Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A two-stage layout-focused visual reasoning method, Doc-CoB, improves document question answering by having the model select key layout boxes and then answer from those boxes.

  9. Clapper: Compact Learning and Video Representation in VLMs

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Clapper achieves 13x visual token compression in video VLMs with maintained or improved QA accuracy using a slow-fast representation and a TimePerceiver module.

  10. Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A two-stage model that analyzes page layout first, then parses text, tables, and formulas in parallel, reports state-of-the-art accuracy and speed on the benchmarks it evaluates.

  11. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  12. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  13. Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Instructify converts image metadata into visual instruction-tuning conversations with open LLMs, matching or exceeding GPT-4-generated data quality on LMM benchmarks.

Pith tools