Pith. sign in

REVIEW 17 cited by

TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.04473 v2 pith:QPRI6YKW submitted 2024-03-07 cs.CV cs.AI

Yuliang Liu , Biao Yang , Qiang Liu , Zhang Li , Zhiyin Ma , Shuo Zhang , Xiang Bai This is my paper · ORCID
classification cs.CVcs.AI
keywords tasksmodellargemultimodaldocumentenhanceinformationscene
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. Our approach introduces enhancement across several dimensions: By adopting Shifted Window Attention with zero-initialization, we achieve cross-window connectivity at higher input resolutions and stabilize early training; We hypothesize that images may contain redundant tokens, and by using similarity to filter out significant tokens, we can not only streamline the token length but also enhance the model's performance. Moreover, by expanding our model's capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability. It also learns to perform screenshot tasks through finetuning. Evaluation on 12 benchmarks shows notable improvements: 5.2% in Scene Text-Centric tasks (including STVQA, TextVQA, and OCRVQA), 6.9% in Document-Oriented tasks (such as DocVQA, InfoVQA, ChartVQA, DeepForm, Kleister Charity, and WikiTableQuestions), and 2.8% in Key Information Extraction tasks (comprising FUNSD, SROIE, and POIE). It outperforms in scene text spotting with a 10.9\% increase and sets a new standard on OCRBench, a comprehensive benchmark consisting of 29 OCR-related assessments, with a score of 561, surpassing previous open-sourced large multimodal models for document understanding. Code will be released at https://github.com/Yuliang-Liu/Monkey.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework

    cs.CV 2026-02 conditional novelty 6.0 of 10

    HART uses a closed-loop 'crop-and-answer' training scheme plus a dynamic-weight GRPO variant to improve LMM grounding and high-resolution reasoning without bounding-box annotations.

  2. CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

    cs.CV 2025-08 conditional novelty 6.0 of 10

    CogVLA pairs instruction-conditioned visual-token aggregation (EFA-Routing) with transformer-layer pruning (LFP-Routing) and bidirectional action decoding (CAtten), reporting LIBERO 97.4%, real-world 70.0%, 2.5x less ...

  3. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  4. Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A fine-tuning paradigm that prompts MLLMs to self-generate OCR text before translating document images improves DIMT quality and reduces catastrophic forgetting of OCR.

  5. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    M4Doc distills the multimodal representations of a frozen MLLM into an image-only encoder, improving document image translation quality and generalization without requiring the MLLM at inference.

  6. LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LIRA improves referring segmentation and reduces hallucination in multimodal LLMs by fusing semantic and pixel features and interleaving local image regions with text descriptions.

  7. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  8. Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free two-step pipeline, localize a bounding box then refine the answer from a high-resolution crop, improves MLLM accuracy on high-resolution image benchmarks.

  9. Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Omni-R1 uses GRPO reinforcement learning to train a global reasoning model that selects keyframes and rewrites queries for a detail model, improving video and audio-visual segmentation and out-of-domain QA.

  10. OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OCR-Reasoning, a 1,069-question benchmark with reasoning-chain annotations for text-rich images, finds that no evaluated multimodal model surpasses 50% accuracy.

  11. Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A two-stage model that analyzes page layout first, then parses text, tables, and formulas in parallel, reports state-of-the-art accuracy and speed on the benchmarks it evaluates.

  12. EventSTR: A Benchmark Dataset and Baselines for Event Stream based Scene Text Recognition

    cs.CV 2025-02 reject novelty 6.0 of 10

    The paper presents the first event-camera dataset for scene text recognition and an LLM-based recognizer, but test-set tuning and contradictory data filtering weaken the evaluation.

  13. UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters

    cs.CV 2025-12 conditional novelty 5.0 of 10

    A 0.1B-parameter text/formula recognition model trained on a new 40M-sample dataset matches or beats much larger OCR models and runs 2-9× faster.

  14. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  15. ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An event-stream scene text recognizer trained with LLM-generated chain-of-thought rationales improves BLEU-1 on EventSTR from 0.638 to 0.648 and accuracy on WordArt* and IC15* by about half a point.

  16. Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.

  17. Describe Anything Model for Visual Question Answering on Text-rich Images

    cs.CV 2025-07 conditional novelty 4.0 of 10

    DAM-QA aggregates answers from full-image and sliding-window views of the Describe Anything Model with a weighted vote, improving text-rich VQA on some benchmarks but not all.

Pith tools