REVIEW 7 cited by
DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2,560$\times$2,560 resolution. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both perception and comprehension abilities of our model, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments conducted on various publicly available benchmarks confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.
Forward citations
Cited by 7 Pith papers
-
Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth
Constraining visual token budgets during SFT and RL forces VLMs to learn functional active perception, yielding ~5% relative gains and strong transfer to unconstrained evaluation.
-
MFH: Marrying Frequency Domain with Handwritten Mathematical Expression Recognition
MFH fuses high-frequency DCT features with spatial features from standard HMER encoders, improving recognition accuracy by about 1 to 2 points on CROHME 2014/2016/2019.
-
Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence
Granite Vision is a ~3B parameter open-weights vision-language model that reaches state-of-the-art scores on document understanding benchmarks despite its small size.
-
EventSTR: A Benchmark Dataset and Baselines for Event Stream based Scene Text Recognition
The paper presents the first event-camera dataset for scene text recognition and an LLM-based recognizer, but test-set tuning and contradictory data filtering weaken the evaluation.
-
Docopilot: Improving Multimodal Models for Document-Level Understanding
A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.
-
ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning
An event-stream scene text recognizer trained with LLM-generated chain-of-thought rationales improves BLEU-1 on EventSTR from 0.638 to 0.648 and accuracy on WordArt* and IC15* by about half a point.
-
Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning
CAR routes each query to either a short answer or full reasoning based on the perplexity of the model's draft answer, improving accuracy and cutting token use on VQA, KIE, and math/common sense benchmarks.
Discussion (0). Continue with ORCID to comment.