REVIEW 18 cited by
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.
Forward citations
Cited by 18 Pith papers
-
DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning
DocTrace reaches 52.9/56.4/85.1 on MMLongBench-Doc, LongDocURL, and SlideVQA by generating explicit evidence graphs, surpassing open-source baselines on all three and large closed models on two.
-
KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval
A 2B single-vector retriever, KoVRE, trained with hard-negative mining and reranker distillation, achieves top aggregate Korean visual document retrieval scores.
-
HierDoc: Hierarchical Page-to-Region Evidence Routing for Long-Document Visual Question Answering
HierDoc trains separate page- and region-selection policies with GRPO, then answers from full selected pages plus selected region crops and text.
-
XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding
XL-DocBench is a 1,519-question, fully human-verified extra-long document QA benchmark across six professional domains; state-of-the-art systems still score at most 44% on it.
-
When Do Multimodal and Graph-Augmented RAG Help? A Controlled Evaluation for Document Question Answering
Under a fixed retrieval stack, knowledge-graph augmentation never reliably improves document-QA accuracy, visual evidence is necessary but insufficient for pixel-only figure questions, and image tokenization dominates...
-
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
MCERF delivers a 41.1% relative accuracy gain on the DesignQA benchmark by combining ColPali vision-language retrieval with four specialized reasoning modes and dynamic routing.
-
R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation
R3G improves vision-centric VQA by generating a reasoning plan before retrieval and reranking candidate images with an MLLM judge on relevance, target match, and answerability.
-
SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding
SARA combines natural-language snippets with semantic compression vectors in RAG to improve answer relevance, correctness, and similarity on 9 datasets across 5 LLMs.
-
Structured Attention Matters to Multimodal LLMs in Document Understanding
Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.
-
Evidence-Grounded Constraint Checking in Construction Documents
A controlled four-image comparison shows region-focused crops beat page overviews on six projects but lose on 23 projects, indicating a resolution–breadth tradeoff rather than a dominant strategy.
-
HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document
A holistic-view multimodal GraphRAG that conflict-resolves concept indices and retrieves via compact concept anchors plus modality-grouped evidence, beating strong baselines on three complex-document QA sets.
-
DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth
OCR tools can be ranked without ground-truth labels by measuring how much a multimodal LLM must correct each tool's output.
-
Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
A new benchmark and method (Spot-IT) aim to improve multimodal LLMs' ability to locate fine details in documents, with reported significant gains.
-
MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs
MMGraphRAG links scene-graph entities from images to text knowledge graph entities via SpecLink, and reports accuracy gains over naive RAG and GraphRAG on multimodal document QA.
-
Docopilot: Improving Multimodal Models for Document-Level Understanding
A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.
-
Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
Semantic clustering of patch embeddings at the final model layer, plus fine-tuning, preserves 94.6 to 98.2% of retrieval performance while reducing stored embedding memory to 1.8 to 11.8% of original.
-
VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
VRAG-RL uses GRPO reinforcement learning with visual cropping actions and a retrieval-aware reward to improve vision-language RAG agents on document benchmarks.
-
The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
The paper proposes a full-paper, evidence-aware retrieval agenda for scientific fact-checking and shows that combining verification feedback with semantic reranking improves evidence recall on SciFact-Open and Check-COVID.
Discussion (0). Continue with ORCID to comment.