Pith. sign in

REVIEW 8 cited by

MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.21439 v2 pith:LXTEBVWO submitted 2024-07-31 cs.AI cs.CLcs.LG

Zhanpeng Chen , Chengjin Xu , Yiyan Qi , Jian Guo This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords generationmultimodaltrainingdataabilityaccurateimagesknowledge-enhanced
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in processing and generating content across multiple data modalities. However, a significant drawback of MLLMs is their reliance on static training data, leading to outdated information and limited contextual awareness. This static nature hampers their ability to provide accurate and up-to-date responses, particularly in dynamic or rapidly evolving contexts. Though integrating Multimodal Retrieval-augmented Generation (Multimodal RAG) offers a promising solution, the system would inevitably encounter the multi-granularity noisy correspondence (MNC) problem, which hinders accurate retrieval and generation. In this work, we propose RagVL, a novel framework with knowledge-enhanced reranking and noise-injected training, to address these limitations. We instruction-tune the MLLM with a simple yet effective instruction template to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. For generation, we inject visual noise during training at the data and token levels to enhance the generator's robustness. Extensive experiments on the subsets of two datasets that require retrieving and reasoning over images to answer a given query verify the effectiveness of our method. Code and models are available at https://github.com/IDEA-FinAI/RagVL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Decoupling macro topological routing from micro visual matching, plus query-driven GNN path decoding, improves multimodal multi-hop retrieval and QA over strong MM-RAG baselines.

  2. POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    POINTS-Seeker-8B is an 8B multimodal model trained from scratch for agentic search that uses seeding and visual-space history folding to outperform prior models on six visual reasoning benchmarks.

  3. VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A CoT-annotated dataset (VaccineRAG) plus segment-level GRPO (Partial-GRPO) improves multimodal large language models' ability to ignore harmful retrieved samples in retrieval-augmented generation tasks.

  4. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  5. MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

    cs.AI 2025-06 conditional novelty 6.0 of 10

    MM-R5, a 7B multimodal re-ranker trained with SFT and GRPO, achieves state-of-the-art page-level recall on MMDocIR by generating per-page reasoning chains.

  6. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  7. UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates

    cs.IR 2026-02 conditional novelty 5.0 of 10

    UniRank natively scores hybrid text-image candidates with a VLM, then SFT plus hard-negative RLHF adapts it to a domain; it outperforms baselines on scientific literature and design patent retrieval.

  8. Filter-And-Refine: A MLLM Based Cascade System for Industrial-Scale Video Content Moderation

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A cascade of an embedding router and a fine-tuned multimodal LLM ranker is claimed to improve content moderation F1 by 66.5% while using 1.5% of the compute of direct LLM deployment.

Pith tools