Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a three-part recipe—EVA-CLIP retrieval, listwise LVLM re-ranking, and feeding one document—governs zero-shot multimodal RAG for visual question answering.

desk verdict Solid mRAG design-space recipe, but the headline agentic gain rests on an unequal baseline and needs a matched ablation. read the letter →

arxiv 2505.24073 v2 pith:PZPCGIUD submitted 2025-05-29 cs.AI cs.CLcs.CV

Chan-Wei Hu , Yueqi Wang , Shuo Xing , Chia-Ju Chen , Suofei Feng , Ryan Rossi , Zhengzhong Tu This is my paper
classification cs.AIcs.CLcs.CV
keywords retrieval-augmentedgenerationmultimodalRAGvisualquestionansweringzero-shotre-rankingself-reflectionlargevision-languagemodelsEVA-CLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to settle which design choices actually matter in multimodal retrieval-augmented generation (mRAG) when the vision-language model is used frozen, with no fine-tuning. It dissects the pipeline into retrieval, re-ranking, and generation, and claims a concrete recipe: retrieve with EVA-CLIP, re-rank with listwise LVLM prompting, and feed only the single most relevant document to the generator. On the E-VQA and InfoSeek benchmarks this recipe raises response accuracy by +2.32 and +0.65 percentage points respectively. The paper further claims that a unified agent, which lets the same LVLM re-rank, draft, and self-verify answers against the retrieved set, adds about 5 percentage points on E-VQA and 2 on InfoSeek. If true, the practical consequence is that strong grounded visual question answering can be achieved by architecture choices and prompting alone, without task-specific training.

What carries the argument

Two mechanisms carry the argument. The first is listwise re-ranking: the re-ranker is prompted to view the entire retrieved candidate list and reorder it holistically, exploiting the LVLM's global relevance judgment and counteracting the lost-in-the-middle positional bias that makes generators overweight the first document. The second is the self-reflection loop of the unified agent: the LVLM evaluates whether a candidate document directly answers the query, generates a tentative answer, verifies that answer against the document's content, and either returns it, moves to the next document, or reports that it cannot answer. That loop is the component the paper credits for suppressing irrelevant context and yielding the additional accuracy gains.

What would settle it

Run the non-unified pipeline on the same top-K documents that the agent sees and let it pick the best answer by scoring each document once (best-of-K without reflection). If best-of-K equals or exceeds the unified agent's accuracy, then the self-reflection mechanism itself contributes nothing beyond a larger document budget.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is a divergence between retrieval accuracy and response accuracy: Recall@K keeps rising with K, but answer accuracy does not—and even drops—because LVLMs attend strongly to early context and are confused by irrelevant documents that arrive with the correct one. Consequently, the paper argues, the optimal generation input after re-ranking is exactly one document, not a larger list. It also finds that a large CLIP-style dual encoder (EVA-CLIP) is the strongest zero-shot retriever among six compared approaches; that listwise re-ranking with an LVLM outperforms pointwise, pairwise, and even a fine-tuned re-ranker in zero-shot use; and that a unified agentic loop of relevance judgment, tentative answering, and self-reflection outperforms the decoupled pipeline. The headline mechanism is that the model should be allowed to reject evidence dynamically and to decline to answer when no document supports a response.

Load-bearing premise

The claimed 5%/2% gain of the unified agent assumes that its self-reflection loop, rather than its access to the full top-K candidate set, is what improves accuracy, because the comparison baseline only receives the top-1 document after re-ranking.

Editorial extensions

If this is right

  • Adding more retrieved documents to a fixed-context LVLM is not free: answer accuracy can fall even when the correct article is inside the retrieved set, so mRAG systems should aim for a small, precisely re-ranked set rather than a large top-K.
  • A large-scale CLIP-style dual encoder can beat LVLM-based retrievers in zero-shot retrieval, so lightweight retrievers remain a strong default for multimodal RAG.
  • Listwise LVLM re-ranking can surpass a fine-tuned re-ranker in zero-shot use, meaning strong re-ranking is attainable without task-specific training.
  • A unified agent that can decline to answer when no document supports it outperforms decoupled retrieve-then-generate pipelines on knowledge-based VQA.
  • Augmenting the query image with a generated caption gives a modest Recall@1 gain, while captioning both query and knowledge-base sides hurts retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests its recipe on two single-hop encyclopedic VQA datasets; the same design choices may not transfer to multi-hop, temporal, or conversational mRAG, where a single document is more likely to be insufficient.
  • If the unified agent's gain is really a document-budget effect, a simpler non-agentic best-of-K scorer over the same top-K documents could recover most of the reported 5% improvement at lower inference cost; this is directly testable.
  • The observed divergence between retrieval accuracy and response accuracy suggests mRAG systems should adaptively decide how many documents to hand to the generator based on estimated relevance of the top candidates, rather than always using a fixed K.
  • Because adding irrelevant documents degrades generation, mRAG evaluations should report response accuracy conditioned on the utility of the retrieved evidence, not retrieval accuracy alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a systematic, zero-shot study of the multimodal retrieval-augmented generation (mRAG) pipeline for large vision-language models. It dissects three stages: retrieval (modality configurations and retrievers), re-ranking (pointwise, pairwise, listwise strategies), and generation (how many and which retrieved documents to provide). Based on experiments on distilled 50K-article knowledge bases from E-VQA and InfoSeek, it proposes a recipe consisting of EVA-CLIP retrieval, listwise LVLM-based re-ranking, and providing only the most relevant document for generation, reporting +2.32%/+0.65% response-accuracy gains. It then introduces a unified agentic framework that iteratively re-ranks and generates through self-reflection, claiming 5%/2% accuracy gains over a decoupled baseline. The paper also reports a lost-in-the-middle effect in generation and shows that more retrieved documents do not improve response accuracy.

Significance. If validated, the paper would provide useful design guidance for building zero-shot multimodal RAG systems, and its component-level findings align with prior evidence on CLIP-style retrieval, LVLM-based listwise re-ranking, and positional bias in long contexts. The empirical coverage is broad: six retrievers, four modality configurations, three re-ranking families, two generation models, and multiple evaluation metrics. However, the headline agentic gain rests on a confounded comparison, and the abstract's aggregate '5%' claim is not derivable from the reported tables. The recipe itself is selected and evaluated on the same benchmarks, making the advertised improvements in-sample descriptions rather than out-of-sample predictions. With matched baselines and tempered claims, the paper's component studies would remain a valuable contribution.

major comments (4)
  1. [Section 6.1; Tables 5-6; Appendix B.4] The comparison underlying the headline agentic gain is confounded. The non-Unified baseline is defined as response accuracy with the single top-1 document given after re-ranking from Section 5, while the Unified agent iterates over the full top-K retrieved set and moves to the next document when the self-reflection check fails (Figures 10-11). Tables 5 and 6 therefore compare top-1-with-single-pass-generation against top-K-with-sequential-access, so the observed 5%/2% gain could be explained entirely by the extra document budget and by single-document-at-a-time presentation, independent of self-reflection. The Limitations section does not acknowledge this asymmetry. Please add an ablation that removes the self-reflection validation but retains sequential access to top-K, or a best-of-K baseline that scores K single-document generations, before attributing the gain to iterative self-reflection.
  2. [Abstract; Section 6.2 Takeaway] The abstract's claim of an 'average performance boost of 5%' is not supported by the tables. From Tables 5 and 6, the GPT-4.1 response-accuracy improvements are 3.89, 1.90, 5.25, and 2.58 percentage points for Qwen-E-VQA, Qwen-InfoSeek, LLaVA-E-VQA, and LLaVA-InfoSeek; their average is 3.4 points, and the InternVL3 judge gives smaller gains of 1.84, 1.30, 2.32, and 1.04 points. No reported aggregation yields a 5% average boost; the only 5+ value is the single E-VQA LLaVA/GPT cell. Please state the exact aggregation rule used for the abstract's headline or revise it to match Tables 5, 6, 8, and 9.
  3. [Tables 8-9; Abstract Recipe] Some of the central recipe differences are small enough that sampling noise is a serious concern. For example, the +0.65% InfoSeek gain mentioned in the abstract corresponds to roughly 33 answers in a 5,000-case evaluation, and several re-ranking and generation gains in Tables 8-9 are around 1-2 percentage points. No confidence intervals, bootstrap estimates, or significance tests are reported, and the test subsets are fixed rather than resampled. Without uncertainty quantification, the claim that the proposed recipe 'yields' these gains is stronger than the evidence supports. Please add confidence intervals or significance tests, or soften the causal framing.
  4. [Abstract Recipe; Sections 3-6] The recipe is selected by evaluating components on the same E-VQA and InfoSeek subsets that are then used to report the recipe's final gain: EVA-CLIP is chosen because it maximizes Recall@5 in Table 2, listwise re-ranking because it maximizes Recall@1 in Table 4, and top-1 generation because it maximizes response accuracy in Figure 2, and the advertised +2.32%/+0.65% is then measured on those same data. This makes the headline an in-sample fit rather than an out-of-sample prediction. The paper should either validate the recipe on additional benchmarks or explicitly frame the contribution as a retrospective design-space analysis rather than a prescriptive, validated recipe.
minor comments (4)
  1. [Section 2.1; Appendix A.1] The distilled knowledge-base construction is not fully specified: please state whether the gold article for each test question is always included in the 50K subset, and describe the category-preserving sampling procedure, since the retrieval Recall numbers in Tables 2-3 depend directly on this design choice.
  2. [Section 3.3; Table 2] The caption-augmented configurations (IC, C) are motivated by a two-stage evaluation, but Table 3 reports caption results only for EVA-CLIP. Reporting the same expansion stage for at least one other retriever would clarify whether the caption finding is retriever-specific.
  3. [Throughout] There are several typographical issues: 'EV A-CLIP' spacing in Table 2, 'becasue' in Section 4.2, 'LLaV Anon-Unified' in the Table 5 header, and 'hallucainated' in the Limitations section. Please proofread.
  4. [Figure 2] The dual-axis figure with blue and red lines, bar plots, and two dashed reference lines is difficult to parse; please add numerical annotations or make the connection to Tables 8 and 9 explicit in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the recipe and agentic gains are empirical, confounded claims rather than reductions to the paper's own inputs.

full rationale

The paper does not derive its headline gains by construction. Every component (EVA-CLIP retriever, listwise re-ranking, top-1 generation, self-reflection agent) is evaluated on the external E-VQA and InfoSeek benchmarks with frozen, zero-shot models; no parameter is fitted to the benchmark and then renamed as a prediction. The recipe is admittedly assembled from results on the same benchmarks, which makes the +2.32%/+0.65% numbers in-sample model-selection estimates rather than out-of-sample validations, but that is an overfitting/statistical concern, not a logical reduction of the claimed result to its inputs. The unified-agent 5%/2% comparison is confounded: Section 6.1 defines the non-Unified baseline as top-1 after re-ranking while the Unified agent iterates over the full retrieved set, so attributing the gain to self-reflection is not isolated from document budget. This is an experimental-control flaw and an omitted matched ablation, not circularity. There is no load-bearing self-citation: the only overlapping-author citation (Zhu et al. 2024, which includes Zhengzhong Tu) supports a general background claim about cross-modality conflicts and is not used to justify the recipe or to forbid alternative designs. No uniqueness theorem or ansatz is imported from the authors' prior work. The Limitations section flags distillation, zero-shot evaluation, and judge alignment but does not acknowledge the top-K/top-1 asymmetry; that omission strengthens the confound concern but does not make the derivation circular. Under the hard rule that circularity requires a specific reduction by construction, the appropriate finding is no circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is empirical; no new physical or mathematical entities are introduced. Its claims rest on the reliability of the benchmarks, judges, and the distilled KB, plus the chosen hyperparameters (K, KB size).

free parameters (3)
  • K (number of retrieved candidates) = 5 (with variants 1 and 10 in retrieval-only results)
    The generation and reranking experiments use top-5; the recipe's 'top-1' finding is conditional on this K. Choosing a different K could change the optimal document-feeding strategy.
  • Distilled knowledge base size = 50,000 articles
    Chosen for computational feasibility; all evaluation queries are answerable within this subset, which may not reflect full-scale KB behavior.
  • Test set sizes = 4,750 (E-VQA), 5,000 (InfoSeek)
    Sample sizes adopted from prior work; small differences in accuracy are reported without variance estimates.
assumptions (4)
  • domain assumption The reference answers in E-VQA and InfoSeek are correct and the 'gold document' contains the answer.
    Evaluation assumes gold-document upper bound is achievable; if some gold annotations are wrong, accuracy estimates are biased.
  • domain assumption LVLM judges (InternVL3-14B and GPT-4.1) reliably assess semantic correctness of answers.
    All response accuracy numbers rely on automated judges; the authors list possible misalignment with human judgment in Limitations.
  • domain assumption The distilled 50k-article KB preserves the category distribution and answerability of the original KBs.
    Retrieval and generation numbers are computed on this subset; the authors note this may introduce distributional biases (Limitations).
  • domain assumption The 'lost in the middle' positional bias observed in text LLMs transfers to LVLMs.
    The reranking and top-1 generation findings are motivated by this prior result (Liu et al. 2024).

how reviews work

0 comments
Cite this review

Pith. "Pith review of mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/PZPCGIUD

@misc{pith2026250524073,
  author       = {Pith},
  title        = {Pith review of: mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZPCGIUD}},
  note         = {Machine review of arXiv:2505.24073}
}
read the original abstract

Large Vision-Language Models (LVLMs) have made remarkable strides in multimodal tasks such as visual question answering, visual grounding, and complex reasoning. However, they remain limited by static training data, susceptibility to hallucinations, and inability to verify claims against up-to-date, external evidence, compromising their performance in dynamic real-world applications. Retrieval-Augmented Generation (RAG) offers a practical solution to mitigate these challenges by allowing the LVLMs to access large-scale knowledge databases via retrieval mechanisms, thereby grounding model outputs in factual, contextually relevant information. Here in this paper, we conduct the first systematic dissection of the multimodal RAG pipeline for LVLMs, explicitly investigating (1) the retrieval phase: on the modality configurations and retrieval strategies, (2) the re-ranking stage: on strategies to mitigate positional biases and improve the relevance of retrieved evidence, and (3) the generation phase: we further investigate how to best integrate retrieved candidates into the final generation process. Finally, we extend to explore a unified agentic framework that integrates re-ranking and generation through self-reflection, enabling LVLMs to select relevant evidence and suppress irrelevant context dynamically. Our full-stack exploration of RAG for LVLMs yields substantial insights, resulting in an average performance boost of 5% without any fine-tuning.

Figures

Figures reproduced from arXiv: 2505.24073 by the authors.

Figure 1
Figure 1. The multi-modal RAG (mRAG) pipeline utilized in our journey to exploit the design space of each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Generation performance of Qwen2-VL-7B-Instruct (top) and LLaVA-OneVision (bottom) across different [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The self-reflection process of unifying re-ranking and generation in a single agentic framework. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Image captioning prompt on the query side. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Image captioning prompt on the knowledge [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Pairwise re-ranking prompt. for model generation and automated judge, respec￾tively. B.4 Unifying re-ranking and generation [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 8
Figure 8. Figure 8: Generation prompt [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 7
Figure 7. Figure 7: Listwise re-ranking prompt. C Experimental Costs The cost for the response evaluation using GPT 4.1 cost approximately $20 in total. D Licenses The datasets we used, InfoSeek and E-VQA, are licensed under Apache License 2.0 and CC BY 4.0, respectively. The retrieval mo…
Figure 10
Figure 10. Figure 10: The evaluation prompt to assess the relation [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The self-reflection prompt to verify the [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Multimodal KB-VQA exhibits a primacy bias where gold passages at prompt start outperform those at the end by 16-26 points, flipping the text-only lost-in-the-middle pattern.

  2. Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Post-hoc selective modality escalation—draft from text+tables, verify the missing modality, then escalate only when predicted accuracy gain exceeds visual cost—matches always-on VLM accuracy on MultiModalQA with far f...

  3. MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    cs.IR 2026-04 unverdicted novelty 6.0 of 10

    MG²-RAG proposes a multi-granularity graph RAG framework that constructs hierarchical multimodal nodes via entity-driven visual grounding and performs structured retrieval, delivering SOTA results on four multimodal t...

  4. Demystifying the Visual Quality Paradox in Multimodal Large Language Models

    cs.CV 2025-06 reject novelty 4.0 of 10

    Multimodal LLM accuracy can improve on visually degraded images, and a lightweight test-time tuning module that modulates input quality yields small accuracy gains on some benchmarks.

Reference graph

Works this paper leans on

71 extracted references · 22 canonical work pages · cited by 4 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. 2025. Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation. arXiv preprint arXiv:2502.08826

    arXiv 2025

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    I \ n igo Alonso, Ander Salaberria, Gorka Azkune, Jeremy Barnes, and Oier Lopez de Lacalle. 2025. Vision-language models struggle to align entities across modalities. arXiv preprint arXiv:2503.03854

    arXiv 2025

  6. [6]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  7. [7]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

    arXiv 2024

  8. [8]

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818--1826

    2024

Show all 71 references
  1. [9]

    Recep Firat Cekinel, Pinar Karagoz, and C a g r C \"o ltekin. 2025. https://aclanthology.org/2025.coling-main.310/ Multimodal fact-checking with vision language models: A probing classifier based solution with embedding strategies . In Proceedings of the 31st International Con...

    2025

  2. [10]

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024 a . Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1...

    2024

  3. [11]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024 b . Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    2024

  4. [12]

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.925 Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference...

    2023 doi

  5. [13]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 c . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Com...

    2024

  6. [14]

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. 2025. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411

    2025 arXiv

  7. [15]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar \'e , Maria Lomeli, Lucas Hosseini, and Herv \'e J \'e gou. 2024. The faiss library. arXiv preprint arXiv:2401.08281

    2024 arXiv

  8. [16]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on mult...

    2024

  9. [17]

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer V...

    2024

  10. [18]

    Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.491 FIRST : Faster improved listwise reranking with single token decoding . In Proceedings of the 2024 Conference on Empirica...

    2024 doi

  11. [19]

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462--12469. IEEE

    2024

  12. [20]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2:1

    2023 arXiv

  13. [21]

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580

    2024 arXiv

  14. [22]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information proc...

    2020

  15. [23]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  16. [24]

    Boyi Li, Philipp Wu, Pieter Abbeel, and Jitendra Malik. 2023. Interactive task planning with language models. arXiv preprint arXiv:2310.10645

    2023 arXiv

  17. [25]

    Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. 2024. Can feedback enhance semantic grounding in large vision-language models? arXiv preprint arXiv:2404.06510

    2024 arXiv

  18. [26]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  19. [27]

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. 2024. Mm-embed: Universal multimodal retrieval with multimodal llms. arXiv preprint arXiv:2411.02571

    2024 arXiv

  20. [28]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

    2024 doi

  21. [29]

    Qi Liu, Haozhe Duan, Yiqun Chen, Quanfeng Lu, Weiwei Sun, and Jiaxin Mao. 2025. Llm4ranking: An easy-to-use framework of utilizing large language models for document reranking. arXiv preprint arXiv:2504.07439

    2025 arXiv

  22. [30]

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797

    2024 arXiv

  23. [31]

    Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.710 Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In Findings of the Association for Computational Linguistics: EMNLP ...

    2023 doi

  24. [32]

    Lang Mei, Siyu Mo, Zhihan Yang, and Chong Chen. 2025. A survey of multimodal retrieval-augmented generation. arXiv preprint arXiv:2504.08748

    2025 arXiv

  25. [33]

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr \'e Araujo, and Vittorio Ferrari. 2023. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International...

    2023

  26. [34]

    Matin Mortaheb, Mohammad A Amir Khojastepour, Srimat T Chakradhar, and Sennur Ulukus. 2025. Re-ranking the context for multimodal retrieval augmented generation. arXiv preprint arXiv:2501.04695

    2025 arXiv

  27. [35]

    Viraj Prabhu, Senthil Purushwalkam, An Yan, Caiming Xiong, and Ran Xu. 2024. Trust but verify: Programmatic vlm evaluation in the wild. arXiv preprint arXiv:2410.13121

    2024 arXiv

  28. [36]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. 2023. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563

    2023 arXiv

  29. [37]

    Vipula Rawte, Aryan Mishra, Amit Sheth, and Amitava Das. 2025. Defining and quantifying visual hallucinations in vision-language models. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 501--510

    2025

  30. [38]

    Ruiyang Ren, Yuhao Wang, Kun Zhou, Wayne Xin Zhao, Wenjie Wang, Jing Liu, Ji-Rong Wen, and Tat-Seng Chua. 2025. Self-calibrated listwise reranking with large language models. In Proceedings of the ACM on Web Conference 2025, pages 3692--3701

    2025

  31. [39]

    Monica Riedler and Stefan Langer. 2024. Beyond text: Optimizing rag with multimodal inputs for industrial applications. arXiv preprint arXiv:2410.21943

    2024 arXiv

  32. [40]

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. A comprehensive survey of hallucination in large language, image, video and audio foundation models. arXiv preprint arXiv:2405.09589

    2024 arXiv

  33. [41]

    Pritish Sahu, Karan Sikka, and Ajay Divakaran. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.470 Pelican: Correcting hallucination in vision- LLM s via claim decomposition and program of thought verification . In Proceedings of the 2024 Conference on Empirical Methods in N...

    2024 doi

  34. [42]

    Neelabh Sinha, Vinija Jain, and Aman Chadha. 2024. Guiding vision-language model selection for visual question-answering across tasks, domains, and knowledge types. arXiv preprint arXiv:2409.09269

    2024 arXiv

  35. [43]

    Zhang Siyue, Xue Yuxiang, Zhang Yiming, Wu Xiaobao, Luu Anh Tuan, and Zhao Chen. 2024. Mrag: A modular retrieval framework for time-sensitive question answering. arXiv preprint arXiv:2412.15540

    2024 arXiv

  36. [44]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389

    2023 arXiv

  37. [45]

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. 2024. Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252

    2024 arXiv

  38. [46]

    Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven R Corman, and Chitta Baral. 2024. Unseentimeqa: Time-sensitive question-answering beyond llms' memorization. arXiv preprint arXiv:2407.03525

    2024 arXiv

  39. [47]

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. 2021. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12884--12893

    2021

  40. [48]

    David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. 2024. Contrastive region guidance: Improving grounding in vision-language models without training. In European Conference on Computer Vision, pages 198--215. Springer

    2024

  41. [49]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

    2024 arXiv

  42. [50]

    Shijie Wang, Dahun Kim, Ali Taalimi, Chen Sun, and Weicheng Kuo. 2025. Learning visual grounding from generative vision and language model. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 8057--8067. IEEE

    2025

  43. [51]

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387--404. Springer

    2024

  44. [52]

    Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. 2020. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575--2584

    2020

  45. [53]

    Feifan Wu, Lingyuan Liu, Wentao He, Ziqi Liu, Zhiqiang Zhang, Haofen Wang, and Meng Wang. 2024. Time-sensitve retrieval-augmented generation for question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 2544--2553

    2024

  46. [54]

    Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024 a . Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085

    2024 arXiv

  47. [55]

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. 2024 b . Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages...

    2024

  48. [56]

    Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2024. Vlm-grounder: A vlm agent for zero-shot 3d visual grounding. In CoRL

    2024

  49. [57]

    Yibin Yan and Weidi Xie. 2024. https://aclanthology.org/2024.findings-emnlp.83 E cho S ight: Advancing visual-language models with W iki knowledge . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1538--1551, Miami, Florida, USA. Association for...

    2024

  50. [58]

    Zhutian Yang, Caelan Garrett, Dieter Fox, Tom \'a s Lozano-P \'e rez, and Leslie Pack Kaelbling. 2024. Guiding long-horizon task and motion planning with vision language models. arXiv preprint arXiv:2410.02193

    2024 arXiv

  51. [59]

    Ziyan Yang, Kushal Kafle, Franck Dernoncourt, and Vicente Ordonez. 2023. Improving visual grounding by encouraging consistent gradient-based explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19165--19174

    2023

  52. [60]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  53. [61]

    Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. Advances in Neural Information Processing Systems, 37:121156--121184

    2024

  54. [62]

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. 2024. Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model. arXiv preprint arXiv:2402.10828

    2024

  55. [63]

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024 a . Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  56. [64]

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. 2024 b . Magiclens: Self-supervised image retrieval with open-ended instructions. In The Forty-first International Conference on Machine Learning (ICML)

    2024

  57. [65]

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2024 c . Gme: Improving universal multimodal retrieval by multimodal llms. arXiv preprint arXiv:2412.16855

    2024 arXiv

  58. [66]

    Zhaxizhuoma Zhaxizhuoma, Pengan Chen, Ziniu Wu, Jiawei Sun, Dong Wang, Peng Zhou, Nieqing Cao, Yan Ding, Bin Zhao, and Xuelong Li. 2024. Alignbot: Aligning vlm-powered customized task planning with user reminders through fine-tuning for household robots. arXiv preprint arXiv:2...

    2024 arXiv

  59. [67]

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. 2024. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14475

    2024 arXiv

  60. [68]

    Weijie Zhou, Manli Tao, Chaoyang Zhao, Haiyun Guo, Honghui Dong, Ming Tang, and Jinqiao Wang. 2025. Physvlm: Enabling visual language models to understand robotic physical reachability. arXiv preprint arXiv:2503.08481

    2025 arXiv

  61. [69]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479

    2025 arXiv

  62. [70]

    Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. 2024. Unraveling cross-modality knowledge conflicts in large vision-language models. arXiv preprint arXiv:2410.03659

    2024 arXiv

  63. [71]

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.