Pith. sign in

REVIEW 5 cited by

Retrieval Augmented Visual Question Answering with Outside Knowledge

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.03809 v2 pith:PULEPN2M submitted 2022-10-07 cs.CL

Weizhe Lin , Bill Byrne This is my paper
classification cs.CL
keywords retrievalanswergenerationknowledgeok-vqatrainingansweringdocuments
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.

  2. Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Controllable retrieval-difficulty curriculum plus reward-propagation sampling lets RL close the pretrain-to-KB-VQA gap and beat prior SOTA on two hard encyclopedic VQA benchmarks.

  3. KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    KG-ViP fuses scene graphs and commonsense graphs via a query-based retrieval-and-fusion pipeline to improve multi-modal LLM performance on visual question answering.

  4. MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.

  5. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

Pith tools