Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces AVHaystacksQA, a task where answering requires retrieving and linking audio-visual evidence across up to 500 videos; it contributes the 3100-pair AVHaystacks benchmark and MAGNET, a multi-agent framework that lifts…

desk verdict This is a benchmark paper worth taking seriously, but the paper's signature claim—that every QA pair genuinely needs both audio and video—is asserted, not demonstrated. read the letter →

arxiv 2506.07016 v2 pith:OCNKSYC3 submitted 2025-06-08 cs.CV cs.AI

Sanjoy Chowdhury , Mohamed Elmoghany , Yohan Abeysinghe , Junjie Fei , Sayan Nag , Salman Khan , Mohamed Elhoseiny , Dinesh Manocha This is my paper
classification cs.CVcs.AI
keywords audio-visualquestionansweringmulti-videoretrievaltemporalgroundingretrieval-augmentedgenerationmulti-agentframeworkvideobenchmarkmultimodalLLMevaluationneedle-in-haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that real-world video question answering is not a single-clip task: queries about how-to collections, language lessons, and video archives often require pulling evidence from many videos at once, using both sound and picture. To study this, it introduces AVHaystacksQA and a new benchmark, AVHaystacks, made of 3100 question-answer pairs over a pool of 500 videos, where most answers must cite at least two distinct clips. It then proposes MAGNET, a model-agnostic pipeline that retrieves candidate videos, spawns one audio-visual agent per video to locate the salient segments, and lets a meta-agent stitch those segments into a grounded, step-by-step answer. The paper reports that MAGNET improves BLEU@4 by up to 89% and GPT-judge scores by up to 65% over baseline systems, and that two new metrics, STEM and MTGS, expose step-order and temporal-grounding errors that end-to-end scores hide. If the paper is right, video QA should be reframed as a retrieval and multi-hop reasoning problem rather than a single-clip comprehension problem.

What carries the argument

MAGNET is the carrying mechanism: a retrieval-augmented, multi-agent pipeline. Its load-bearing parts are, first, AV-RAG, a two-branch retriever that scores each video by cosine similarity between the query text embedding and both the fused audio-visual embedding from ImageBind and the embedding of a Gemini-generated caption; second, the Salient Frame Selector (SFS), a dynamic-programming routine that picks k representative frames from m sampled candidates by minimizing summed pairwise affinity, where affinity combines visual similarity with a sine-based temporal separation penalty so the chosen frames are both visually diverse and spread across time; and third, the agent layer, in which one Qwen 2.5 Omni instance per shortlisted video emits its candidate segments and partial answers, and a GPT-4o meta-agent aggregates them into the final step-wise answer. The paper also contributes two evaluation objects: STEM, a step-wise error metric that uses Hungarian matching to count missing, hallucinated, and out-of-order steps along with video-ID mismatches, and MTGS, the mean temporal IoU over video IDs that appear in both prediction and ground truth.

What would settle it

Give a strong text-only LLM the cleaned transcripts (and, optionally, auto-generated captions) of the top-ranked videos for each AVHaystacks question, with no frames and no audio, and score its answers with the paper's own BLEU@4, GPT-judge, and STEM protocol. If transcript-only answers match or exceed MAGNET's scores on a significant sample, the benchmark is answerable from text and the reported audio-visual gains are an artifact of retrieval rather than of multimodal reasoning. A cheaper check is to have annotators judge whether a random sample of 200 ground-truth answers is derivable from the transcript alone.

Watch

Extended reading notes

Core claim

The central claim is that current audio-visual models are poor at multi-video retrieval and reasoning, and that a pipeline built from off-the-shelf components can close much of the gap. Concretely, the authors claim three things: that AVHaystacks is the first benchmark whose answers require linking evidence across multiple videos, with 82% of pairs citing at least two videos from a pool of up to 500; that state-of-the-art LMMs score poorly on it, both at retrieving the right videos and at grounding each answer step in the right time interval; and that MAGNET, a retrieval stage using ImageBind embeddings of fused audio-visual features and of Gemini-generated captions, a salient-frame selector, per-video Qwen 2.5 Omni agents, and a GPT-4o aggregator, raises retrieval recall and answer quality substantially, achieving up to 89% and 65% relative improvements over baselines on BLEU@4 and GPT evaluation scores while nearly matching a Gemini 1.5 Pro upper bound placed inside the same framework.

Load-bearing premise

The load-bearing premise is that every AVHaystacks question genuinely needs both audio and visual information and cannot be answered from transcripts or captions alone; the curation pipeline asserts this via LLM prompting and human filtering, and although an ablation shows MAGNET does best with both modalities, the paper never measures how transcript-only or caption-only systems fare on the benchmark.

Editorial extensions

If this is right

  • Multi-video audio-visual QA becomes a testable benchmark task, so future LMMs can be measured on retrieval-plus-reasoning rather than single-clip comprehension.
  • Large gains are available without waiting for end-to-end training: the paper's ablation shows that combining off-the-shelf retrieval, per-video agents, and a meta-agent already lifts answer quality, and fine-tuning the per-video agent pushes an open model close to the closed-source upper bound.
  • STEM and MTGS give evaluators a way to attribute failures to specific causes: steps that are missing, hallucinated, or out of order, and segments that are mis-grounded in time or in which video they come from.
  • The modality ablation supports the paper's premise that both channels carry complementary evidence, with audio-plus-visual inputs outperforming either modality alone across every reported metric.
  • Retrieval behavior matters to answer quality: the top-k and frame-selection ablations show that giving agents more candidate videos and denser frames improves grounding until it plateaus, after which extra input mostly adds compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decisive test the paper leaves unrun is a transcript-only probe: feed cleaned subtitles, with no frames and no audio, to a strong text LLM under the same retrieval ranking and score with the same metrics; if it approaches MAGNET's scores, the benchmark is measuring text reasoning, not audio-visual reasoning.
  • Because every MAGNET component is model-agnostic, the same retrieve-then-agents-then-meta-agent pattern should transfer to other retrieval-heavy multimodal settings, such as document or image haystacks; the authors list end-to-end trainable modules as future work but do not discuss this transfer.
  • The ablations on top-k and frame count imply retrieval budget is a tunable policy variable, so plotting answer quality against compute per query would give practitioners an explicit cost-quality curve the paper does not draw.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces AVHaystacksQA, a multi-video audio-visual question-answering task in which a model must retrieve relevant videos from a large corpus, localize salient temporal segments, and synthesize a grounded step-wise answer. To support this task, the authors present AVHaystacks, a benchmark of roughly 3100 annotated QA pairs built from 500 YouTube videos across 27 topics, and MAGNET, a retrieval-augmented multi-agent framework that combines ImageBind-based audio-visual retrieval, a salient frame selection module (SFS), per-video AVLLM agents (Qwen 2.5 Omni, VideoSALMONN, Unified IO2), and a GPT-4o meta-agent for final answer synthesis. The paper also proposes two evaluation metrics: STEM, a step-wise error metric based on Hungarian matching, and MTGS, a matched temporal grounding score. Experiments compare MAGNET with several baselines on AVHaystacks-50 and AVHaystacks-Full, and ablations study modality contribution, frame sampling, top-k selection, meta-agent choice, and threshold sensitivity.

Significance. If the benchmark's central premise is valid, AVHaystacks fills a genuine gap: no existing benchmark systematically evaluates multi-video audio-visual retrieval and reasoning with temporal grounding, and MAGNET is a practical, model-agnostic pipeline that shows consistent gains across its own ablations, including a human evaluation component. The benchmark construction effort is substantial, and the STEM and MTGS metrics address an under-evaluated aspect of grounded answer generation. However, the significance is conditional: the paper never verifies that the QA pairs actually require both audio and visual information, and the headline gains are reported against zero-shot baselines while MAGNET is fine-tuned, so the claimed improvements cannot currently be attributed to the framework rather than to in-distribution training.

major comments (5)
  1. [§2, Supp. F.2–F.3] The central premise of AVHaystacks—that each QA pair 'genuinely requires both audio and visual understanding' and cannot be answered from text alone—is asserted but never tested. The only filtering evidence is an LLM prompt (Supp. Listing 1: 'must be answer by a video, audio and can not be answered by text only?') plus human inspection, while the construction pipeline is transcript-centric: captions are cleaned and segmented (§2, F.2–F.4), segment-aware questions are generated from transcript segments (Listing 2), and ground-truth answers are step-wise texts with video/timestamp references (Listing 5). No experiment feeds only transcripts, only audio, or only video to any baseline or to MAGNET. If a transcript-only model reproduces most of the reported BLEU/GPT-eval/STEM/MTGS scores, then AVHaystacksQA mostly measures text-based retrieval and summarization over captions, not audio-visual reasoning, and the claimed 89%/65% relative improvements would not support the task as defined. Please add explicit modality-ablation baselines—transcript-only, audio-only, video-only, and full—and report all metrics for each.
  2. [§4.2, Tables 2–3, Supp. I] The central comparison is confounded: fine-tuned MAGNET variants are compared to zero-shot baselines. MAGNET+Qwen-2.5-Omni-FT is LoRA fine-tuned on AVHaystacks (Supp. I), while VideoRAG, Video-RAG, Qwen2.5 Omni, Unified IO2, and VideoSALMONN are evaluated without any fine-tuning. The large relative gains could therefore reflect exposure to the training distribution rather than the multi-agent retrieval architecture. To make the claim that MAGNET 'significantly enhances' performance, the authors should either fine-tune all baselines with the same LoRA recipe or restrict the headline claims to comparisons with equally trained systems.
  3. [Abstract, §4.3] The headline numbers 'up to 89% and 65% relative improvements' are not reproducible from Table 2. On AVHaystacks-50, the best BLEU@4 is 57.67 (MAGNET+Gemini 1.5 Pro) versus 43.16 for VideoRAG, which is a 33.6% relative gain; the best GPT Eval is 8.03 versus 6.32, a 27% relative gain. The corresponding Full-split gains are 34.2% and 27.3%. Please state which baseline and which split yield the 89% and 65% figures, or correct the numbers in the abstract and Section 4.3.
  4. [§4.1, §3.3] The GPT-as-a-judge evaluation is potentially biased because the judge model (GPT) is from the same model family as the meta-agent (GPT-4o, Section 3.3). This creates a same-family write-and-grade loop in which stylistic similarity to GPT-generated text may inflate scores independently of answer correctness. Given that GPT Eval is one of the two headline metrics, the paper should report correlation between GPT Eval and human evaluation, and ideally also include a judge from a different model family (e.g., Gemini or Claude) to verify that the conclusions are not judge-family artifacts.
  5. [§4.1, Table 4] The retrieval evaluation is incomplete and underspecified: Section 4.1 defines R@1, R@3, and R@5, but Table 4 reports only R@3 and R@5, and the row labeled 'Ours' does not specify which MAGNET configuration was used (backbone, top-k, SFS parameters). Please include R@1 and explicitly state the configuration and, if different settings were tried, how the reported point was selected.
minor comments (5)
  1. [§4.4] The text says 'switching from Uniform to SFS increases BLEU@4 score by 0.17', but the corresponding Table 6 values (36.58 to 53.61 for Qwen-2.5-Omni-FT) show an increase of about 17 points, not 0.17. Please correct the decimal error.
  2. [Figure 2] The caption refers to 'Tab. 2' for the improvement from the salient frame selection module, but the SFS ablation appears in Table 6. The citation should be updated.
  3. [Abstract, §2, Supp. Table 7] The abstract and Section 2 state '3100' QA pairs, while the supplementary data (Table 7) sums to 3147 questions. These numbers should be reconciled.
  4. [References] References [8] and [97] are the same work (VideoRAG) and are cited separately; the duplication should be removed. Reference [24] cites a blog URL rather than the official Gemini model documentation or paper.
  5. [§5] The related work section uses 'NeXT-QA' while Table 1 and the reference list use 'NExT-QA'; please standardize the spelling.

Circularity Check

1 steps flagged · score 4.0 of 10

The main circularity is a same-family write-and-grade loop: GPT-4o serves as the meta-agent that writes final answers and a GPT judge scores them, so the headline GPT Eval gains are not independent; the rest of the framework is empirical rather than derivationally circular.

  1. other [Section 3.3 and Section 4.1]
    "In our setup, GPT-4o[26] acts as the meta-agent, which ingests the agent responses and synthesises a coherent, contextually grounded final answer for the input query. ... we adopt the GPT-as-a-Judge framework to score the predicted responses against the ground truth on a 10-point scale, subsequently normalizing these scores for consistency."

    The reported GPT Eval metric is produced by a GPT-family judge for answers that were synthesized by GPT-4o as the meta-agent in the very same pipeline. Therefore the GPT Eval column measures how well GPT-generated output matches the stylistic and substantive expectations of a GPT grader, not an independent assessment of answer quality. The claimed relative improvements on GPT Eval, such as the 65% gain, are entangled with same-family grader preference and cannot serve as external validation of the framework.

full rationale

The paper's central contributions are empirical: a new benchmark, a retrieval-augmented multi-agent framework, and two new metrics. STEM and MTGS are formal definitions with an internal human-validation check, and the QA/retrieval results are compared against external baselines like VideoRAG and Video-RAG, so the framework's behavior does not reduce by construction to its inputs. The explicit circular issue is confined to the GPT-based evaluation: GPT-4o writes the final answers as the meta-agent and a GPT judge scores them, making the GPT Eval scores a same-family write-and-grade loop that partially undermines the headline GPT Eval improvements. The stronger concern about the benchmark's audio-visual premise, namely that the requirement that questions 'must be answer by a video, audio and can not be answered by text only' is asserted via an LLM prompt and human inspection rather than demonstrated by a transcript-only control, is a missing-control validity risk and not a circular derivation, so it does not raise the score further. No load-bearing self-citation chain or uniqueness-theorem import is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the benchmark's construct validity (that its questions require audio-visual reasoning) and on several pretrained-model assumptions. The free parameters are tuned on the benchmark itself. No new physical or theoretical entities are introduced.

free parameters (4)
  • gamma (SFS separation penalty) = 20
    Separation penalty factor in the Salient Frame Selector; tuned across 5, 10, 15, 20, 25 on AVHaystacks-Full (Fig. 3) and the best value is used for all reported results.
  • top-k retrieved videos = 6
    Number of videos passed to the agents; chosen by ablation on AVHaystacks-Full (Tab. 8).
  • m (number of sampled frames) = 75
    Number of uniformly sampled candidate frames in SFS; chosen by ablation (Tab. 11).
  • tau_s (text similarity threshold in STEM) = 0.5
    Cosine similarity threshold for step matching in Algorithm 2; set to 0.5 with limited sensitivity analysis reported.
assumptions (5)
  • domain assumption Each QA pair in AVHaystacks requires both audio and visual information and cannot be answered from transcripts or captions alone.
    This is the central validity premise of the benchmark; asserted from LLM prompt design and human filtering in Section 2, but never tested directly.
  • domain assumption ImageBind embeddings align text, audio, and visual content well enough for cross-video retrieval in this 500-video corpus.
    Used in Eq. 1 for AV-RAG; the success of retrieval depends on this pretrained model's alignment.
  • domain assumption Gemini 1.5 Pro generated captions are accurate and serve as reliable retrieval evidence.
    Captions are encoded by ImageBind to form E_cap in Eq. 1; errors would propagate to retrieval.
  • domain assumption GPT-as-a-Judge scores correlate with answer quality for this task.
    GPT Eval is a headline metric in Section 4.1; the judge is not shown to agree with human scores beyond an unreported correlation.
  • ad hoc to paper Hungarian matching with cosine threshold 0.5 correctly identifies step correspondences in STEM.
    Algorithm 2 sets tau_s=0.5; the choice is not derived and may affect SM, SH, SO counts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks." pith.science (2026). https://pith.science/paper/OCNKSYC3

@misc{pith2026250607016,
  author       = {Pith},
  title        = {Pith review of: MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCNKSYC3}},
  note         = {Machine review of arXiv:2506.07016}
}
read the original abstract

Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance. Project: https://schowdhury671.github.io/magnet_project/

Figures

Figures reproduced from arXiv: 2506.07016 by the authors.

Figure 1
Figure 1. A new task and a benchmark. We introduce a novel task AVHaystacksQA, which requires multi-video linkage and reasoning to generate the most informative answer. To this end, we curate a new benchmark AVHaystacks comprising 3100 QA pairs obtained through careful inspection. demand searching through hundreds or thousands of video segments, identifying the most relevant audio-visual snippets, and synthesizing coherent, e… view at source ↗
Figure 2
Figure 2. Overview of MAGNET. Given a user query, AV-RAG retrieves the top-K relevant videos (with audio), which are then processed by dynamically spawned Audio-Visual agents and a meta-agent aggregator to generate temporally grounded, step-wise responses. An adaptive, modality-agnostic frame selection module improves performance over baselines (see Tab. 2). Multiple Audio-Visual Entity Linkage. To enable fine-grained, cross-… view at source ↗
Figure 3
Figure 3. Effect of γ on eval metrics for MAGNET +Qwen-2.5-Omni-FT and MAGNET +Gemini-1.5-Pro. and Human Eval score by 1.03. A similar trend is observed with Gemini 1.5 Pro where the best results are obtained with our proposed sampling strategy as seen in Tab. 6. These results underscore the advantage of semantically guided sampling over uniform strategies, as SFS more effectively captures informative segments, leading to bet… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Qualitative results. Powered by efficient video retrieval pipeline and multi-agent configu￾rations of MAGNET +Qwen-2.5-Omni-FT demonstrates strong reasoning abilities by first identifying the key videos followed by audio-visual temporal grounding to localise the salien…
Figure 5
Figure 5. Figure 5: Number of hours of videos per topic category in AVHaystacks benchmark. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Number of videos per topic category in AVHaystacks benchmark. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Distribution of videos based on their duration AVHaystacks benchmark. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Number of videos referred to in each question AVHaystacks benchmark. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison of MAGNET and Video-SALMONN on cooking video tutorial. multi-agent audio-visual reasoning modules consistently outperform the baselines on AVHaystacksQA. While the baseline models often struggle to identify and retrieve the most informative video…
Figure 10
Figure 10. Figure 10: Performance comparison of MAGNET and Unified IO2 on english pronunciation tutorial. temporal importance for reasoning tasks, likely due to its sharper penalisation of semantically redundant or temporally proximal frames. This indicates that thoughtful frame selection …
Figure 11
Figure 11. Figure 11: Performance comparison of MAGNET and Qwen-2.5-Omni on Urdu pronunciation tutorial [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of MAGNET and Video-RAG on vocal exercise tutorial. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of MAGNET and VideoRAG on 3D printing tutorial. findings highlight the importance of balancing informativeness and precision in frame selection by carefully tuning the similarity threshold. E Human Evaluation on STEM: Tab. 13 presents a comparat…
Figure 14
Figure 14. Figure 14: Steps involved in benchmark data collection. [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Failure case of MAGNET. H Failure case [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: User study guidelines. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: User consent application. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A joint distillation and policy-learning framework claims near-teacher accuracy on egocentric action recognition, active speaker localization, and behavior anticipation at a fraction of the compute.

Reference graph

Works this paper leans on

127 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

    Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, and Di Hu. Crab: A unified audio-visual scene understanding model with explicit cooperation. arXiv preprint arXiv:2503.13068, 2025

    work page Pith review arXiv 2025

  2. [2]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision, pages 52–70. Springer, 2024

    2024

  3. [3]

    Vita: Towards open-source interactive omni multimodal llm

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024

    arXiv 2024

  4. [4]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024

    2024

  5. [5]

    Crema: Generalizable and efficient video-language reasoning via multimodal modular fusion

    Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Generalizable and efficient video-language reasoning via multimodal modular fusion. arXiv preprint arXiv:2402.05889, 2024

    arXiv 2024

  6. [6]

    Avicuna: Audio-visual llm with interleaver and context-boundary alignment for temporal referential dialogue

    Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio-visual llm with interleaver and context-boundary alignment for temporal referential dialogue. arXiv e-prints, pages arXiv–2403, 2024

    2024

  7. [11]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    arXiv 2024

  8. [12]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472–19495, 2024

    2024

Show all 127 references
  1. [13]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024

    2024 arXiv

  2. [15]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  3. [16]

    Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline

    Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22942–22951, 2023

    2023

  4. [17]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842–72866, 2023

    2023

  5. [18]

    Avqa: A dataset for audio-visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480–3491, 2022

    2022

  6. [19]

    Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios

    Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In European Conference on Computer Vision, pages 146–164. Springer, 2024. 10

    2024

  7. [20]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118, 2022

    2022

  8. [21]

    Vggsound: A large-scale audio- visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020

    2020

  9. [22]

    Saven-vid: Synergistic audio-visual integration for enhanced understanding in long video context

    Jungang Li, Sicheng Tao, Yibo Yan, Xiaojie Gu, Haodong Xu, Xu Zheng, Yuanhuiyi Lyu, Linfeng Zhang, and Xuming Hu. Saven-vid: Synergistic audio-visual integration for enhanced understanding in long video context. arXiv preprint arXiv:2411.16213, 2024

    2024 arXiv

  10. [23]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023

    2023

  11. [24]

    Gemini: Google’s multimodal ai model

    Google AI. Gemini: Google’s multimodal ai model. Google AI Research, 2024. https://fireflies. ai/blog/gemini-vs-gpt-4

    2024

  12. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  13. [26]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  14. [27]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  15. [28]

    Variants of the hungarian method for assignment problems

    Harold W Kuhn. Variants of the hungarian method for assignment problems. Naval research logistics quarterly, 3(4):253–258, 1956

    1956

  16. [29]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024

    2024

  17. [30]

    video-salmonn: Speech-enhanced audio-visual large language models

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406.15704, 2024

    2024 arXiv

  18. [31]

    High-fidelity audio compression with improved rvqgan

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems , 36: 27980–27993, 2023

    2023

  19. [32]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  20. [33]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  21. [34]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22...

    2024

  22. [35]

    Video question answering: Datasets, algorithms and challenges

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering: Datasets, algorithms and challenges. arXiv preprint arXiv:2203.01225, 2022

    2022 arXiv

  23. [36]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  24. [37]

    Movieqa: Understanding stories in movies through question-answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631–4640, 2016

    2016

  25. [38]

    Are we asking the right questions in movieqa? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019

    Bhavan Jasani, Rohit Girdhar, and Deva Ramanan. Are we asking the right questions in movieqa? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019

    2019

  26. [39]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019

    2019

  27. [40]

    How2: a large-scale dataset for multimodal language understanding

    Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze. How2: a large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347, 2018. 11

    2018 arXiv

  28. [41]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021

    2021

  29. [42]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Syst...

    2023

  30. [43]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. arXiv preprint arXiv:2405.09711, 2024

    2024 arXiv

  31. [44]

    Agqa: A benchmark for compositional spatio-temporal reasoning

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021

    2021

  32. [45]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  33. [46]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems , 37: 28828–28857, 2024

    2024

  34. [47]

    Just ask: Learning to answer questions from millions of narrated videos

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1686–1697, 2021

    2021

  35. [48]

    Instructionbench: An instructional video understanding benchmark

    Haiwan Wei, Yitian Yuan, Xiaohan Lan, Wei Ke, and Lin Ma. Instructionbench: An instructional video understanding benchmark. arXiv preprint arXiv:2504.05040, 2025

    2025 arXiv

  36. [49]

    Hd-epic: A highly-detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. arXiv preprint arXiv:2502.04144, 2025

    2025 arXiv

  37. [50]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision ...

    2022

  38. [51]

    Ego-exo4d: Understand- ing skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understand- ing skilled human activity from first-and third-person perspectives. In Proceedin...

    2024

  39. [52]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on comput...

    2018

  40. [53]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2...

    2023

  41. [54]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  42. [55]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  43. [56]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  44. [57]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  45. [58]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  46. [59]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024. 12

    2024 arXiv

  47. [60]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  48. [61]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  49. [62]

    Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889, 2024

    2024

  50. [63]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023

    2023 arXiv

  51. [64]

    Flash- vstream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash- vstream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    2024 arXiv

  52. [65]

    Videoagent: A memory- augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. arXiv preprint arXiv:2403.11481, 2024

    2024 arXiv

  53. [66]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517, 2024

    2024 arXiv

  54. [67]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024

    2024 arXiv

  55. [68]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems...

    2020

  56. [69]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, 2023

    2023

  57. [70]

    Sentence-level prompts benefit composed image retrieval

    Yang Bai, Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Chun-Mei Feng. Sentence-level prompts benefit composed image retrieval. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. Spotlight Presentation

    2024

  58. [71]

    Vqa4cir: Boosting composed image retrieval with visual question answering

    Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman Khan, Wangmeng Zuo, Xinxing Xu, Rick Siow Mong Goh, and Yong Liu. Vqa4cir: Boosting composed image retrieval with visual question answering. arXiv preprint arXiv:2312.12273, 2023. URL https://arxiv.org/abs/2312.12273

    2023 arXiv

  59. [72]

    Searching for best practices in retrieval-augmented generation

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval-augmented generation. arXiv preprint arXiv:2407.01219, 2024

    2024 arXiv

  60. [73]

    Retrieval-augmented generation for natural language processing: A survey

    Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, et al. Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.13193, 2024

    2024 arXiv

  61. [74]

    A survey on retrieval-augmented text generation for large language models

    Yizheng Huang and Jimmy Huang. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024

    2024 arXiv

  62. [75]

    Retrieval-augmented generation for ai-generated content: A survey

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473, 2024

    2024 arXiv

  63. [76]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023

    2023 arXiv

  64. [77]

    Realm: Retrieval- augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 3929–3938. PMLR, 2020

    2020

  65. [78]

    SAIL: Search-augmented instruction learning

    Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. SAIL: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023

    2023 arXiv

  66. [79]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag...

    2020

  67. [80]

    Document haystacks: Vision-language reasoning over piles of 1000+ documents

    Jun Chen, Dannong Xu, Junjie Fei, Chun-Mei Feng, and Mohamed Elhoseiny. Document haystacks: Vision-language reasoning over piles of 1000+ documents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24817–24826, 2025

    2025

  68. [81]

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. MuRAG: Multimodal retrieval- augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5558–55...

    2022

  69. [82]

    Gonzalez, Trevor Darrell, and David M

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual haystacks: A vision-centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766, 2024

    2024 arXiv

  70. [83]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024

    arXiv

  71. [84]

    Visual-rag: Benchmarking text-to-image retrieval augmented generation for visual knowledge intensive queries

    Yin Wu, Quanyu Long, Jing Li, Jianfei Yu, and Wenya Wang. Visual-rag: Benchmarking text-to-image retrieval augmented generation for visual knowledge intensive queries. arXiv preprint arXiv:2502.16636, 2025

    2025 arXiv

  72. [85]

    Vdocrag: Retrieval-augmented generation over visually-rich documents

    Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Vdocrag: Retrieval-augmented generation over visually-rich documents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24827–24837, 2025

    2025

  73. [86]

    Retrieval augmented visual question answering with outside knowledge

    Weizhe Lin and Bill Byrne. Retrieval augmented visual question answering with outside knowledge. arXiv preprint arXiv:2210.03809, 2022

    2022 arXiv

  74. [87]

    Fact-aware multimodal retrieval augmentation for accurate medical radiology report generation

    Liwen Sun, James Zhao, Megan Han, and Chenyan Xiong. Fact-aware multimodal retrieval augmentation for accurate medical radiology report generation. arXiv preprint arXiv:2407.15268, 2024

    2024 arXiv

  75. [88]

    Rule: Reliable multimodal rag for factuality in medical vision language models

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1081–1093, 2024

    2024

  76. [89]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  77. [90]

    Gpt-4o: Enhanced multimodal language model

    OpenAI. Gpt-4o: Enhanced multimodal language model. OpenAI Research, 2024. https://openai. com/index/hello-gpt-4o/

    2024

  78. [91]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=1tZbq88f27

    2024

  79. [92]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022

    2022 arXiv

  80. [93]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2024

    2024 arXiv

  81. [94]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  82. [95]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024

    2024 arXiv

  83. [96]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  84. [97]

    Videorag: Retrieval- augmented generation with extreme long-context videos

    Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Videorag: Retrieval- augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549, 2025

    2025 arXiv

  85. [98]

    V-desirr: Very fast deep embedded single image reflection removal

    BH Prasad, Lokesh R Boregowda, Kaushik Mitra, Sanjoy Chowdhury, et al. V-desirr: Very fast deep embedded single image reflection removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2390–2399, 2021

    2021

  86. [99]

    Measured albedo in the wild: Filling the gap in intrinsics evaluation

    Jiaye Wu, Sanjoy Chowdhury, Hariharmano Shanmugaraja, David Jacobs, and Soumyadip Sengupta. Measured albedo in the wild: Filling the gap in intrinsics evaluation. In 2023 IEEE International Conference on Computational Photography (ICCP), pages 1–12. IEEE, 2023. 14

    2023

  87. [100]

    Adverb: Visually guided audio dereverberation

    Sanjoy Chowdhury, Sreyan Ghosh, Subhrajyoti Dasgupta, Anton Ratnarajah, Utkarsh Tyagi, and Dinesh Manocha. Adverb: Visually guided audio dereverberation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7884–7896, 2023

    2023

  88. [101]

    Melfusion: Synthesizing music from image and language cues using diffusion models

    Sanjoy Chowdhury, Sayan Nag, KJ Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. Melfusion: Synthesizing music from image and language cues using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26826–26835, 2024

    2024

  89. [102]

    Foleygen: Visually-guided audio generation

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2024

    2024

  90. [103]

    Codi-2: In-context interleaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27425–27434, 2024

    2024

  91. [104]

    Listen to the pixels

    Sanjoy Chowdhury, Subhrajyoti Dasgupta, Sudip Das, and Ujjwal Bhattacharya. Listen to the pixels. In 2021 IEEE International Conference on Image Processing (ICIP), pages 2568–2572. IEEE, 2021

    2021

  92. [105]

    Audvisum: Self- supervised deep reinforcement learning for diverse audio-visual summary generation

    Sanjoy Chowdhury, Aditya Patra, Subhrajyoti Dasgupta, and Ujjwal Bhattacharya. Audvisum: Self- supervised deep reinforcement learning for diverse audio-visual summary generation. In BMVC, page 315, 2021

    2021

  93. [106]

    Audio–visual representation learning for anomaly events detection in crowds

    Junyu Gao, Hao Yang, Maoguo Gong, and Xuelong Li. Audio–visual representation learning for anomaly events detection in crowds. Neurocomputing, 582:127489, 2024

    2024

  94. [107]

    Representation learning for semantic alignment of language, audio, and visual modalities

    Parthasaarathy Sudarsanam, Irene Martín-Morató, and Tuomas Virtanen. Representation learning for semantic alignment of language, audio, and visual modalities. arXiv preprint arXiv:2505.14562, 2025

    2025 arXiv

  95. [108]

    Aurelia: Test-time reasoning distillation in audio-visual llms

    Sanjoy Chowdhury, Hanan Gani, Nishit Anand, Sayan Nag, Ruohan Gao, Mohamed Elhoseiny, Salman Khan, and Dinesh Manocha. Aurelia: Test-time reasoning distillation in audio-visual llms. arXiv preprint arXiv:2503.23219, 2025

    2025 arXiv

  96. [109]

    Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual llms. arXiv preprint arXiv:2501.02135, 2025

    2025 arXiv

  97. [110]

    Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-language model

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-language model. arXiv preprint arXiv:2312.12423, 2023

    2023 arXiv

  98. [111]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  99. [112]

    Apollo: unified adapter and prompt learning for vision language models

    Sanjoy Chowdhury, Sayan Nag, and Dinesh Manocha. Apollo: unified adapter and prompt learning for vision language models. arXiv preprint arXiv:2312.01564, 2023

    2023 arXiv

  100. [113]

    Towards determining perceived audience intent for multimodal social media posts using the theory of reasoned action

    Trisha Mittal, Sanjoy Chowdhury, Pooja Guhan, Snikitha Chelluri, and Dinesh Manocha. Towards determining perceived audience intent for multimodal social media posts using the theory of reasoned action. Scientific Reports, 14(1):10606, 2024

    2024

  101. [114]

    Can llms generate human-like wayfinding instructions? towards platform-agnostic embodied instruction synthesis

    Vishnu Sashank Dorbala, Sanjoy Chowdhury, and Dinesh Manocha. Can llms generate human-like wayfinding instructions? towards platform-agnostic embodied instruction synthesis. arXiv preprint arXiv:2403.11487, 2024

    2024 arXiv

  102. [115]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions

    Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214, 2024

    2024

  103. [116]

    Safari: Adaptive sequence transformer for weakly supervised referring expression segmentation

    Sayan Nag, Koustava Goswami, and Srikrishna Karanam. Safari: Adaptive sequence transformer for weakly supervised referring expression segmentation. In European Conference on Computer Vision, pages 485–503. Springer, 2024

    2024

  104. [117]

    V olta: Vision-language transformer with weakly-supervised local-feature alignment

    Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik J Shah, Yann LeCun, and Rama Chellappa. V olta: Vision-language transformer with weakly-supervised local-feature alignment. Transactions on Machine Learning Research, 2023

    2023

  105. [118]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vis...

    2023

  106. [119]

    Semantic audio-visual navigation

    Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15516–15525, 2021

    2021

  107. [120]

    Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023. 15

    2023 arXiv

  108. [121]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098–89124, 2024

    2024

  109. [122]

    Av-odyssey bench: Can your multimodal llms really understand audio-visual information? arXiv preprint arXiv:2412.02611, 2024

    Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, et al. Av-odyssey bench: Can your multimodal llms really understand audio-visual information? arXiv preprint arXiv:2412.02611, 2024

    2024 arXiv

  110. [123]

    Omnibench: Towards the future of universal omni-language models

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272, 2024

    2024

  111. [124]

    Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:2411.19772, 2024

    2024 arXiv

  112. [125]

    Avhbench: A cross-modal hallucination benchmark for audio-visual large language models

    Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXiv:2410.18325, 2024. 16 MAGNET : A Multi-agent Framework for Finding Audio-Visual Ne...

    2024 arXiv

  113. [126]

    We provide step-by-step, clear instructions about the task, the reasoning process, and the expected output

    step 5 References: 1.txt 0017s > 0074s, 8.txt 0045s > 0270s, 2.txt 0050s > 0100s, 3.txt 0110s > 0150s G SFS Prompt Below, we add the prompt used to select the key frames using the SFS algorithm. We provide step-by-step, clear instructions about the task, the reasoning process,...

  114. [127]

    Presence of objects or actions explicitly mentioned in the question

  115. [128]

    Scenes that clearly align with the question’s context

  116. [129]

    Visual elements directly related to the question details

  117. [130]

    Location or background context, even if the main object/action is not visible

  118. [131]

    Semantically related or typically co-occurring objects

  119. [132]

    26 Figure 15: Failure case of MAGNET

    Human motion or activity suggesting relevant events Step 2: Output List the selected image indices using the format: [idx1, idx2, idx3, ...] The objective is to select visual evidence useful for answering the question, not to answer the question itself. 26 Figure 15: Failure c...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.