Pith. sign in

REVIEW 4 major objections 4 minor 123 references

This paper claims that a fully open audio-visual language model, trained with a three-stage curriculum and timestamp-grounded chain-of-thought, outperforms similarly sized open models and rivals or surpasses much larger closed ones on long,

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:17 UTC pith:HPUJVBJ5

load-bearing objection A real open AV-LLM dataset and recipe worth engaging with; the headline benchmark claims need a contamination check and full results before they should be believed. the 4 major comments →

arxiv 2607.16107 v1 pith:HPUJVBJ5 submitted 2026-07-17 eess.AS cs.CV

Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

Sreyan Ghosh , Arushi Goel , Kaousheik Jayakumar , Lasha Koroshinadze , Nishit Anand , Siddharth Gururani , Hanrong Ye , Pritam Biswas
show 14 more authors
Yuanhang Su Ehsan Hosseini-Asl Sang-gil Lee Zhifeng Kong Jaehyeon Kim Sungwon Kim S Sakshi Ramani Duraiswami Dinesh Manocha Andrew Tao Mohammad Shoeybi Bryan Catanzaro Ming-Yu Liu Wei Ping
This is my paper
classification eess.AS cs.CV
keywords audio-visual large language modellong-form video understandingcross-modal reasoningtemporal reasoningtimestamp-grounded chain-of-thoughtcurriculum learningsynthetic question answeringopen-source multimodal model
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AV-Flamingo is an attempt to show that open, fully reproducible audio-visual models can lead the field on long-video understanding rather than trailing proprietary systems. The paper builds a model that jointly processes sound and vision by interleaving audio and visual tokens in time, then trains it in three stages—short clips, then videos up to 15 minutes, then reasoning—on a new dataset of roughly seven million caption and question-answer instances. Its central claim is that this recipe yields clear wins over open models of similar size and results that are competitive with, and sometimes better than, much larger open-weight and closed systems, especially where questions require temporal and cross-modal reasoning. The stakes are practical: if true, teams without access to proprietary data and massive budgets can still build frontier audio-visual assistants, and the released dataset gives the field a reusable substrate for long-form audio-visual reasoning.

Core claim

The paper's central discovery claim is that long-form audio-visual understanding does not require a fundamentally larger model—it requires training data and a reasoning procedure that force the model to align what it hears with what it sees over time. The authors identify three ingredients: AV-Skills, a curated dataset of hundreds of thousands of hours of video with skill-targeted questions spanning counting, temporal order, causal inference, and cross-modal reference; a three-stage curriculum that moves from short-context perception to 15-minute long-context reasoning; and Temporal Audio-Visual Interleaved Chain-of-Thought (TAVIT), which requires every intermediate reasoning step to be anch

What carries the argument

The load-bearing mechanism is Temporal Audio-Visual Interleaved Chain-of-Thought (TAVIT): a reasoning format in which each intermediate step in the model's answer is explicitly tagged with timestamps and must interleave evidence from both audio and visual streams, so the language model cannot answer from one modality alone. Around it, the paper assembles the AV-Skills dataset—approximately seven million curated caption and question-answer instances across thirteen skill categories for short and long videos—and a three-stage curriculum that first fine-tunes on short clips, then extends context to 15-minute videos, then applies supervised and reinforcement learning on timestamped reasoning cha

Load-bearing premise

The reported 'clear margins' depend on AV-Skills not substantially overlapping with evaluation videos; the paper concedes that AV-Skills draws on the same public and open-internet sources used by those benchmarks, so if overlap is large the gains could reflect memorization rather than reasoning.

What would settle it

A public audit that fingerprints the training videos (e.g., perceptual hashes or video IDs) and compares them against the test videos of the long-video benchmarks would settle the question: if a large fraction of test clips appear in training, the claimed margins are contaminated; if overlap is negligible, the reasoning claim is supported. A second, cheaper check is to rerun the model on temporally shuffled audio-visual tracks—if performance does not drop sharply, the model is not actually using cross-modal temporal alignment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the reported margins hold, open models of around seven billion parameters can close the gap with much larger proprietary systems on long audio-visual benchmarks, shifting the default choice for reproducible research.
  • The three-stage curriculum establishes a training recipe—short-context perception first, long-context grounding second, timestamped reasoning third—that other builders can copy without proprietary data.
  • Releasing the training corpus and code means the AV-Skills skill taxonomy becomes a reusable instrument for diagnosing and training long-video models.
  • Timestamp-grounded chain-of-thought improves temporal alignment and interpretability; the paper's own examples show outputs that cite exact moments, making model reasoning auditable.
  • Because the model transfers to unseen tasks and retains strong audio-only and vision-only performance, the paper claims the joint training does not come at the cost of single-modality competence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If training and test videos overlap as little as the paper hopes, the strongest claim—'clear margins' on long-video benchmarks—would likely degrade on adversarial or out-of-distribution videos; the paper's own limitation note makes this the first thing to audit.
  • The TAVIT format points to a general design for long-context multimodal agents: forcing intermediate reasoning to cite timestamps should also help with retrieval, grounding, and error localization in other applications, though the paper does not explore this.
  • A cheap check of the dataset's value is to train a short-context-only variant and compare it against the full curriculum on the same long-video questions; the paper's ablation isolates AV-Skills-Short but does not fully separate curriculum order from data quantity.
  • The synthetic annotation pipeline—caption generation followed by prompt-driven question-answer synthesis—is itself transferable, but annotation errors are hard to detect at seven-million-instance scale, so benchmark gains should be corroborated with human spot-checks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AV-Flamingo, a fully open audio-visual large language model initialized from OmniVinci and trained on a new AV-Skills dataset (~100K short and ~140K long hours; ~7M caption/QA instances), with a three-stage short-to-long-to-CoT curriculum and a TAVIT timestamp-grounded chain-of-thought framework. The authors report results on a table of omni-modal, audio, video, and ASR benchmarks and claim that AV-Flamingo outperforms similarly sized open models by clear margins, especially on long and complex real-world videos, while remaining competitive with much larger closed/open-weight models. The paper also commits to releasing model, code, data, and project/demo assets.

Significance. If the reported results are trustworthy, AV-Flamingo would be a valuable open baseline and resource for long-form audio-visual reasoning. The released code, model, and datasets, along with detailed training recipes and TAVIT examples, are concrete contributions that the community can build on. The central claim, however, depends on the validity of the evaluation numbers, which are currently not protected against known contamination and are reported without uncertainty quantification. The paper's own limitation section concedes possible overlap between AV-Skills and prior training/benchmark data, but no analysis is provided. Because the headline contribution is an empirical performance claim, this gap is load-bearing.

major comments (4)
  1. [§3.2, §6, Appendix F.2] The central performance claim may be inflated by train/test overlap. AV-Skills is built from public corpora (YouTube-8M, HD-VILA, InternVid, VidChapters, HarmonySet) and open-internet video, and Appendix F.2 stores raw YouTube URLs for AV-Think. Benchmarks such as Video-MME, MMOU, DailyOmni, and LongVideoBench are themselves constructed from similar public video pools. The paper reports no deduplication, video-ID check, frame-similarity analysis, or per-benchmark overlap statistic; §6 explicitly concedes 'potential overlap with prior training data.' This is not resolved by the current evidence. Please quantify overlap (e.g., near-duplicate retrieval) and report scores on non-overlapping test subsets, or otherwise show that the 'clear margins' in the abstract are not memorization artifacts.
  2. [§4, §5, Table 1] Table 1 is the only results table but omits at least two benchmarks named in §4 as evaluated (ActivityNetQA, Common Voice 15), while the paper claims 15+ benchmarks. No error bars, repeated runs, or significance tests are reported. Several headline margins are small (Video-MME 70.7 vs. 67.3; WorldSense 50.3 vs. 48.2; MMAU average 73.49 vs. 72.42) and could be within run-to-run variability. Report all claimed benchmarks, multiple seeds or bootstrap confidence intervals, and per-row sample sizes so the 'clear margins' claim is assessable.
  3. [§5, Table 1; §3.2] The strongest omni results include MMOU and MMAU, benchmarks co-authored by members of the same group, and the model is initialized from the group's OmniVinci checkpoint and trained on the group's AF3/AudioSkills data lineage. Co-authorship does not by itself invalidate the results, but combined with the unresolved contamination question it creates a material risk of evaluation bias. Please state the authors' roles in these benchmarks, provide third-party evaluation, or show robustness when MMOU/MMAU are excluded.
  4. [§3.3, Table 6] The three-stage curriculum and TAVIT are presented as key contributions, but no ablation isolates their effects. Table 6 tests only the cumulative addition of AV-Skills-Short and AV-Skills-Long; it does not test stage order, context-length scheduling, or whether the short→long→CoT ordering matters. Similarly, AVF-Think vs. AVF-Instruct in Table 1 is not accompanied by a controlled comparison against no-CoT or non-grounded CoT. Either add these ablations or soften the causal claims about the curriculum and TAVIT.
minor comments (4)
  1. [§3.2, Table 2] The text says AV-Skills-Long is between 60 seconds and 15 minutes, while Table 2's long-duration rows appear partially overlapping (e.g., Long AV Captioning 51K hours plus 13 QA subsets summing to ~140K hours). Clarify whether these components are disjoint and how the total 3.2M instances is obtained.
  2. [Appendix H] The paper calls AV-Flamingo 'fully open,' but several assets are released under non-commercial licenses (e.g., NVIDIA OneWay Noncommercial) and the source media of public datasets remain under third-party terms. Clarify in the abstract or introduction what 'fully open' means with respect to model weights, data artifacts, and code.
  3. [Eq. (1)] The GRPO objective notation is slightly imprecise: the expectation over q and the sampled group {o_i} is written as one expectation, but the group-size averaging is then inside. This is understandable but could be cleaned up for the camera-ready.
  4. [§3.1, Figure 2] The caption describes 'Rotary Time Embedding' while the body uses 'Constrained Rotary Time Embeddings (CRTE)'. Use one consistent name.

Circularity Check

2 steps flagged

The 'clear margins' claim leans on a self-authored benchmark (MMOU) whose QA pairs were used during data curation to design AV-Skills; the central claim still has independent external support.

specific steps
  1. fitted input called prediction [Section 3.2 (Data Curation) -> Section 5 (Results), Table 1]
    "As a first step in data curation, we identify the key skills required for strong real-world audio-visual understanding. To do so, we evaluate frontier omni-modal and audio-visual models, including the Qwen-Omni series, Gemini, and VideoLLaMA, on challenging benchmarks such as WorldSense and MMOU, and analyze their outputs to identify systematic capability gaps. In particular, we convert benchmark QA pairs into open-ended form..."

    The QA pairs from WorldSense and MMOU are used during data curation to choose which skills AV-Skills must teach and to shape the synthetic QA generation prompts (Figures 6-20). The same benchmarks are then reported as evidence of improvement, with Section 5 stating 'Results on MMOU emphasize AVF is best-in-class for long and complex audio-visual understanding.' Thus the MMOU/WorldSense scores are not independent tests of the trained model: the training-data distribution was explicitly adjusted to those benchmarks' questions. The reported gains on those benchmarks are partly a consequence of this feedback, not a surprise from an untouched held-out evaluation.

  2. self citation load bearing [Section 5 (Results) and References (Goel et al., 2026)]
    "Results on MMOU emphasize AVF is best-in-class for long and complex audio-visual understanding."

    The benchmark that anchors the 'long and complex real-world audio-visual' component of the headline claim is MMOU (Goel et al., 2026), whose author list overlaps with the present paper's authors (A. Goel, S. Ghosh, et al.). The paper presents no independent evidence that MMOU's test videos are disjoint from the AV-Skills training pool, which is built from 'YouTube-8M (Abu-El-Haija et al., 2016), HD-VILA (Xue et al., 2022), InternVid (Wang et al., 2024b), and VidChapters...' plus open-internet videos. When combined with the gap-analysis use of MMOU QA pairs, the MMOU result is not an external check of the central claim; it is a self-authored, training-informed benchmark. The claim is not wholly circular because external benchmarks such as Video-MME, LibriSpeech, and DailyOmni also support t

full rationale

The paper is an empirical systems paper, not a formal derivation, so most components are not circular in the equation-reduction sense. The architecture largely follows prior work by the same group (OmniVinci initialization, AF-Whisper, CRTE), but that is normal engineering reuse and does not by itself make the benchmark claims circular. The main circularity concern is the soft feedback path: Section 3.2 explicitly says the authors evaluated frontier models on WorldSense and MMOU and 'convert benchmark QA pairs into open-ended form' to identify capability gaps, and then AV-Skills was designed to address those gaps. Reporting gains on those same benchmarks, especially on the self-authored MMOU benchmark, makes the 'long and complex real-world audio-visual' part of the abstract's central claim partly fitted to the evaluation instrument rather than independently demonstrated. The paper also concedes in Section 6 that 'AV-Skills is built from public datasets and open-internet videos, which may introduce source bias and potential overlap with prior training data,' and no deduplication or per-benchmark overlap analysis is reported; Appendix F.2 even stores raw YouTube URLs for AV-Think samples. These facts weaken the external validity of the headline margins, but they are more contamination/correctness risks than strict circularity. Because the model also performs well on several external benchmarks (Video-MME, LibriSpeech, DailyOmni, etc.), the central claim retains independent content. I therefore assign a score of 4 rather than a higher score: some self-citation and a training-to-benchmark feedback path are present, but the result is not entirely reducible to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are postulated; TAVIT and AV-Skills are methods/datasets, not entities. The central claim rests on hand-chosen training hyperparameters, synthetic-data quality, and the absence of train/test contamination, all of which the paper only partially validates.

free parameters (4)
  • Stage-wise data mixing ratios = e.g., AV-Skills-Short 1.0/0.5 epochs; AV-Skills-Long 2.0 epochs; see Table 2
    The central performance claim depends on hand-chosen re-weighting of dozens of datasets across three training stages; these ratios are not derived or optimized.
  • Context length caps = 16K tokens (pre-train), 32K tokens (mid/post)
    Caps define what counts as long-form and bound the training recipe; hand-chosen.
  • Duration thresholds for AV-Skills split = ≤60s short; 60s–15min long
    Determines dataset composition and curriculum; arbitrary cutoffs.
  • GRPO group size G = 5
    Sampling group size in reinforcement learning; hand-set.
axioms (4)
  • domain assumption LLM-generated (GPT) captions and QA annotations in AV-Skills and AV-Think are accurate and unbiased enough to teach cross-modal reasoning.
    The entire dataset is synthetically annotated; only qualitative examples are shown, no human evaluation of annotation quality is reported (Sec. 3.2, Appendix B).
  • domain assumption The open-internet videos used in AV-Skills do not substantially overlap with evaluation benchmark videos.
    The authors themselves flag this in Sec. 6: 'potential overlap with prior training data.' If false, benchmark gains are inflated.
  • domain assumption Benchmark accuracy on multiple-choice QA is a faithful measure of real-world long-video audio-visual understanding.
    The model is evaluated mostly on MQ/ASR benchmarks; the paper's own Sec. 6 notes 'current benchmarks do not fully capture open-ended real-world deployment.'
  • ad hoc to paper The three-stage curriculum ordering (short → long → CoT) is necessary for the reported performance.
    No ablation varies the curriculum order or stage composition; Table 6 only ablates data presence, not order.

pith-pipeline@v1.3.0-alltime-deepseek · 28942 in / 14418 out tokens · 116878 ms · 2026-08-01T21:17:59.483345+00:00 · methodology

0 comments
read the original abstract

We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large language model (AV-LLM) for joint understanding and reasoning over audio, images, and long-form videos. Unlike prior AV-LLMs that primarily focus on short clips, AV-Flamingo is designed for understanding and reasoning over long and complex real-world (audio-visual) videos. To support this, we make three key contributions: (i) Audio-Visual-Skills, a large-scale collection of real-world videos with ~7M caption and question-answer training instances designed to emphasize temporal, compositional, and cross-modal audio-visual reasoning; (ii) a novel three-stage curriculum that progressively trains the model from short-range perception to long-horizon multi-event reasoning; and (iii) Temporal Audio-Visual Interleaved Chain-of-Thought, a reasoning framework that explicitly grounds intermediate reasoning steps to timestamps in long audio-visual streams, improving temporal alignment and interpretability. Extensive experiments across 15+ audio-visual, omni-modal, audio, and vision benchmarks show that AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with, and in some cases surpasses, much larger open-weight and closed models, particularly on long and complex real-world audio-visual understanding and reasoning tasks. Beyond benchmark performance, AV-Flamingo exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability.

Figures

Figures reproduced from arXiv: 2607.16107 by Andrew Tao, Arushi Goel, Bryan Catanzaro, Dinesh Manocha, Ehsan Hosseini-Asl, Hanrong Ye, Jaehyeon Kim, Kaousheik Jayakumar, Lasha Koroshinadze, Ming-Yu Liu, Mohammad Shoeybi, Nishit Anand, Pritam Biswas, Ramani Duraiswami, Sang-gil Lee, Siddharth Gururani, Sreyan Ghosh, S Sakshi, Sungwon Kim, Wei Ping, Yuanhang Su, Zhifeng Kong.

Figure 1
Figure 1. Figure 1: Comparison of AVF with current SOTA model on various benchmarks. Video constitutes 82% of global internet traffic, and users spend over 2.5 hours per day watching online videos on average (Kumar, 2026). These videos are often long, diverse, and information-rich, combining complex visuals, graphics, speech, sounds, and music in a continuous stream. This closely mirrors how humans perceive the world: as an o… view at source ↗
Figure 2
Figure 2. Figure 2: AV-Flamingo training and architecture. Top: Starting from OmniVinci, AV-Flamingo is trained in three stages: pre-training on AV-Skills short-context data, mid-training on AV-Skills long-context data, and post-training on chain-of-thought data, producing AVF-Instruct and AVF-Think. Bottom: AV-Flamingo accepts image, video, audio, and text inputs. Visual and audio streams are encoded through separate encoder… view at source ↗
Figure 3
Figure 3. Figure 3: Examples of types of questions in Audio-Visual-Skills. More examples are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Category distribution of the videos used in the training datasets. C. Audio-Visual Flamingo Training Details In this section, we present the training settings of our model across all stages, each with specific configurations. Details are in [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Additional QA pair examples from AV-Skills-Long. Representative samples spanning eight long￾form skill categories used during mid-training: Needle-in-the-Haystack, Temporal Order, Temporal Referring, Topic Summary, Counting, Sub-scene, Comparative, and Temporal Attribute. Examples include both open-ended and multiple-choice formats. E. GRPO-based RL training Details GRPO for Audio-Visual Understanding and … view at source ↗
Figure 6
Figure 6. Figure 6: Prompt for Comparative Reasoning QA generation. Instructs the LLM to produce a multiple-choice question (with up to 10 options) that hinges on a key contrast or similarity between two distinct audio-visual segments in a long video, requiring both modalities to answer. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for Context Understanding QA generation. Instructs the LLM to produce a multiple-choice question that depends on the broader situational setting, background, or scene conditions inferable only by jointly reasoning over visuals, sounds, and speech. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt for Counting QA generation. Instructs the LLM to produce a question requiring the model to count occurrences of a specific audio-visual event across the full video, verifying noisy audio captions against visual cues. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt for Detailed Captioning with Transcripts. (Part 1 of 2) 33 [PITH_FULL_IMAGE:figures/full_fig_p033_9.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt for Detailed Captioning with Transcripts. (Part 2 of 2) Instructs the LLM to produce a single comprehensive audio-visual caption (up to 1000 words) that integrates speech transcripts in-line with descriptions of visuals, sounds, and scene dynamics across the entire video [PITH_FULL_IMAGE:figures/full_fig_p034_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt for Detailed Captioning. Instructs the LLM to produce a single audio-visual caption (up to 500 words) summarizing the full video from a joint perceptual perspective, without segment-wise structure or explicit timestamps. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for Audio-Visual Event Alignment QA generation. Instructs the LLM to produce a multiple-choice question identifying the exact audio cue that coincides with a specific visual frame, or vice versa, requiring tight cross-modal synchronization. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for Event Sequence QA generation. Instructs the LLM to produce a multiple-choice question about the relative temporal ordering of two or more significant audio-visual events spanning the video timeline. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt for Holistic Reasoning QA generation. Instructs the LLM to produce an open-ended question that requires integrating audio-visual evidence distributed across a substantial portion of the video, rather than any single localized moment. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt for Inference QA generation. Instructs the LLM to produce a multiple-choice question whose answer can only be deduced by combining multiple implicit audio-visual clues spread across the video, with the correct option deliberately not stated directly. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt for Needle-in-the-Haystack QA generation. Instructs the LLM to produce an open-ended question targeting a specific but important moment in a long video, where broader audio-visual context is still required to identify it correctly. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Prompt for Audio-Visual Referring QA generation. Instructs the LLM to produce up to two question–answer pairs that link a visual event to its corresponding sound, or a sound to its corresponding visual scene, exercising cross-modal grounding. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt for Sub-scene Understanding QA generation. Instructs the LLM to produce an open-ended question about a meaningful intermediate portion of a long video, defined by the events that precede and follow it. 42 [PITH_FULL_IMAGE:figures/full_fig_p042_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for Temporal Reasoning QA generation. (Part 1 of 2). 43 [PITH_FULL_IMAGE:figures/full_fig_p043_18.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for Temporal Reasoning QA generation. (Part 2 of 2) Instructs the LLM to produce four question–answer pairs covering Temporal Order, Temporal Referring, and Temporal Attribute reasoning over the chronological structure and evolving dynamics of the video. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Prompt for Topic-Level Reasoning QA generation. Instructs the LLM to produce an open-ended question about the main activity, objective, or overall theme of the video, requiring synthesis across the full timeline. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Prompt used for synthesizing AV-Think reasoning triplets (Part 1 of 2). 46 [PITH_FULL_IMAGE:figures/full_fig_p046_20.png] view at source ↗
Figure 20
Figure 20. Figure 20: Prompt used for synthesizing AV-Think reasoning triplets (Part 2 of 2). Given timestamped audio and visual captions extracted from a long-form video, we prompt an LLM to generate a question–answer– reasoning triplet along with an integrated audio-visual caption. The prompt instructs the model to (i) require holistic understanding across both modalities, (ii) demand multi-hop temporal reasoning rather than… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

123 extracted references · 1 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Flamingo: a visual language model for few-shot learning , author=. Advances in neural information processing systems , volume=

  2. [2]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  3. [3]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  4. [4]

    arXiv preprint arXiv:2410.12109 , year=

    Omcat: Omni context aware transformer , author=. arXiv preprint arXiv:2410.12109 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2412.05271 , year=

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling , author=. arXiv preprint arXiv:2412.05271 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2511.21631 , year=

    Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2501.13106 , year=

    Videollama 3: Frontier multimodal foundation models for image and video understanding , author=. arXiv preprint arXiv:2501.13106 , year=

    Pith/arXiv arXiv

  8. [8]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

    Qwen Team , month =. Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  9. [9]

    arXiv preprint arXiv:2305.10790 , year=

    Listen, think, and understand , author=. arXiv preprint arXiv:2305.10790 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2310.13289 , year=

    Salmonn: Towards generic hearing abilities for large language models , author=. arXiv preprint arXiv:2310.13289 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2407.10759 , year=

    Qwen2-audio technical report , author=. arXiv preprint arXiv:2407.10759 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2507.08128 , year=

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models , author=. arXiv preprint arXiv:2507.08128 , year=

    Pith/arXiv arXiv

  13. [13]

    Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations , pages=

    Video-llama: An instruction-tuned audio-visual language model for video understanding , author=. Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations , pages=

    2023

  14. [14]

    arXiv preprint arXiv:2406.07476 , year=

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms , author=. arXiv preprint arXiv:2406.07476 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2406.15704 , year=

    video-salmonn: Speech-enhanced audio-visual large language models , author=. arXiv preprint arXiv:2406.15704 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2506.15220 , year=

    video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models , author=. arXiv preprint arXiv:2506.15220 , year=

    arXiv

  17. [17]

    arXiv preprint arXiv:2505.18110 , year=

    Watch and listen: Understanding audio-visual-speech moments with multimodal llm , author=. arXiv preprint arXiv:2505.18110 , year=

    arXiv

  18. [18]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2403.05530 , year=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. arXiv preprint arXiv:2403.05530 , year=

    Pith/arXiv arXiv

  20. [20]

    2025 , eprint=

    Qwen2.5-Omni Technical Report , author=. 2025 , eprint=

    2025

  21. [21]

    arXiv preprint arXiv:2509.17765 , year=

    Qwen3-omni technical report , author=. arXiv preprint arXiv:2509.17765 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2503.01743 , year=

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras , author=. arXiv preprint arXiv:2503.01743 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2510.15870 , year=

    OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM , author=. arXiv preprint arXiv:2510.15870 , year=

    arXiv

  24. [24]

    European Conference on Computer Vision , pages=

    Llama-vid: An image is worth 2 tokens in large language models , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  25. [25]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  26. [26]

    arXiv preprint arXiv:2603.14145 , year=

    MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos , author=. arXiv preprint arXiv:2603.14145 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2510.20579 , year=

    Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence , author=. arXiv preprint arXiv:2510.20579 , year=

    arXiv

  28. [28]

    2025 , eprint=

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models , author=. 2025 , eprint=

    2025

  29. [29]

    Proceedings of the ACM Web Conference 2024 , pages=

    Unitime: A language-empowered unified model for cross-domain time series forecasting , author=. Proceedings of the ACM Web Conference 2024 , pages=

    2024

  30. [30]

    arXiv preprint arXiv:2507.02001 , year=

    Temporal chain of thought: Long-video understanding by thinking in frames , author=. arXiv preprint arXiv:2507.02001 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2503.11197 , year=

    Reinforcement learning outperforms supervised fine-tuning: A case study on audio question answering , author=. arXiv preprint arXiv:2503.11197 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2511.15848 , year=

    Step-Audio-R1 Technical Report , author=. arXiv preprint arXiv:2511.15848 , year=

    arXiv

  33. [33]

    European Conference on Computer Vision , pages=

    Sharegpt4v: Improving large multi-modal models with better captions , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  34. [34]

    2024 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) , pages=

    A Framework for Synthetic Audio Conversations Generation Using Large Language Models , author=. 2024 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) , pages=. 2024 , organization=

    2024

  35. [35]

    Qwen3.5-Omni: Scaling Up, Toward Native Omni-Modal AGI , url =

    Qwen Team , month =. Qwen3.5-Omni: Scaling Up, Toward Native Omni-Modal AGI , url =

  36. [36]

    arXiv preprint arXiv:2604.24954 , year=

    Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence , author=. arXiv preprint arXiv:2604.24954 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    Pith/arXiv arXiv

  38. [38]

    2025 , eprint=

    Improving LLM Video Understanding with 16 Frames Per Second , author=. 2025 , eprint=

    2025

  39. [39]

    arXiv preprint arXiv:2406.16852 , year=

    Long context transfer from language to vision , author=. arXiv preprint arXiv:2406.16852 , year=

    Pith/arXiv arXiv

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Streaming long video understanding with large language models , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    2025 , eprint=

    Aligned Better, Listen Better for Audio-Visual Large Language Models , author=. 2025 , eprint=

    2025

  42. [42]

    arXiv preprint arXiv:2511.10289 , year=

    Music flamingo: Scaling music understanding in audio language models , author=. arXiv preprint arXiv:2511.10289 , year=

    arXiv

  43. [43]

    Yu, Fan and Zhang, Shiliang and Fu, Yihui and Xie, Lei and Zheng, Siqi and Du, Zhihao and Huang, Weilong and Guo, Pengcheng and Yan, Zhijie and Ma, Bin and Xu, Xin and Bu, Hui , booktitle=. M2. 2022 , organization=

    2022

  44. [44]

    The Second

    Vincent, Emmanuel and Barker, Jon and Watanabe, Shinji and Le Roux, Jonathan and Nesta, Francesco and Matassoni, Marco , booktitle=. The Second. 2013 , address=

    2013

  45. [45]

    2024 IEEE Spoken Language Technology Workshop (SLT) , pages=

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation , author=. 2024 IEEE Spoken Language Technology Workshop (SLT) , pages=. 2024 , organization=

    2024

  46. [46]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Must: The first dataset and unified framework for multispectral uav single object tracking , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  47. [47]

    Computer speech & language , volume=

    MuST-C: A multilingual corpus for end-to-end speech translation , author=. Computer speech & language , volume=. 2021 , publisher=

    2021

  48. [48]

    arXiv preprint arXiv:2007.10310 , year=

    Covost 2 and massively multilingual speech-to-text translation , author=. arXiv preprint arXiv:2007.10310 , year=

    Pith/arXiv arXiv 2007

  49. [49]

    Oriental COCOSDA 2017 , year=

    AIShell-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline , author=. Oriental COCOSDA 2017 , year=

    2017

  50. [50]

    2018 , howpublished=

    aidatatang\_200zh: A Free. 2018 , howpublished=

    2018

  51. [51]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  52. [52]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Harmonyset: A comprehensive dataset for understanding video-music semantic alignment and temporal synchronization , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  53. [53]

    2016 , eprint=

    Movie Description , author=. 2016 , eprint=

    2016

  54. [54]

    2024 , eprint=

    MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions , author=. 2024 , eprint=

    2024

  55. [55]

    2022 , eprint=

    MovieCLIP: Visual Scene Recognition in Movies , author=. 2022 , eprint=

    2022

  56. [56]

    2024 , eprint=

    MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions , author=. 2024 , eprint=

    2024

  57. [57]

    2022 , eprint=

    Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions , author=. 2022 , eprint=

    2022

  58. [58]

    2024 , eprint=

    InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation , author=. 2024 , eprint=

    2024

  59. [59]

    2023 , eprint=

    VidChapters-7M: Video Chapters at Scale , author=. 2023 , eprint=

    2023

  60. [60]

    Abu-El-Haija, Sami and Kothari, Nisarg and Lee, Joonseok and Natsev, Paul and Toderici, George and Varadarajan, Balakrishnan and Vijayanarasimhan, Sudheendra , journal=

  61. [61]

    and Ellis, Daniel P

    Gemmeke, Jort F. and Ellis, Daniel P. W. and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R. Channing and Plakal, Manoj and Ritter, Marvin , booktitle=. Audio. 2017 , address=

    2017

  62. [62]

    2023 , eprint=

    Sigmoid Loss for Language Image Pre-Training , author=. 2023 , eprint=

    2023

  63. [63]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  64. [64]

    1993 , howpublished =

    Switchboard-1 Release 2 (LDC97S62) , author =. 1993 , howpublished =

    1993

  65. [65]

    2023 , eprint=

    Joint Audio and Speech Understanding , author=. 2023 , eprint=

    2023

  66. [66]

    2024 , eprint=

    MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark , author=. 2024 , eprint=

    2024

  67. [67]

    2025 , eprint=

    MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix , author=. 2025 , eprint=

    2025

  68. [68]

    2026 , eprint=

    MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark , author=. 2026 , eprint=

    2026

  69. [69]

    2019 , eprint=

    ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering , author=. 2019 , eprint=

    2019

  70. [70]

    2024 , eprint=

    LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding , author=. 2024 , eprint=

    2024

  71. [71]

    2026 , eprint=

    WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs , author=. 2026 , eprint=

    2026

  72. [72]

    2026 , eprint=

    Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities , author=. 2026 , eprint=

    2026

  73. [73]

    2024 , eprint=

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution , author=. 2024 , eprint=

    2024

  74. [74]

    2024 , eprint=

    Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion , author=. 2024 , eprint=

    2024

  75. [75]

    2024 , eprint=

    InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks , author=. 2024 , eprint=

    2024

  76. [76]

    2024 , eprint=

    LongVILA: Scaling Long-Context Visual Language Models for Long Videos , author=. 2024 , eprint=

    2024

  77. [77]

    2025 , eprint=

    NVILA: Efficient Frontier Visual Language Models , author=. 2025 , eprint=

    2025

  78. [78]

    2024 , eprint=

    VILA: On Pre-training for Visual Language Models , author=. 2024 , eprint=

    2024

  79. [79]

    2025 , eprint=

    Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction , author=. 2025 , eprint=

    2025

  80. [80]

    2025 , eprint=

    Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction , author=. 2025 , eprint=

    2025

Showing first 80 references.