Pith. sign in

REVIEW 5 major objections 8 minor 62 references

The paper claims that fine-grained, event-level control of video sound can be achieved by a training-free pipeline that first converts video and instructions into a symbolic plan of sounding events, then renders each event with a text-to-au

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:15 UTC pith:3LMCLTTU

load-bearing objection Useful new task/benchmark and a training-free agent that likely improves instruction following, but the headline controllability gain rests on an unvalidated same-model detector, so treat the numbers as provisional. the 5 major comments →

arxiv 2512.24731 v2 pith:3LMCLTTU submitted 2025-12-31 cs.CV

EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

Bingxuan Li , Yiming Cui , Yicheng He , Yiwei Wang , Shu Zhang , Longyin Wen , Yulei Niu This is my paper
classification cs.CV
keywords video-to-audio generationevent-centric controlhierarchical controlsounding event representationcontrollable sound generationbenchmarkagentic frameworkslow-fast thinking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that video-grounded sound generation should be formulated as an event-level rather than video-level task, and that a training-free agentic pipeline—one that reasons about each sounding event, turns it into a symbolic tuple (time, description, properties), and renders each event separately—can follow fine-grained instructions about timing, timbre, and volume far better than monolithic video-to-audio models. Why care: current models are visually dominant, ignore text instructions, and cannot locate 'the second meow' or 'insert a one-second magic explosion at 00:07'. The paper introduces a new benchmark (EchoFoley-6k) with 6,000 video-instruction-annotation triplets and dense event annotations, plus an evaluation suite, and reports that its EchoVidia system outperforms baselines by 40.7% in controllability and 12.5% in perceptual quality.

Core claim

The central discovery claimed is that existing video-text-to-audio systems suffer from a visual-dominance bias: they sync sound to visible motion but largely ignore textual instructions, especially for fine-grained event-level edits. EchoVidia addresses this by first detecting and temporally localizing all sounding events in the video using a slow-fast thinking strategy (a fast 1-fps global pass plus a slow 16x-stretched pass for precise boundaries), converting the video and user instruction into a symbolic event plan e=(t,d,p), applying sound-design actions to insert, delete, or modify events, and then generating each event with a text-to-audio diffusion model and mixing the layers. The cla

What carries the argument

The symbolic sounding-event representation e=(t,d,p)—temporal interval, semantic description (subject-action-object), and controllable properties (pitch, volume, intensity, spatial)—is the interface that carries the argument. Slow-fast thinking (1-fps overview plus 16x slow-motion analysis) is the mechanism that yields precise event localization, and the agentic action pool (reasoning, sound-design, generation actions) converts the symbolic plan into audio. The EchoFoley-6k benchmark with its TempCtl, TimbCtl, and VolCtl metrics is the measurement that grounds the claimed gains.

Load-bearing premise

The temporal-control metric assumes that the audio-boundary detector's predictions are an unbiased measure of whether the sound actually starts and stops where instructed, even though the detector comes from the same model family as the system that plans the events and helps build the annotations.

What would settle it

Run a blind, human-annotated onsets/offsets study on the generated audio for a random sample of EchoFoley-6k, with annotators labeling when the requested event actually occurs; if the human-measured TempCtl gap between EchoVidia and the best baseline drops well below the reported +0.29, then the claimed temporal control advantage is partly a measurement artifact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If EchoVidia's gains are real, training-free composition of a video-language model and a text-to-audio model can replace monolithic VT2A systems for instruction-driven sound editing.
  • The symbolic event plan makes the generation process interpretable and editable: users can inspect and modify when, what, and how each sound is produced.
  • A benchmark with dense event-level annotations enables systematic comparison of controllability, going beyond video-level fidelity.
  • The documented visual-dominance bias implies that future VT2A models should be evaluated on conflicting text-visual instructions, not only on alignment.
  • The slow-fast strategy could be adopted in any VideoLLM-based temporal grounding task, not just audio.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the temporal metric uses an audio-boundary detector from the same model family that does the event reasoning and helped build the dataset, the reported +0.29 TempCtl advantage could be inflated by the detector recognizing EchoVidia's planned structure; a human-annotated boundary study would settle whether the control gain is auditory or computational.
  • The event-planning interface suggests a portable control pattern: any generative modality with time-localized content (music, dialogue, visual effects) could adopt the same triple of symbolic representation, slow-fast grounding, and per-event rendering.
  • A testable extension would train the event-plan controller end-to-end, which could reduce the 120–270-second per-sample reasoning latency while preserving the controllability gains.
  • The benchmark's 500 unique videos and 6,000 instructions derived from LLM-proposed stories may under-represent rare or abstract sound concepts; expanding to user-generated prompts could change the relative ordering of methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces EchoFoley, a video-grounded sound generation task with event-centric hierarchical control (instance, group, video levels) and a symbolic event representation (timestamp, description, properties). It constructs EchoFoley-6k, a benchmark of 6,000 video–instruction pairs with event annotations, and proposes EchoVidia, a training-free agentic pipeline that uses Gemini 2.5 Pro for event reasoning and Stable Audio Open for synthesis. Experiments report large gains over VT2A baselines, e.g., TempCtl 0.72 vs. 0.43, TimbCtl 0.78 vs. 0.48, VolCtl 0.75 vs. 0.69, and an averaged 40.7% controllability improvement, plus human Instruction Adherence of 3.80 vs. 2.60. The paper also evaluates VideoLLMs on sounding-event detection and localization.

Significance. If the quantitative claims hold, the paper makes a useful contribution: a well-motivated task formulation, a dense event-annotation benchmark with human refinement, and a training-free framework that demonstrates how an LLM agent plus TTA models can achieve fine-grained temporal and timbre control. The event-awareness study (Task 1/2) is a welcome addition. However, the main headline numbers depend on an automatic temporal-control metric whose detector is the same model family used to build the dataset and plan the method's outputs, and several reported numbers are internally inconsistent. The core idea is sound and worth pursuing, but the evidence as currently presented does not yet support the claimed margins.

major comments (5)
  1. [§4.3, Appendix A.4.1, Table 2] TempCtl is computed with a Gemini 2.5 Pro boundary detector on generated audio, while EchoVidia uses Gemini 2.5 Pro for event planning and Gemini/GPT-5 were used to propose dataset events. EchoVidia renders each event from the same textual description and mixes it at planned timestamps, so its outputs are structurally aligned with the detector's query; baselines produce holistic audio where the detector may force a 'most prominent occurrence.' No validation of Gemini's boundary predictions against human-annotated onsets/offsets on generated audio is provided. Since the +0.29 TempCtl gap drives much of the 40.7% controllability claim, this is a load-bearing risk. Please validate the detector on a human-labeled sample for all models, or use an independent audio-event detector, and report agreement; if bias exists, re-evaluate the claim.
  2. [§4.3 (Timbre Control) vs. Appendix A.4.2] The main-text definition of TimbCtl states the audio segment is 'extracted from the video's original audio A,' not from the generated audio. If taken literally, TimbCtl would be independent of generation quality and identical for all models. The pseudocode in A.4.2 correctly uses the generated waveform A_hat, and Table 2's variation implies the implementation followed the pseudocode, but the main text is an internal contradiction. This is one of the three controllability metrics underlying the headline result, so the exact input waveform must be specified unambiguously.
  3. [§6.2, Table 2] The summary of Table 2 is inconsistent with the table. (i) 'consistently outperforming the strongest baseline by +1.20, +0.40, and +0.32' is wrong: HunyuanVideo-Foley-xl has A–V Coherence 4.20 vs. EchoVidia's 3.93, so EchoVidia is lower; Perceptual Quality difference is 3.79–3.73 = +0.06, not +0.32. (ii) The claimed '+0.19' for volume/depth is 0.75–0.69 = +0.06. (iii) 'roughly 55%' average improvement differs from the abstract's 40.7%, which itself equals (0.29+0.30+0.06)/(0.43+0.48+0.69) ≈ 40.6%. These numerical errors overstate the result and must be corrected.
  4. [Abstract, §4.2, Table 1] The benchmark scale is inconsistent: the abstract and §4.2 state '42,000 fine-grained sounding event annotations,' while Table 1 reports Total Sounding Events 3,612 and 7.2 events per video (≈3,600). If the 42,000 refers to event-instruction associations (6,000 instructions × ~7 events), this must be stated explicitly; if it is the number of unique annotated events, Table 1 is wrong. This inconsistency affects the benchmark's stated scale and should be resolved.
  5. [§5, Table 2, Appendix A.6] Main comparative results are reported as single point estimates with no confidence intervals, standard deviations, or significance tests. A.6 states automatic evaluation uses 100 randomly selected cases and human evaluation uses 50 pairs, but the table does not indicate variance across cases or runs. Human evaluation has 6 raters and kappa 0.62, but no per-metric dispersion is given. Given the small human sample and the fact that some gaps are small (e.g., Perceptual Quality 3.79 vs. 3.73), statistical support is needed before claiming a 'substantial margin' on every dimension.
minor comments (8)
  1. [§4.3, Eq. (TempCtl)] The displayed formula for TempCtl writes a sum without 1/|C| normalization, while the pseudocode in A.4.1 correctly averages. Make the normalization explicit in the main text; the same issue appears in the VolCtl display.
  2. [§6.2] The metric is called 'DepthCtl' in §6.2 but 'VolCtl' and 'V olume Control' elsewhere; use a consistent name.
  3. [§4.3] Typos: 'loundness' should be 'loudness'; 'V olume' formatting; 'V olumeCtl' / 'DepthCtl' naming.
  4. [§6.3] Section header 'Border Impact' is a typo for 'Broader Impact'.
  5. [§4.1] The 'PE Video Dataset' is not defined or referenced in the bibliography; please provide a citation.
  6. [Appendix A.4.1] The detector is called 'AudioLLM-based' in §4.3 but is Gemini 2.5 Pro, a multimodal LLM; clarify that it is an audio-language model, not a dedicated event-boundary detector, and describe its prompt and decoding in more detail.
  7. [Appendix A.3.1] Reference [2] for GPT-5 appears to be missing from the reference list; the citation is incomplete.
  8. [Appendix B.1] The slow–fast description 'downsample to 16 fps, then temporally stretch by 16x to obtain the video at 1 fps' is confusing; please clarify the actual frame rates presented to the model.

Circularity Check

1 steps flagged

TempCtl is scored by Gemini 2.5 Pro locating event descriptions in audio that EchoVidia generates from those same descriptions at Gemini-planned timestamps, making the headline controllability gain a same-model self-detection loop rather than an independent temporal-control measurement.

specific steps
  1. self definitional [Section 4.3 (Temporal Control) and Appendix A.4.1; cross-ref Section 6.1 and Appendix B.3.2]
    "For temporal localization, we directly query a Gemini2.5-pro[11] with the generated audio segment and the textual description of the event. Gemini is prompted to determine when the described sound first appears and when it ends, returning a predicted start and end time in seconds. ... If the event occurs multiple times, choose the occurrence that best matches the description. If unsure, choose the most prominent occurrence."

    EchoVidia's inference procedure passes its finalized symbolic event plan to the sound generation module, and Appendix B.3.2 states 'Event-level segments are generated individually and mixed with moviepy package.' Therefore every EchoVidia output contains a segment generated from the same event description and placed at the plan's t_start/t_end. TempCtl then asks Gemini 2.5 Pro to locate that exact description in the audio and scores IoU against the annotated interval. The same Gemini model family proposed the dataset's events (Sec. 4.1) and serves as EchoVidia's VideoLLM (B.3.2), and the prompt forces an interval even when the event is absent ('If unsure, choose the most prominent occurrence'). TempCtl thus measures Gemini's ability to recover its own plan from audio built to match that pl

full rationale

The central 40.7% controllability claim is driven primarily by TempCtl, where the evaluation loop is closed by the same model: Gemini 2.5 Pro proposes events in dataset curation, Gemini 2.5 Pro plans EchoVidia's events, and Gemini 2.5 Pro detects the planned events in EchoVidia's output for scoring. This is not an independent adjudication. TimbCtl uses CLAP and VolCtl uses loudness thresholds, which are external and provide partial independent evidence; the human Instruction Adherence score (3.80 vs 2.60) is also independent, though it is based on only 50 items and does not isolate temporal alignment. I do not find load-bearing self-citation circularity: the cited prior works by the authors (refs. [24], [50], [53]) appear only in related-work context and do not justify the central result. The dataset's human refinement stage also weakens, but does not eliminate, the same-model contamination, since the final annotations are human-corrected. The paper is therefore not fully circular, but its headline controllability advantage is partially built into the metric construction, warranting a score of 6 rather than 0-2.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claim rests on the benchmark annotations and the validity of the automatic control metrics. The loudness thresholds are calibrated to the development set; the remaining assumptions are domain assumptions about whether visible motion determines diegetic sound and whether Gemini-based localization can serve as an unbiased judge. No new physical entities are postulated; 'sounding events' are a data representation.

free parameters (2)
  • Loudness thresholds tau1, tau2 = not reported
    Used in VolCtl to map relative loudness to low/medium/high categories; the authors state the thresholds were calibrated on the development set (Appendix A.4.3), making the volume metric partly fitted to the benchmark.
  • Slow-fast video sampling rates (1 fps fast view, 16x slow-motion stretch) = 1 fps / 16 fps / 16x
    Hand-chosen hyperparameters of the slow-fast thinking strategy with no sensitivity analysis; the reported event-localization gains depend on these specific rates.
axioms (5)
  • domain assumption A sounding event can be faithfully captured by the tuple e=(t_start, t_end, subject/action/object, pitch/volume/intensity/spatial).
    The entire task definition and benchmark annotation scheme assume this symbolic representation is sufficient for creative audio control; Section 3.1 states it without empirical support.
  • domain assumption Human-verified, LLM-proposed instructions and event annotations form a valid ground truth for creative sound control.
    EchoFoley-6k is built from LLM-generated stories and event proposals refined by human annotators (Section 4.1); if the LLM proposals systematically miss plausible sounds, the benchmark is biased.
  • domain assumption Visible motion is sufficient to infer all diegetic non-ambient sounds, and off-screen or ambient sounds can be ignored.
    The video filtering and annotation instructions remove videos with ambiguous or off-screen sounds (Appendix A.3.1), so the benchmark and tasks only measure visually grounded events.
  • ad hoc to paper Gemini 2.5 Pro's localization of events in generated audio is a reliable and unbiased oracle for temporal control.
    TempCtl relies on a Gemini-based onset/offset predictor (Appendix A.4.1) with no validation against human-annotated boundaries, yet it is used to measure the central controllability claim.
  • domain assumption Stable Audio Open 1.0 can render event-level prompts with sufficient fidelity for the event-centric mixing pipeline.
    EchoVidia's generation stage decomposes the soundtrack into per-event Stable Audio generations; the framework's final quality is bounded by this assumption, acknowledged in the limitations section.

pith-pipeline@v1.3.0-alltime-deepseek · 21014 in / 12641 out tokens · 137870 ms · 2026-08-03T13:15:25.378272+00:00 · methodology

0 comments
read the original abstract

Sound effects build an essential layer of multimodal storytelling, shaping the emotional atmosphere and the narrative semantics of videos. Despite recent advancement in video-text-to-audio (VT2A), the current formulation faces three key limitations: First, an imbalance between visual and textual conditioning that leads to visual dominance; Second, the absence of a concrete definition for fine-grained controllable generation; Third, weak instruction understanding and following, as existing datasets rely on brief categorical tags. To address these limitations, we introduce EchoFoley, a new task designed for video-grounded sound generation with both event level local control and hierarchical semantic control. Our symbolic representation for sounding events specifies when, what, and how each sound is produced within a video or instruction, enabling fine-grained controls like sound generation, insertion, and editing. To support this task, we construct EchoFoley-6k, a large-scale, expert-curated benchmark containing over 6,000 video-instruction-annotation triplets. Building upon this foundation, we propose EchoVidia a sounding-event-centric agentic generation framework with slow-fast thinking strategy. Experiments show that EchoVidia surpasses recent VT2A models by 40.7% in controllability and 12.5% in perceptual quality.

Figures

Figures reproduced from arXiv: 2512.24731 by Bingxuan Li, Longyin Wen, Shu Zhang, Yicheng He, Yiming Cui, Yiwei Wang, Yulei Niu.

Figure 1
Figure 1. Figure 1: Motivation of EchoFoley. In creative storytelling, sound shapes the story we perceive. Given a silent video, generat￾ing audio that matches this story-shaped imagination requires fine￾grained control over how each sound is crafted and transformed over time. We formulate such task with event-centric hierarchical control (e.g., instance control, group control), and propose effec￾tive solution. even enabling … view at source ↗
Figure 2
Figure 2. Figure 2: Data Curation Pipeline of EchoFoley-6k 4.1. Data Curation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average controllability scores for different levels of con [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Task 2: Sounding Event Localization. Intersection￾over-Union between predicted and ground-truth temporal spans, computed only on correctly detected events. els exhibit notable boundary drift—indicating that while they often detect which events occur, they struggle to match the precise onset and offset times. 6. Method: EchoVidia To address the limitations of existing video-to-audio gen￾eration models on la… view at source ↗
Figure 7
Figure 7. Figure 7: EchoVidia Inference Procedure an action pool—a set of 12 atomic operations spanning vi￾sual reasoning and sound generation (see appendix for more details). Within this pool, video reasoning actions identify sounding events, retrieve temporal cues, and crop relevant visual segments. Sound design actions allow the agent to add, remove, or modify event representations, controlling their semantic attributes an… view at source ↗
Figure 8
Figure 8. Figure 8: Human Evaluation UI B. Details on Method: EchoVidia This section provides additional details of the proposed EchoVidia framework, including the slow–fast thinking strategy, action-pool architecture, implementation details, and limitations. B.1. Slow–Fast Thinking Strategy Details The Slow–Fast (SF) Thinking strategy is designed to compensate for the limited sounding-event awareness found in current VideoLL… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 8 linked inside Pith

  1. [1]

    https : / / sound - effects

    BBC Sound Effects — sound-effects.bbcrewind.co.uk. https : / / sound - effects . bbcrewind . co . uk. [Accessed 22-10-2025]. 2

    2025

  2. [2]

    Enumerate all visually identifiable actions that can produce sound

  3. [3]

    Perception encoder: The best visual embeddings are not at the output of the net- work

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the net- work. InNIPS, 2025. 4

    2025

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  5. [5]

    Soundingactions: Learning how actions sound from narrated egocentric videos

    Changan Chen, Kumar Ashutosh, Rohit Girdhar, David Har- wath, and Kristen Grauman. Soundingactions: Learning how actions sound from narrated egocentric videos. InCVPR,

  6. [6]

    Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos

    Changan Chen, Puyuan Peng, Ami Baid, Sherry Xue, Wei- Ning Hsu, David Harwath, and Kristen Grauman. Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos. InECCV, 2024. 2

    2024

  7. [7]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zis- serman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020. 2, 4

    2020

  8. [8]

    Yingsound: Video-guided sound ef- fects generation with multi-modal chain-of-thought controls

    Zihao Chen, Haomin Zhang, Xinhan Di, Haoyu Wang, Sizhe Shan, Junjie Zheng, Yunming Liang, Yihan Fan, Xinfa Zhu, Wenjie Tian, et al. Yingsound: Video-guided sound ef- fects generation with multi-modal chain-of-thought controls. arXiv preprint arXiv:2412.09168, 2024. 2

    Pith/arXiv arXiv 2024

  9. [9]

    Video-guided foley sound generation with multimodal con- trols

    Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Ni- eto, David Bourgin, Andrew Owens, and Justin Salamon. Video-guided foley sound generation with multimodal con- trols. InCVPR, 2025. 2

    2025

  10. [10]

    Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis. InCVPR, 2025. 2, 5

    2025

  11. [11]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 3, 2, 13

    Pith/arXiv arXiv 2025

  12. [12]

    Stable audio open

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. InICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 13

    2025

  13. [13]

    Fine-grained audio–visual event localization

    Baoyu Fan, Lu Liu, Xiaochuan Li, Runze Zhang, Liang Jin, and Jin Zhang. Fine-grained audio–visual event localization. IEEE Transactions on Neural Networks and Learning Sys- tems, 2025. 3

    2025

  14. [14]

    Vita: Towards open-source interactive omni multimodal llm.arXiv:2408.05211, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm.arXiv:2408.05211, 2024. 3

    Pith/arXiv arXiv 2024

  15. [15]

    TALL: temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via language query. In ICCV, 2017. 3

    2017

  16. [16]

    Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception

    Junyu Gao, Mengyuan Chen, and Changsheng Xu. Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception. InCVPR, 2023. 3

    2023

  17. [17]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. InICASSP, 2017. 2

    2017

  18. [18]

    Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos. InCVPR, 2025. 3

    2025

  19. [19]

    A dataset for audio-visual sound event detection in movies

    Rajat Hebbar, Digbalay Bose, Krishna Somandepalli, Veena Vijai, and Shrikanth Narayanan. A dataset for audio-visual sound event detection in movies. InICASSP, 2023. 3

    2023

  20. [20]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. Localizing mo- ments in video with natural language. InICCV, 2017. 3

    2017

  21. [21]

    Egocentric audio-visual object localization

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. InCVPR,

  22. [22]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. InNAACL-HLT, 2019. 2

    2019

  23. [23]

    Berg, and Mohit Bansal

    Jie Lei, Tamara L. Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. InNIPS, 2021. 3

    2021

  24. [24]

    Metal: A multi-agent framework for chart generation with test-time scaling

    Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, and Nanyun Peng. Metal: A multi-agent framework for chart generation with test-time scaling. InACL, 2025. 2

    2025

  25. [25]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06355, 2023. 3

    Pith/arXiv arXiv 2023

  26. [26]

    Hear-your-click: Interactive video-to-audio generation via object-aware contrastive audio-visual fine-tuning, 2025

    Yingshan Liang, Keyu Fan, Zhicheng Du, Yiran Wang, Qingyang Shi, Xinyu Zhang, Jiasheng Lu, and Peiwu Qin. Hear-your-click: Interactive video-to-audio generation via object-aware contrastive audio-visual fine-tuning, 2025. 2

    2025

  27. [27]

    Video-llava: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection. InEMNLP, 2024. 3

    2024

  28. [28]

    Thinksound: Chain-of- thought reasoning in multimodal large language models for audio generation and editing

    Huadai Liu, Jialei Wang, Kaicheng Luo, Wen Wang, Qian Chen, Zhou Zhao, and Wei Xue. Thinksound: Chain-of- thought reasoning in multimodal large language models for audio generation and editing. InNIPS, 2025. 2, 5

    2025

  29. [29]

    Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models. InNIPS, 2023. 2

    2023

  30. [30]

    Ave-clip: Audioclip-based multi-window temporal transformer for au- dio visual event localization

    Tanvir Mahmud and Diana Marculescu. Ave-clip: Audioclip-based multi-window temporal transformer for au- dio visual event localization. InWACV, 2023. 3

    2023

  31. [31]

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multi- modal research.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3339–3354, 2024. 2

    2024

  32. [32]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    Pith/arXiv arXiv

  33. [33]

    Audiogenie: A training-free multi-agent framework for diverse multimodality-to-multiaudio generation

    Yan Rong, Jinting Wang, Guangzhi Lei, Shan Yang, and Li Liu. Audiogenie: A training-free multi-agent framework for diverse multimodality-to-multiaudio generation. InACM Multimedia, 2025. 2, 5, 6

    2025

  34. [34]

    Foley control: Align- ing a frozen latent text-to-audio model to video.arXiv preprint arXiv:2510.21581, 2025

    Ciara Rowles, Varun Jampani, Simon Donn ´e, Shimon Vainer, Julian Parker, and Zach Evans. Foley control: Align- ing a frozen latent text-to-audio model to video.arXiv preprint arXiv:2510.21581, 2025. 2

    arXiv 2025

  35. [35]

    Photorealistic text-to-image dif- fusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lili Li, Jay Whang, Emily Denton, Gabriel Goh, Antoine Sablay- rolles, Ishan Misra, et al. Photorealistic text-to-image dif- fusion models with deep language understanding. InNIPS,

  36. [36]

    Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation.arXiv preprint arXiv:2508.16930, 2025

    Sizhe Shan, Qiulin Li, Yutao Cui, Miles Yang, Yuehai Wang, Qun Yang, Jin Zhou, and Zhao Zhong. Hunyuanvideo- foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation.arXiv preprint arXiv:2508.16930, 2025. 2, 5

    Pith/arXiv arXiv 2025

  37. [37]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. InCVPR,

  38. [38]

    Vidi: Large multimodal models for video un- derstanding and editing.arXiv preprint arXiv:2504.15681,

    Vidi Team, Celong Liu, Chia-Wen Kuo, Dawei Du, Fan Chen, Guang Chen, Jiamin Yuan, Lingxi Zhang, Lu Guo, Lusha Li, et al. Vidi: Large multimodal models for video un- derstanding and editing.arXiv preprint arXiv:2504.15681,

    Pith/arXiv arXiv

  39. [39]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. InECCV, 2018. 2

    2018

  40. [40]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. InECCV, 2020. 2

    2020

  41. [41]

    Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, Carleigh Wood, Ann Lee, and Wei-Ning Hsu. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. 2025. 5

    2025

  42. [42]

    Kling-foley: Multimodal diffu- sion transformer for high-quality video-to-audio generation

    Jun Wang, Xijuan Zeng, Chunyu Qiang, Ruilong Chen, Shiyao Wang, Le Wang, Wangjing Zhou, Pengfei Cai, Ji- ahui Zhao, Nan Li, et al. Kling-foley: Multimodal diffu- sion transformer for high-quality video-to-audio generation. arXiv preprint arXiv:2506.19774, 2025. 2

    arXiv 2025

  43. [43]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Tay- lor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InICASSP, 2023. 5, 4

    2023

  44. [44]

    Son- icvisionlm: Playing sound with vision language models

    Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Son- icvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 26866–26875, 2024. 2

    2024

  45. [45]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. InCVPR,

  46. [46]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. InCVPR,

  47. [47]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. InEMNLP, 2023. 3

    2023

  48. [48]

    Audio-synchronized visual animation

    Lin Zhang, Shentong Mo, Yijing Zhang, and Pedro Mor- gado. Audio-synchronized visual animation. InECCV, 2024. 2

    2024

  49. [49]

    Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds.arXiv preprint arXiv:2407.01494, 2024

    Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foley- crafter: Bring silent videos to life with lifelike and synchro- nized sounds.arXiv preprint arXiv:2407.01494, 2024. 2

    Pith/arXiv arXiv 2024

  50. [50]

    Reffly: Melody-constrained lyrics editing model

    Songyan Zhao, Bingxuan Li, Yufei Tian, and Nanyun Peng. Reffly: Melody-constrained lyrics editing model. InNAACL,

  51. [51]

    Temporal relational reasoning in videos

    Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Tor- ralba. Temporal relational reasoning in videos. InECCV,

  52. [52]

    Towards open-vocabulary audio-visual event localization

    Jinxing Zhou, Dan Guo, Ruohao Guo, Yuxin Mao, Jingjing Hu, Yiran Zhong, Xiaojun Chang, and Meng Wang. Towards open-vocabulary audio-visual event localization. InCVPR,

  53. [53]

    change the second meow into a lion roar

    Yu Zhou, Bingxuan Li, Mohan Tang, Xiaomeng Jin, Te- Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, and Nanyun Peng. Contrastive visual data augmentation. In ICML, 2025. 2 Appendix Contents A . Details on Benchmark: EchoFoley-6k 2 A.1 . Data Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 A.2 . Comparison with...

    2025

  54. [54]

    Describe the global scene (1{2 sentences)

  55. [55]

    cat meow

    For each action, provide: - a short semantic label (e.g., "cat meow", "object impact") - the rough order index (1, 2, 3, ...) WITHOUT timestamps - a short justification (why it may produce sound) OUTPUT FORMAT (STRICT): EVENT_LIST = [ {index: i, label: "...", justification: "..."}, ... ] Do NOT include timestamps or any speculation unrelated to visible mo...

  56. [56]

    Merge events with similar semantics from both lists

  57. [57]

    Refine event timing using SLOW_VIEW when available

  58. [58]

    Assign timestamps (t_start, t_end) in seconds

  59. [59]

    ", action:

    Remove duplicates and ensure chronological ordering. OUTPUT FORMAT (STRICT): MERGED_EVENTS = [ {label: "...", t_start: x.xx, t_end: y.yy}, ... ] No explanations. Only the list above. Verification:re-query timestamps or re-evaluate event descriptions if inconsistencies or contradictions arise. SYSTEM ROLE: You are the Sounding Event Structuring Agent. You ...

  60. [60]

    Identify all referenced events

  61. [61]

    Apply the required edits using ONLY: - ADD_EVENT - DELETE_EVENT - MODIFY_DESCRIPTION - MODIFY_TIME - MODIFY_PROPERTIES

  62. [62]

    <text prompt>

    Validate chronological ordering and value ranges. OUTPUT FORMAT (STRICT): UPDATED_EVENT_PLAN = [ {t: (...), d: {...}, p: {...}}, ... ] No reasoning statements. Only the updated plan. Generation Prompt:specifying how the symbolic plan should be converted to audio-generation commands. SYSTEM ROLE: You are the Audio Generation Planner. You convert symbolic e...