Pith. sign in

REVIEW 3 major objections 4 minor 56 references

Audio remixing is a many-to-many problem, so a flow-matching generative model learns it better than a discriminative one.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:50 UTC pith:6CHNPTL6

load-bearing objection A solid method paper whose headline claim outruns its own ablations: the rollout-only model nearly matches the full system, so 'best addressed through generative modeling' is not actually supported. the 3 major comments →

arxiv 2602.03762 v4 pith:6CHNPTL6 submitted 2026-02-03 eess.AS cs.LG

Conditional Flow Matching for Visually-Guided Acoustic Highlighting

Hugo Malard , Gael Le Lan , Daniel Wong , David Lou Alon , Yi-Chiao Wu , Sanjeel Parekh This is my paper
classification eess.AS cs.LG
keywords acoustic highlightingflow matchinggenerative audioaudio-visual alignmentaudio remixingrollout losscross-modal conditioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that visually-guided acoustic highlighting—rebalancing the loudness of speech, music, and effects in a video's audio to match what is shown—is inherently a many-to-many distribution-transport problem, not a one-to-one regression. To exploit that, the authors cast the task as conditional flow matching, in which a vector field transports a poorly-balanced mix toward the distribution of well-balanced mixes under visual guidance. Two additions make it work: a rollout loss that backpropagates through the full generated trajectory, penalizing drift at the final step and teaching the model to self-correct; and an adapter that injects audio features into the visual encoder so the conditioning pathway itself selects the source to highlight. On the Muddy Mix benchmark, their model (VisAH-FM) improves over the prior discriminative baseline on all reported metrics, supporting the paper's conclusion that generative modeling is the right framing for audio remixing.

Core claim

The central discovery is that a generative formulation with trajectory-level supervision beats the discriminative baseline for acoustic highlighting. Concretely, the model learns a time-dependent velocity field v_θ(x_t, t, c) that interpolates from a degraded mix x_0 to a well-balanced mix x_1, conditioned on visual features c. The rollout loss adds a term ||x̂_T − x_T||², where x̂_T is the output after T=4 recurrent integration steps, which stabilizes long-range trajectories and prevents early errors in source selection from compounding. Additionally, a cross-modal adapter injects CLAP audio embeddings into intermediate CLIP layers via cross-attention, letting the conditioning encoder ident

What carries the argument

The load-bearing object is the conditional flow-matching vector field, defined by the interpolant x_t = (1−t)x_0 + t x_1 and the ODE dx_t/dt = (x_1 − x_t)/(1 − t). A U-Net adapted from the DEMUCS-based VisAH backbone estimates this field as v_θ(x_t, t, c) = x_0 − u_θ(x_t, t, c), where u_θ is the model's direct prediction, enabling warm-start from pretrained discriminative weights. Two mechanisms carry the improvement: (1) the rollout loss, which runs the full T-step flow during training and penalizes the squared error between the final prediction and ground truth, exposing the model to its own intermediate errors (exposure-bias mitigation); and (2) the conditioning adapter, which computes cr

Load-bearing premise

The central claim rests on the assumption that the Muddy Mix pseudo-data generation (separation, adjustment, remixing) and the third-party metrics (PASST KL divergence, ImageBind score, and separator-based loudness difference) faithfully reflect what human listeners perceive as good acoustic highlighting.

What would settle it

A controlled listening test with a larger panel (e.g., 50+ participants) on real-world poorly-balanced videos—not synthetic Muddy Mix—where the discriminative baseline is preferred at or above chance would directly contradict the paper's conclusion; alternatively, a metric-inversion result where VisAH-FM improves KLD and ImageBind but lowers subjective alignment would show the metrics do not track perceptual quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the generative framing is right, the same distribution-matching objective should transfer to other ambiguous audio tasks, such as source separation and style transfer, where discriminative regression is known to struggle.
  • The rollout loss — backpropagating through the full flow with a final-step MSE — is presented as a general countermeasure to exposure bias; the paper shows it outperforms consistency losses and input-noise injection in this setting.
  • Injecting audio features into the visual encoder (CLAP into CLIP) is shown to be more effective than adding text captions, and the paper notes it avoids the cost of a large text encoder, making the method cheaper at inference.
  • With only four integration steps, the flow-based model adds minor compute beyond the one-time encoder passes, suggesting the approach is practical for video editing pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the model learns a continuous trajectory between mixes, a natural extension not explored in the paper is semantic interpolation: stopping the flow at an intermediate step could give users controllable 'amount of highlight,' which the linearity analysis in semantic space supports.
  • The on-the-fly random remixing during training effectively augments the data distribution; this suggests that training-time stochasticity in the conditioning source selection is a cheap regularizer that may transfer to other augmentations like random gain or equalization perturbations.
  • The ablation showing text features add nothing when audio is injected hints that a single shared audio-visual embedding (rather than separate text and audio branches) could simplify the conditioning stack; a testable extension is replacing CLIP/CLAP with a jointly trained audio-visual encoder.
  • The paper's own limitation—training on paired artificially-mixed data—points to a concrete next step: adapting the CFM framework to unpaired data via optimal-transport or adversarial alignment, which would unlock real-world recordings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VisAH-FM, a conditional flow matching framework for visually-guided acoustic highlighting. It builds on the discriminative VisAH/DEMUCS backbone, adds a flow-matching objective with time-step conditioning, a rollout loss that backpropagates through T=4 integration steps to penalize endpoint drift, and a cross-modal adapter that injects CLAP audio features into intermediate CLIP layers. Experiments on the Muddy Mix dataset report consistent improvements over the VisAH baselines (CLIP and T5) across semantic, signal, and remixing metrics, with ablations of the rollout loss, conditioning module, and hyperparameters, plus a small subjective test and trajectory analyses.

Significance. If the reported gains are robust, the paper makes a useful empirical contribution: it shows that iterative self-correcting refinement is highly effective for audio remixing, and it provides a practical early-fusion adapter for audio-visual conditioning. The trajectory analyses in Fig. 4 and Table 12 are informative and go beyond a simple 'it works' evaluation. However, the headline conclusion—that visually-guided audio remixing is 'best addressed through generative modeling'—is not supported by the paper's own ablation, because a model trained with only the rollout loss (no flow-matching objective) nearly matches the full system. The contribution is better characterized as an effective iterative-refinement architecture with a useful conditioning module; this is still a valid contribution, but it is distinct from the generative-modeling claim made in the Abstract and Conclusion.

major comments (3)
  1. [§3.2.1 / Table 3] The 'Rollout' row is the critical control for the paper's central claim. This model is trained with only the rollout loss (Eq. 5) and no CFM loss (Eq. 4), so it is a deterministic recurrent refinement model, not a generative distribution-matching model. On standard CLIP conditioning it attains KLD 9.92 and LDif 7.71, which are within a small margin of the full FM+Rollout model (KLD 9.79, LDif 7.87) and far better than the discriminative VisAH CLIP baseline (KLD 11.37, LDif 9.66). The paper itself acknowledges in §3.2.1 that 'even without the intermediate supervision and linearity prior imposed by flow matching, the model achieves strong performance,' yet the Abstract and Conclusion attribute the gains to generative modeling. Without error bars or significance tests, the small difference between Rollout and FM+Rollout cannot be distinguished from noise. To sustain the generative-modeling
  2. [§3.1 / Table 5] The main results in Table 1 compare VisAH-FM trained with on-the-fly random degraded mixes against VisAH baselines trained on the fixed Muddy Mix dataset. This confounds the choice of model class with a change in training data distribution. The fixed-dataset experiment in Table 5 is an attempt to control for this, but it only reports the 'VisAH Text' baseline, not the 'VisAH CLIP' baseline used in Table 1, and the gains are smaller. In that controlled comparison, the 'Rollout' row (no CFM loss) is worse than the discriminative baseline on KLD (11.38 vs 11.02) and only marginally better on LDif (8.87 vs 9.23), which is inconsistent with the main-paper story that rollout-only refinement is responsible for the improvement. A proper controlled comparison over the same fixed dataset, including both VisAH variants, is needed to support the central claim.
  3. [§3.3 and all tables] The subjective test uses only 5 participants and 8 videos (60% vs 10% win rate), which is far too small to be conclusive. In addition, all quantitative tables report point estimates without error bars, confidence intervals, or repeated-seed variance. This is especially problematic for the small differences used to argue for the rollout loss and the adapter (e.g., KLD 9.79 vs 9.92 in Table 3; KLD 9.70 vs 9.79 in Table 2). Please add variability estimates and, if possible, a larger perceptual study.
minor comments (4)
  1. [§2.2 / Eq. (2)] Typo: 'velocity field field' should be 'velocity field'. Also, Section 1 contains 'the the data manifold' and should be corrected.
  2. [§2.2.2 / Eqs. (6)-(7)] Equation (7) uses 'A' in 'adapter(Fk, E, A)' but the right-hand side and Eq. (6) do not show how 'A' enters; clarify the notation.
  3. [Appendix Table 8] The Env value of 0.32 for the 3-step row appears to be an order-of-magnitude typo (other rows are around 2.8); please check and correct.
  4. [Throughout] Notation is inconsistent: 'PaSST' vs 'PASST', 'PMLR' vs 'PmLR'. Also, Tables 3 and 10 duplicate the same ablation; consider consolidating.

Circularity Check

0 steps flagged

No constructed circularity; central claim is weakened by the paper's own rollout-only ablation, but that is an attribution/support issue, not a definitional or self-citation circularity.

full rationale

No circular step in the derivation chain. The target (well-balanced audio) and conditioning (poorly balanced audio plus video) come from the Muddy Mix pseudo-data process, and all reported metrics (PASST-KLD, ImageBind score, separator-based LDif) are computed with external models against ground-truth test audio; no evaluation quantity is defined in terms of VisAH-FM's own outputs or fitted parameters. The paper builds on the authors' prior VisAH work [14] for architecture and baseline, but this is an empirical comparison with a published baseline, not an unverified self-citation that carries the argument. The main caveat is flagged explicitly in Section 3.2.1: Table 3's 'Rollout' row, trained with Eq. 5 only and no flow-matching loss, reaches KLD 9.92 / LDif 7.71 versus 9.79 / 7.87 for the full model, and the text concedes 'even without the intermediate supervision and linearity prior imposed by flow matching, the model achieves strong performance.' This undermines the causal attribution of gains to generative flow matching, but it is not circularity: the result is not equivalent to its inputs by construction, and no fitted parameter is renamed as a prediction. Self-citations such as [14] and [47] are not load-bearing in a circular way. The small subjective test is an evaluation limitation, not a circularity concern.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on standard flow matching math plus domain assumptions about the dataset and metrics. No new physical entities are introduced. The free parameters are standard training and loss weights, all ablated or swept.

free parameters (5)
  • rollout coefficient λ = 0.3 (swept 0.1–1.0)
    Weight on the auxiliary endpoint MSE loss; 0.3 chosen as best in ablations (Table 4/11); results stable across range.
  • number of flow steps T = 4
    Discretization of flow time; set to 4 following prior work [4] and used for both training and inference; varied in Table 4/8.
  • bridge-matching noise std = 1e-5
    Baseline hyperparameter for noise-injected trajectory training.
  • adapter insertion layer = layer 18 of CLIP
    Chosen as the insertion point for the audio cross-modal adapter; no systematic search reported.
  • learnable adapter scale λ_E = initialized 0, learned
    Controls audio-feature contribution to CLIP features; initialized to 0 to preserve pretrained CLIP; trained.
axioms (5)
  • standard math Flow matching with linear interpolant xt=(1−t)x0+t x1 and conditional vector field (x1−xt)/(1−t) is a valid generative objective.
    Foundation of CFM (Eq. 1-3), cited to [23].
  • domain assumption Muddy Mix pseudo-data generation (separate sources, random adjust levels, remix) produces valid 'poorly balanced' vs 'well-balanced' pairs.
    Task definition relies on synthetic degradation reflecting real-world mixing problems; no real-world validation.
  • domain assumption PASST-KLD, ImageBind, and separation-based loudness difference correlate with perceived audio-visual highlighting quality.
    All conclusions about improvement are drawn from these metrics; no human evaluation beyond 5-participant/8-video test.
  • domain assumption Generative models are better suited to many-to-many remapping because both input and output distributions have mutual uncertainty.
    Motivated by prior work [28] but not directly tested for this task; an assumption framing the paper's argument.
  • domain assumption Pretrained CLIP/CLAP features contain sufficient audio-visual alignment for source selection.
    Conditioning quality depends on these frozen encoders; acknowledged in limitations.

pith-pipeline@v1.3.0-alltime-deepseek · 17091 in / 9308 out tokens · 93456 ms · 2026-08-03T04:50:02.187673+00:00 · methodology

0 comments
read the original abstract

Visually-guided acoustic highlighting seeks to rebalance audio in alignment with the accompanying video, creating a coherent audio-visual experience. While visual saliency and enhancement have been widely studied, acoustic highlighting remains underexplored, often leading to misalignment between visual and auditory focus. Existing approaches use discriminative models, which struggle with the inherent ambiguity in audio remixing, where no natural one-to-one mapping exists between poorly-balanced and well-balanced audio mixes. To address this limitation, we reframe this task as a generative problem and introduce a Conditional Flow Matching (CFM) framework. A key challenge in iterative flow-based generation is that early prediction errors -- in selecting the correct source to enhance -- compound over steps and push trajectories off-manifold. To address this, we introduce a rollout loss that penalizes drift at the final step, encouraging self-correcting trajectories and stabilizing long-range flow integration. We further propose a conditioning module that fuses audio and visual cues before vector field regression, enabling explicit cross-modal source selection. Extensive quantitative and qualitative evaluations show that our method consistently surpasses the previous state-of-the-art discriminative approach, establishing that visually-guided audio remixing is best addressed through generative modeling.

Figures

Figures reproduced from arXiv: 2602.03762 by Daniel Wong, David Lou Alon, Gael Le Lan, Hugo Malard, Sanjeel Parekh, Yi-Chiao Wu.

Figure 1
Figure 1. Figure 1: We propose to cast visually-guided acoustic highlight [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of rollout loss. Unlike the regular flow [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of VisAH-FM architecture: Building upon the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Three-fold analysis of the impact of the rollout loss. Combined with the flow matching loss, the rollout allows stable trajectory [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the behavior of the model trained with and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison between VisAH and VisAH-FM. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison of VisAH-FM models trained with and without rollout loss. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison between VisAH-FM and the discriminative VisAH. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 17 linked inside Pith

  1. [1]

    Stochastic interpolants: A unifying frame- work for flows and diffusions, 2023.URL https://arxiv

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Eijnden. Stochastic interpolants: A unifying frame- work for flows and diffusions, 2023.URL https://arxiv. org/abs/2303.08797, 3, 2023. 6

    Pith/arXiv arXiv 2023

  2. [2]

    Diffusion-based unsupervised audio-visual speech enhancement

    Jean-Eudes Ayilo, Mostafa Sadeghi, Romain Serizel, and Xavier Alameda-Pineda. Diffusion-based unsupervised audio-visual speech enhancement. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 2

    2025

  3. [3]

    D-flow: Differentiating through flows for controlled generation.arXiv preprint arXiv:2402.14017, 2024

    Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation.arXiv preprint arXiv:2402.14017, 2024. 4

    Pith/arXiv arXiv 2024

  4. [4]

    Lbm: Latent bridge match- ing for fast image-to-image translation.arXiv preprint arXiv:2503.07535, 2025

    Cl ´ement Chadebec, Onur Tasar, Sanjeev Sreetharan, and Benjamin Aubin. Lbm: Latent bridge match- ing for fast image-to-image translation.arXiv preprint arXiv:2503.07535, 2025. 3, 5, 6

    Pith/arXiv arXiv 2025

  5. [5]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101,

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101,

  6. [6]

    Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 28901–28911, 2025. 8

    2025

  7. [7]

    Samwise: Infusing wisdom in sam2 for text-driven video segmentation

    Claudia Cuttano, Gabriele Trivigno, Gabriele Rosi, Carlo Masone, and Giuseppe Averta. Samwise: Infusing wisdom in sam2 for text-driven video segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3395–3405, 2025. 4

    2025

  8. [8]

    Crome: cross-modal adapters for efficient multi- modal llm.arXiv preprint arXiv:2408.06610, 2024

    Sayna Ebrahimi, Sercan O Arik, Tejas Nama, and Tomas Pfister. Crome: cross-modal adapters for efficient multi- modal llm.arXiv preprint arXiv:2408.06610, 2024. 4

    Pith/arXiv arXiv 2024

  9. [9]

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech sepa- ration.arXiv preprint arXiv:1804.03619, 2018

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech sepa- ration.arXiv preprint arXiv:1804.03619, 2018. 9

    Pith/arXiv arXiv 2018

  10. [10]

    Visualvoice: Audio- visual speech separation with cross-modal consistency

    Ruohan Gao and Kristen Grauman. Visualvoice: Audio- visual speech separation with cross-modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15490–15500. IEEE, 2021. 9

    2021

  11. [11]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15180–15190, 2023. 5

    2023

  12. [12]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 4

    2020

  13. [13]

    Davis: High-quality audio-visual separation with generative diffusion models

    Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Davis: High-quality audio-visual separation with generative diffusion models. 2023. 2, 9

    2023

  14. [14]

    Learning to highlight audio by watching movies

    Chao Huang, Ruohan Gao, JMF Tsang, Jan Kurcius, Cagdas Bilen, Chenliang Xu, Anurag Kumar, and Sanjeel Parekh. Learning to highlight audio by watching movies. InProceed- ings of the Computer Vision and Pattern Recognition Confer- ence, pages 23925–23935, 2025. 1, 2, 3, 4, 5, 8

    2025

  15. [15]

    High-quality sound separation across diverse categories via visually-guided generative modeling.arXiv preprint arXiv:2509.22063, 2025

    Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. High-quality sound separation across diverse categories via visually-guided generative modeling.arXiv preprint arXiv:2509.22063, 2025. 9

    arXiv 2025

  16. [16]

    Mu- sic mixing style transfer: A contrastive learning approach to disentangle audio effects

    Junghyun Koo, Marco A Mart ´ınez-Ram´ırez, Wei-Hsiang Liao, Stefan Uhlich, Kyogu Lee, and Yuki Mitsufuji. Mu- sic mixing style transfer: A contrastive learning approach to disentangle audio effects. InICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 1–5. IEEE, 2023. 8

    2023

  17. [17]

    Efficient training of audio transformers with patchout.arXiv preprint arXiv:2110.05069, 2021

    Khaled Koutini, Jan Schl ¨uter, Hamid Eghbal-Zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout.arXiv preprint arXiv:2110.05069, 2021. 5, 7, 2

    Pith/arXiv arXiv 2021

  18. [18]

    Seeing through the conversation: Audio-visual speech separation based on diffusion model

    Suyeon Lee, Chaeyoung Jung, Youngjoon Jang, Jaehun Kim, and Joon Son Chung. Seeing through the conversation: Audio-visual speech separation based on diffusion model. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12632–12636. IEEE, 2024. 9

    2024

  19. [19]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846–11858, 2021. 1

    2021

  20. [20]

    Storm: A diffusion-based stochastic regen- eration model for speech enhancement and dereverberation

    Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusion-based stochastic regen- eration model for speech enhancement and dereverberation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2724–2737, 2023. 3

    2023

  21. [21]

    Av-nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Informa- tion Processing Systems, 36:37472–37490, 2023

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Informa- tion Processing Systems, 36:37472–37490, 2023. 5

    2023

  22. [22]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794–2804, 2023. 1

    2023

  23. [23]

    Flow matching for generative mod- eling.ICLR, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.ICLR, 2022. 2, 3

    2022

  24. [24]

    Let us build bridges: Understanding and extending diffusion gener- ative models.arXiv preprint arXiv:2208.14699, 2022

    Xingchao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion gener- ative models.arXiv preprint arXiv:2208.14699, 2022. 6

    Pith/arXiv arXiv 2022

  25. [25]

    Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection

    Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3042–3051, 2022. 1

    2022

  26. [26]

    Multi-source diffusion models for simultaneous music generation and sep- aration.arXiv preprint arXiv:2302.02257, 2023

    Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodol`a. Multi-source diffusion models for simultaneous music generation and sep- aration.arXiv preprint arXiv:2302.02257, 2023. 2

    Pith/arXiv arXiv 2023

  27. [27]

    Deep learning for black-box modeling of audio effects.Applied Sciences, 10(2):638, 2020

    Marco A Mart ´ınez Ram ´ırez, Emmanouil Benetos, and Joshua D Reiss. Deep learning for black-box modeling of audio effects.Applied Sciences, 10(2):638, 2020. 8

    2020

  28. [28]

    On discriminative vs

    Andrew Ng and Michael Jordan. On discriminative vs. gen- erative classifiers: A comparison of logistic regression and naive bayes.Advances in neural information processing sys- tems, 14, 2001. 2

    2001

  29. [29]

    Elucidating the exposure bias in diffusion models.arXiv preprint arXiv:2308.15321, 2023

    Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models.arXiv preprint arXiv:2308.15321, 2023. 4

    Pith/arXiv arXiv 2023

  30. [30]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    Pith/arXiv arXiv

  31. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 4

    2021

  32. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 4, 6

    2020

  33. [33]

    Model- ing nonlinear audio effects with end-to-end deep neural net- works

    Marco A Mart ´ınez Ram´ırez and Joshua D Reiss. Model- ing nonlinear audio effects with end-to-end deep neural net- works. InICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 171–175. IEEE, 2019. 8

    2019

  34. [34]

    Sequence level training with recurrent neural networks

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. InICLR, 2016. 4

    2016

  35. [35]

    Hybrid transformers for music source separation

    Simon Rouard, Francisco Massa, and Alexandre D ´efossez. Hybrid transformers for music source separation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE,

    2023

  36. [36]

    Source separation by flow matching.arXiv preprint arXiv:2505.16119, 2025

    Robin Scheibler, John R Hershey, Arnaud Doucet, and Henry Li. Source separation by flow matching.arXiv preprint arXiv:2505.16119, 2025. 2

    Pith/arXiv arXiv 2025

  37. [37]

    Physical modeling using recurrent neu- ral networks with fast convolutional layers

    Sebastian Schlecht, Julian Parker, Maximilian Sch ¨afer, and Rudolf Rabenstein. Physical modeling using recurrent neu- ral networks with fast convolutional layers. InInternational Conference on Digital Audio Effects, pages 138–145. DAFx,

  38. [38]

    Improved techniques for training consistency models.ICLR, 2024

    Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models.ICLR, 2024. 3

    2024

  39. [39]

    Consistency models.ICLR, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models.ICLR, 2023. 3

    2023

  40. [40]

    Style transfer of audio effects with differentiable signal pro- cessing.arXiv preprint arXiv:2207.08759, 2022

    Christian J Steinmetz, Nicholas J Bryan, and Joshua D Reiss. Style transfer of audio effects with differentiable signal pro- cessing.arXiv preprint arXiv:2207.08759, 2022. 8

    Pith/arXiv arXiv 2022

  41. [41]

    Audiox: Diffusion transformer for anything-to-audio generation.arXiv preprint arXiv:2503.10522, 2025

    Zeyue Tian, Yizhu Jin, Zhaoyang Liu, Ruibin Yuan, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. Audiox: Diffusion transformer for anything-to-audio generation.arXiv preprint arXiv:2503.10522, 2025. 8

    Pith/arXiv arXiv 2025

  42. [42]

    Improving and generalizing flow-based generative models with minibatch optimal transport.arXiv preprint arXiv:2302.00482, 2023

    Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport.arXiv preprint arXiv:2302.00482, 2023. 3

    Pith/arXiv arXiv 2023

  43. [43]

    Diff- mst: Differentiable mixing style transfer.arXiv preprint arXiv:2407.08889, 2024

    Soumya Sai Vanka, Christian Steinmetz, Jean-Baptiste Rolland, Joshua Reiss, and George Fazekas. Diff- mst: Differentiable mixing style transfer.arXiv preprint arXiv:2407.08889, 2024. 8

    Pith/arXiv arXiv 2024

  44. [44]

    Self-consistency improves chain of thought reason- ing in language models.arXiv preprint arXiv:2203.11171,

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reason- ing in language models.arXiv preprint arXiv:2203.11171,

    Pith/arXiv arXiv

  45. [45]

    A generalized band- split neural network for cinematic audio source separation

    Karn N Watcharasupat, Chih-Wei Wu, Yiwei Ding, Iroro Orife, Aaron J Hipple, Phillip A Williams, Scott Kramer, Alexander Lerch, and William Wolcott. A generalized band- split neural network for cinematic audio source separation. IEEE Open Journal of Signal Processing, 5:73–81, 2023. 5

    2023

  46. [46]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 6

    2022

  47. [47]

    Flowdec: A flow-based full-band general audio codec with high perceptual quality.arXiv preprint arXiv:2503.01485,

    Simon Welker, Matthew Le, Ricky TQ Chen, Wei-Ning Hsu, Timo Gerkmann, Alexander Richard, and Yi-Chiao Wu. Flowdec: A flow-based full-band general audio codec with high perceptual quality.arXiv preprint arXiv:2503.01485,

    Pith/arXiv arXiv

  48. [48]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 4, 5

    2023

  49. [49]

    Visually informed binaural au- dio generation without binaural audios

    Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural au- dio generation without binaural audios. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15485–15494, 2021. 5

    2021

  50. [50]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. InProceedings of the European conference on com- puter vision (ECCV), pages 570–586, 2018. 9 Conditional Flow Matching for Visually-Guided Acoustic Highlighting Supplementary Material

    2018

  51. [51]

    Table 5 presents the results obtained on this variant of the dataset

    Experiments on the Fixed Muddy Mix Dataset To evaluate the impact of on-the-fly source sampling (i.e., randomly selecting which source to enhance for each audio example at each iteration), we trained the VisAH-FM model on the fixed, pre-generated Muddy Mix dataset, that was originally proposed. Table 5 presents the results obtained on this variant of the ...

  52. [52]

    Additional Experiments 7.1. Ablation of the Time Conditioning When the timestep is fed only to the latent transformer, the encoder does not have access to temporal information, which could theoretically lead to suboptimal representa- tions. Table 6 compares a model using timestep condi- tioning exclusively within the latent transformer against a model whe...

  53. [53]

    Signal Metrics for All Ablations In this section, we report the signal metrics for all the abla- tions in the main paper. Most signal-level metrics follow similar trends as the semantic metrics, except for VisAH-FM trained with text- Adapter Modality IB Score↑KLD↓LDif↓Mag↓Env↓Was↓ ✓V+A29.12 9.70 7.778.28 2.74 0.63 ✓V + T 29.08 9.76 7.85 8.34 2.74 0.63 ✓V+...

  54. [54]

    However, when its contribution becomes too dominant, the trajectories exhibit non-linear behavior again

    Linearity of Trajectories in Semantic Space The analysis in Section 3.2.2 showed that the rollout loss helps linearize the inference trajectory. However, when its contribution becomes too dominant, the trajectories exhibit non-linear behavior again. This analysis was conducted in the time–frequency domain. Here we analyze linearity of trajectories in a se...

  55. [55]

    The rollout loss allows more consistent predictions across steps, resulting in more highlighted sources

    In-depth Qualitative Analysis Figure 7 shows further qualitative samples that highlight the difference of behavior between the flow matching mod- els trained with and without loss. The rollout loss allows more consistent predictions across steps, resulting in more highlighted sources. Figure 8 shows qualitative compari- son between VisAH and our model Vis...

  56. [56]

    Future work should evaluate the model on real-world data once such datasets become available

    Limitations and Future Work VisAH-FM delivers meaningful performance gains com- pared to the discriminative VisAH, but it is computationally more demanding and inherits limitations from the under- lying CLIP/CLAP representations, which may lead to fail- ures when audio–visual cues are weak. Future work should evaluate the model on real-world data once suc...