Pith. sign in

REVIEW 3 major objections 5 minor 80 references

Learning to Highlight Audio by Watching Movies

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Movies teach an AI to rebalance audio using the video as a guide.

desk verdict New task and dataset are real contributions; the visual-guidance claim is thinner than the paper suggests, because the synthetic test can be largely passed by audio-only restoration. read the letter →

arxiv 2505.12154 v1 pith:BQTQRQJS submitted 2025-05-17 cs.CV cs.SDeess.AS

Chao Huang , Ruohan Gao , J. M. F. Tsang , Jan Kurcius , Cagdas Bilen , Chenliang Xu , Anurag Kumar , Sanjeel Parekh This is my paper · ORCID
classification cs.CVcs.SDeess.AS
keywords visually-guidedacoustichighlightingaudioremixingaudio-visuallearningmoviesupervisionpseudo-datagenerationtransformersourceseparationvideo-to-audiorefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces visually-guided acoustic highlighting: given a video whose audio track is unbalanced (dialogue drowned by waves, music swelling over speech), the goal is to automatically rebalance the audio so that what the viewer sees emphasized is also what is heard clearly. The central claim is that this can be learned by watching movies, because professionally mixed film audio already encodes the intended highlighting, and the authors make this learning signal usable by deliberately ruining good mixes. They separate movie audio into speech, music, and sound-effect stems, apply random boosts and cuts of 6–12 dB, and remix to create 'poorly mixed' inputs paired with the original as ground truth. They then train a transformer-based model, VisAH, which uses the video (or its text captions) as context to undo the damage, and report that it outperforms audio-only remixing baselines on waveform, semantic, and time-alignment metrics and in a human ranking test.

What carries the argument

The load-bearing mechanism is the separate–adjust–remix pseudo-data generation process. A cinematic source-separation model splits each movie clip's ground-truth audio into speech, music, and sound-effects stems (plus a residual so the stems sum exactly to the original); the loudest stem is suppressed and the others boosted by randomly chosen amounts among $\{-12,-9,-6\}$ dB and $\{+6,+9,+12\}$ dB, and the stems are added back to form the poorly mixed input. The model that must undo this is VisAH: a dual U-Net whose magnitude-spectrogram and waveform encoders produce a shared latent representation, a latent highlighting transformer whose decoder cross-attends over temporal context (per-frame vision features or embedded frame captions, each passed through its own transformer encoder), and a zero-initialized $1\times 1$ convolution so the network's output starts as no change and is added as a residual offset to the latent audio features. Training uses a multiscale STFT $\ell^1$ loss between predicted and ground-truth audio.

What would settle it

Take a collection of genuinely raw recordings known to be poorly mixed (amateur vlogs, phone footage with wind or crowd noise), have audio engineers produce a reference rebalanced mix for each, and compare VisAH's output against both the input and an audio-only remixer on waveform distance and audio-visual alignment. If VisAH fails to beat the input or matches the audio-only baseline on these real recordings, the central claim that video guidance learned from movies transfers to real-world footage is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a new audio-visual task and a working solution: learn a mapping $(a, v) \mapsto s$ from poorly mixed audio and its accompanying video to a highlighted audio signal that preserves the content of $a$ while rebalancing speech, music, and sound effects in line with what the video emphasizes. The enabling discovery is that movie audio, fetched from a large condensed-movie corpus, provides free ground-truth highlighting, and that a synthetic pipeline—imperfect separation into three stems, loudness adjustment with amounts from $\{ \pm 6, \pm 9, \pm 12 \}$ dB, then linear remixing—produces input/output pairs that let a model learn the transformation. The paper's VisAH model combines a dual U-Net audio backbone (spectrogram and waveform branches) with a latent transformer that attends over temporal context, either per-frame vision features or text-caption embeddings, and predicts an offset added back to the original features so content is preserved. Evaluations show consistent improvement over three baselines across five metrics, with temporal context mattering more than single-frame context and text captions giving the best guidance; a small human study ranks VisAH first or second in 77% of cases, and in about a third of cases above the original movie audio.

Load-bearing premise

The weakest load-bearing premise is that the synthetic ruin-and-restore procedure (separate, adjust, remix) produces inputs that represent how real videos are actually poorly mixed, so that a model trained on those pairs transfers to genuine recordings; the paper validates real-world behavior only qualitatively.

Editorial extensions

If this is right

  • If the central claim holds, any amateur video with lopsided audio can be automatically rebalanced by reading the video's visual emphasis, restoring dialogue or ambient sound without manual stem editing.
  • The same model can serve as a post-processing step for video-to-audio generation systems, measurably improving the audio-visual alignment of generated soundtracks.
  • Both video frames and text captions carry usable highlighting guidance, and temporal context across frames or captions matters more than a single frame, so future systems should condition on full scenes rather than stills.
  • The pseudo-data generation recipe (imperfect separation, loudness adjustment, linear remixing) yields a large training set of 15,078 movie clips, and the paper's difficulty analysis suggests performance scales with how far the input is from the target mix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The movie-trained model likely encodes Hollywood-style mixing aesthetics; presenting it with video from different cultures or genres, or with shuffled video–audio pairs, would reveal how much of the highlighting is driven by genuine visual semantics rather than a learned loudness prior.
  • Because the guidance channel is flexible, the same architecture could be steered by an explicit text prompt (such as 'focus on the music' or 'make the crowd recede'), turning highlighting into a controllable semantic editing tool that goes beyond video-only conditioning.
  • The separate–adjust–remix pipeline could be applied recursively or with multiple separators, as the paper itself notes as future work, letting the approach scale to any professionally mixed audio-visual content without new manual annotation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a new task, visually-guided acoustic highlighting (L2H), in which poorly mixed audio is transformed into a well-balanced mix using the accompanying video as guidance. The authors propose VisAH, a transformer-based dual U-Net architecture that takes both spectrogram and waveform inputs, encodes contextual information (video frames or text captions) with transformer encoders, and uses a transformer decoder with cross-attention to output highlighted audio. A new dataset, the Muddy Mix Dataset, is built from the Condensed Movie Dataset by separating movie audio into speech, music, and sound effects, applying random loudness gains/suppressions, and remixing. The method is evaluated on a held-out test set against several baselines using five metrics (MAG, ENV, KLD, delta-IB, Wasserstein distance), with ablations on context type, number of transformer encoder layers, and dataset difficulty, plus a small subjective ranking test. The central claim is that VisAH consistently outperforms baselines in both quantitative and subjective evaluation.

Significance. If the central claim holds, the paper proposes a genuinely new audio-visual task and a practical data-driven approach to audio rebalancing that extends prior music-only remixing work to general movie and video content. The Muddy Mix dataset and the separation-adjustment-remix pseudo-data pipeline are useful contributions, and the paper's systematic ablations (context types, encoder depth, difficulty levels) are a strength. The authors also provide a project page with qualitative examples and a downstream application to video-to-audio generation refinement. However, the evidence that the model actually learns visually guided highlighting is currently not as strong as the abstract claims: the quantitative evaluation is performed on a synthetic test set generated by the same pipeline used for training, and the no-context ablation is close to the full model. The small subjective test (9 participants, 10 videos) further limits the strength of the human-evaluation claim.

major comments (3)
  1. [Section 4 and Table 1] The main quantitative evaluation is performed on a test set constructed by the same three-step pipeline (separation, adjustment, remixing) that generates the training data, with the original movie audio as ground truth and known per-stem loudness gains. Because the degradation is an invertible, stem-level gain change and the source separator is the same one used in training-data creation, an audio-only model can learn to approximately undo the augmentation without any visual understanding. Table 2 indeed shows that the no-context VisAH model achieves MAG 10.35 vs. 10.08 for temporal-text guidance and KLD 11.95 vs. 11.01, a modest gap with no error bars. The paper therefore demonstrates effective audio restoration on a synthetic augmentation, but not yet that VisAH learns visually guided highlighting that transfers to non-synthetic poor mixes. I request either an evaluation on real-world poorly mixed audio with a reference or human study, or a test design in which the target highlighting is not recoverable from the input audio alone.
  2. [Tables 1-4 and Section 5.3] All quantitative results are reported as single point estimates without error bars, confidence intervals, or significance tests. This is load-bearing because several key comparisons have small margins: in Table 2, temporal text guidance improves over no-context by only 0.27 MAG, 0.94 KLD, and 0.19 delta-IB, and in Table 3, vision context with 6 layers is worse than with 3 layers. Without variance estimates, the claims that contextual guidance matters and that VisAH significantly outperforms baselines are not statistically supported. Please report results over multiple training seeds or bootstrap confidence intervals, and use a paired significance test for the subjective ranking data.
  3. [Section 8, Section 15, and project page] The real-world applicability claims rest on qualitative demonstrations. Section 8 shows a real waterfall-dominated clip where VisAH fails to rebalance speech against the waterfall, and Section 15 explicitly lists the separation-adjustment-remix strategy as a limitation. The application to web videos and to MovieGen-generated audio is only shown qualitatively on the project page. Given that the quantitative benchmark is synthetic, the claim in the abstract and Section 6 that the method "outperforms several baselines in both quantitative and subjective evaluation" and generalizes to real content needs either a real-world benchmark with quantitative or human evaluation, or a clearly scoped statement that real-world transfer is a preliminary qualitative finding.
minor comments (5)
  1. [Throughout] There are several typos, including "mutimodal" in the contributions list and "generes" in Section 4; please proofread the manuscript.
  2. [Section 3.2.3, Eq. (5)] The MR-STFT loss is used in the main text but defined only in the supplementary material; please define it briefly in the main text or move the definition there.
  3. [Section 4, reference [8]] The text cites the "Cinematic Sound Demixing Challenge" but reference [8] is titled "The Sound Demixing Challenge 2023 - Music Demixing Track"; please verify the citation or add the correct reference for the cinematic demixing challenge.
  4. [Table 4 and Section 5.3.2] The statement that metrics show continuous improvement as difficulty decreases is not supported by Table 4: Low-P has worse MAG (9.55) than Moderate-P (8.73) and worse W-dis (0.80) than Moderate-P (0.65), so the trend is not monotonic across all metrics.
  5. [Section 8, Fig. 8] The breathing example described as a failure actually demonstrates behavior aligned with the visual context (close-up of a breathing face) while diverging from the movie ground truth; the caption should clarify that this is a divergence from the particular movie mix, not a failure to follow visual guidance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the target is an external ground-truth movie mix, and the evaluation metrics are independent of the training loss.

full rationale

The claimed derivation is a supervised audio-to-audio translation model whose target is the original movie mix s, an external ground truth. The synthetic 'poorly mixed' input a is produced by a three-step separation/adjustment/remixing of s (Sec. 4), and VisAH is trained with MR-STFT loss against s (Eq. 5). This is a standard paired-data restoration setup; the target is not a fitted parameter, and the evaluation metrics (MAG, ENV, KLD via PaSST, ΔIB via ImageBind, and Wasserstein distance) are computed on held-out clips and do not appear in the training loss. The fact that the test set shares the same synthetic augmentation pipeline as training is a distributional/generalization limitation, acknowledged in Sec. 15, not circularity: predicting s from a is not equivalent by construction to any input of the model, since a is a deliberately degraded version and the network must learn the mapping. The paper's self-citations are only in related-work context and do not carry the load-bearing argument; no uniqueness theorem or ansatz is imported from the authors' prior work. Therefore no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the premise that movies provide valid ground-truth highlighting effects and that the synthetic degradation pipeline produces realistic poorly-mixed audio. These are domain assumptions rather than proven statements. The method also relies on standard pretrained models and deep learning components. No new physical entities are introduced.

free parameters (3)
  • Loudness adjustment levels = +12/+9/+6 dB and -12/-9/-6 dB
    Hand-chosen gains for dataset generation (Section 4, 'Adjustment'); the ablation in Table 4 shows model performance varies with the level, so this choice directly shapes the task difficulty.
  • Context transformer encoder layers = 6 for text, 3 for vision
    Selected via ablation in Table 3; the final model uses these depths, and performance peaks at these values.
  • Audio latent dimension = 768
    Set in Section 5.1 as Ca=768; a capacity choice that affects the transformer module and was not swept.
assumptions (4)
  • domain assumption Movie audio is a reliable example of well-highlighted audio, so that original movie mixes can serve as ground truth for highlighting.
    The entire supervision scheme (Section 4) relies on the premise that professional movie mixes embody the intended highlighting effects.
  • ad hoc to paper Poorly mixed audio can be simulated by separating movie audio into speech, music, and sound effects, adjusting their loudness, and remixing.
    The pseudo-data generation process (Section 4) assumes this three-stem model and discrete gain adjustments capture the essence of real-world poor mixing. The paper itself notes limitations in Section 15.
  • domain assumption The visual stream and its captions provide sufficient information to infer which sources should be highlighted.
    The task formulation (Section 3.1) assumes the video conveys the intent for audio highlighting; failure cases (Fig. 8) show this is not always true.
  • standard math Standard deep learning background results (STFT, transformer attention, U-Net) hold as used.
    Architecture relies on established components without proof, which is standard practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Highlight Audio by Watching Movies." pith.science (2026). https://pith.science/paper/BQTQRQJS

@misc{pith2026250512154,
  author       = {Pith},
  title        = {Pith review of: Learning to Highlight Audio by Watching Movies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQTQRQJS}},
  note         = {Machine review of arXiv:2505.12154}
}
read the original abstract

Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.

Figures

Figures reproduced from arXiv: 2505.12154 by the authors.

Figure 1
Figure 1. We propose a new task that aims to transform poorly [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of VisAH: (a) Our model takes a poorly mixed audio waveform as input and produces the highlighted audio using a dual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We generate poorly mixed audio from the well-mixed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: We perform a qualitative comparison by visualizing the waveform and magnitude spectrograms of the highlighted audio results [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: We demonstrate that our VisAH method enhances the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Failure cases analysis: Our method highlights the breath [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 7
Figure 7. Figure 7: Failure case analysis: the sound effect (waterfall) over [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 10
Figure 10. Figure 10: Design of magnitude and waveform encoders. Each [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 9
Figure 9. Figure 9: Screenshot of subjective test interface. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 12
Figure 12. Figure 12: The improvement trend across the three difficulty levels [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 61 canonical work pages

  1. [1]

    Self-supervised learning of audio-visual objects from video.European Conference on Computer Vision (ECCV), 2020

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video.European Conference on Computer Vision (ECCV), 2020. 2

    work page 2020

  2. [2]

    Condensed movies: Story based retrieval with con- textual embeddings, 2020

    Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zis- serman. Condensed movies: Story based retrieval with con- textual embeddings, 2020. 2, 5

    work page 2020

  3. [3]

    Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos

    Changan Chen, Puyuan Peng, Ami Baid, Sherry Xue, Wei- Ning Hsu, David Harwath, and Kristen Grauman. Ac- tion2sound: Ambient-aware generation of action sounds from egocentric videos. InECCV, 2024. 2

    work page 2024

  4. [4]

    iquery: Instruments as queries for audio-visual sound separation

    Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang, Ziyao Zeng, and Jianbo Shi. iquery: Instruments as queries for audio-visual sound separation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14675–14686, 2023. 3

    work page 2023

  5. [5]

    Deep cross-modal audio-visual generation

    Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chen- liang Xu. Deep cross-modal audio-visual generation. In Proceedings of the on Thematic Workshops of ACM Multime- dia 2017, 2017. 2

    work page 2017

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 4

    arXiv 2024

  7. [7]

    Hybrid spectrogram and waveform source separation

    Alexandre Défossez. Hybrid spectrogram and waveform source separation. InProceedings of the ISMIR 2021 Work- shop on Music Source Separation, 2021. 2, 3

    work page 2021

  8. [8]

    Giorgio Fabbro, Stefan Uhlich, Chieh-Hsin Lai, Woosung Choi, Marco A. Mart’inez-Ram’irez, Wei-Hsiang Liao, Igor Gadelha, Geraldo Ramos, Eddie Hsu, Hugo Rodrigues, Fabian-Robert Stoter, Alexandre D’efossez, Yi Luo, Jian- wei Yu, Dipam Chakraborty, Sharada Prasanna Mohanty, Roman A. Solovyev, Alexander L. Stempkovskiy, Tatiana Habruseva, Nabarun Goswami, Ta...

    work page 2023

Show all 80 references
  1. [9]

    2.5 d visual sound

    Ruohan Gao and Kristen Grauman. 2.5 d visual sound. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324–333, 2019. 2

    2019

  2. [10]

    Visualvoice: Audio- visual speech separation with cross-modal consistency

    Ruohan Gao and Kristen Grauman. Visualvoice: Audio- visual speech separation with cross-modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15490–15500. IEEE, 2021. 2

    2021

  3. [11]

    Learning to separate object sounds by watching unlabeled video

    Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35–53, 2018. 2

    2018

  4. [12]

    Visualechoes: Spatial visual represen- tation learning through echolocation

    Ruohan Gao, Changan Chen, Ziad Al-Halah, Carl Schissler, and Kristen Grauman. Visualechoes: Spatial visual represen- tation learning through echolocation. InEuropean Conference on Computer Vision (ECCV), 2020. 2

    2020

  5. [13]

    Geometry- aware multi-task learning for binaural audio generation from video

    Rishabh Garg, Ruohan Gao, and Kristen Grauman. Geometry- aware multi-task learning for binaural audio generation from video. InBritish Machine Vision Conference (BMVC), 2021. 2

    2021

  6. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 6

    2023

  7. [15]

    Semiautomatic visual-attention modeling and its application to video compression

    Yury Gitman, Mikhail Erofeev, Dmitriy Vatolin, Bolshakov Andrey, and Fedorov Alexey. Semiautomatic visual-attention modeling and its application to video compression. In2014 IEEE international conference on image processing (ICIP), pages 1105–1109. IEEE, 2014. 2

    2014

  8. [16]

    Graph- based visual saliency.Advances in neural information pro- cessing systems, 19, 2006

    Jonathan Harel, Christof Koch, and Pietro Perona. Graph- based visual saliency.Advances in neural information pro- cessing systems, 19, 2006. 2

    2006

  9. [17]

    Discriminative sounding objects localization via self-supervised audiovisual matching

    Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33, 2020. 2

    2020

  10. [18]

    Davis: High-quality audio-visual sep- aration with generative diffusion models.arXiv preprint arXiv:2308.00122, 2023

    Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Davis: High-quality audio-visual sep- aration with generative diffusion models.arXiv preprint arXiv:2308.00122, 2023. 2

    2023 arXiv

  11. [19]

    Egocentric audio-visual object localization.arXiv preprint arXiv:2303.13471, 2023

    Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization.arXiv preprint arXiv:2303.13471, 2023. 2

    2023 arXiv

  12. [20]

    Scaling concept with text- guided diffusion models.arXiv preprint arXiv:2410.24151,

    Chao Huang, Susan Liang, Yunlong Tang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Scaling concept with text- guided diffusion models.arXiv preprint arXiv:2410.24151,

    arXiv

  13. [21]

    Modeling and driving human body soundfields through acoustic primitives.arXiv preprint arXiv:2407.13083,

    Chao Huang, Dejan Markovic, Chenliang Xu, and Alexander Richard. Modeling and driving human body soundfields through acoustic primitives.arXiv preprint arXiv:2407.13083,

    arXiv

  14. [22]

    Fresca: Unveiling the scaling space in diffusion models.arXiv preprint arXiv:2504.02154, 2025

    Chao Huang, Susan Liang, Yunlong Tang, Li Ma, Yapeng Tian, and Chenliang Xu. Fresca: Unveiling the scaling space in diffusion models.arXiv preprint arXiv:2504.02154, 2025. 2

    2025 arXiv

  15. [23]

    A model of saliency-based visual attention for rapid scene analysis.IEEE Transactions on pattern analysis and machine intelligence, 20(11):1254–1259, 1998

    Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis.IEEE Transactions on pattern analysis and machine intelligence, 20(11):1254–1259, 1998. 2

    1998

  16. [24]

    Vinet: Pushing the limits of visual modality for audio-visual saliency prediction

    Samyak Jain, Pradeep Yarlagadda, Shreyank Jyoti, Shyam- gopal Karthik, Ramanathan Subramanian, and Vineet Gandhi. Vinet: Pushing the limits of visual modality for audio-visual saliency prediction. In2021 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IR...

  17. [25]

    Deepvs: A deep learning based video saliency prediction approach

    Lai Jiang, Mai Xu, Tie Liu, Minglang Qiao, and Zulin Wang. Deepvs: A deep learning based video saliency prediction approach. InThe European Conference on Computer Vision (ECCV), 2018. 2

    2018

  18. [26]

    Listen, chat, and edit: Text-guided soundscape mod- ification for enhanced auditory experience.arXiv preprint arXiv:2402.03710, 2024

    Xilin Jiang, Cong Han, Yinghao Aaron Li, and Nima Mes- garani. Listen, chat, and edit: Text-guided soundscape mod- ification for enhanced auditory experience.arXiv preprint arXiv:2402.03710, 2024. 6, 1

    2024 arXiv

  19. [27]

    Mu- sic mixing style transfer: A contrastive learning approach to disentangle audio effects

    Junghyun Koo, Marco A Martínez-Ramírez, Wei-Hsiang Liao, Stefan Uhlich, Kyogu Lee, and Yuki Mitsufuji. Mu- sic mixing style transfer: A contrastive learning approach to disentangle audio effects. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal ...

    2023

  20. [28]

    Efficient training of audio transformers with patchout

    Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Ger- hard Widmer. Efficient training of audio transformers with patchout. InInterspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 2753–27...

    2022

  21. [29]

    Contextual encoder–decoder network for visual saliency prediction.Neural Networks, 129:261–270, 2020

    Alexander Kroner, Mario Senden, Kurt Driessens, and Rainer Goebel. Contextual encoder–decoder network for visual saliency prediction.Neural Networks, 129:261–270, 2020. 2

    2020

  22. [30]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846–11858, 2021. 2

    2021

  23. [31]

    Neural acoustic context field: Rendering realistic room impulse response with neural fields.arXiv preprint arXiv:2309.15977, 2023

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Neural acoustic context field: Rendering realistic room impulse response with neural fields.arXiv preprint arXiv:2309.15977, 2023. 2

    2023 arXiv

  24. [32]

    Av-nerf: Learning neural fields for real- world audio-visual scene synthesis

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real- world audio-visual scene synthesis. InConference on Neural Information Processing Systems (NeurIPS), 2023. 2, 6

    2023

  25. [33]

    Language-guided joint audio-visual editing via one-shot adaptation

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Language-guided joint audio-visual editing via one-shot adaptation. InProceedings of the Asian Conference on Computer Vision, pages 1011–1027, 2024. 2

    2024

  26. [34]

    Univtg: Towards unified video-language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 2794–2804,

  27. [35]

    Audi- oldm: Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audi- oldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023. 6

    2023 arXiv

  28. [36]

    Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection

    Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3042–3051, 2022. 2

    2022

  29. [37]

    Play as you like: Timbre-enhanced multi-modal music style transfer

    Chien-Yu Lu, Min-Xin Xue, Chia-Che Chang, Che-Rung Lee, and Li Su. Play as you like: Timbre-enhanced multi-modal music style transfer. InProceedings of the aaai conference on artificial intelligence, pages 1061–1068, 2019. 1

    2019

  30. [38]

    Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation

    Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language pro- cessing, 27(8):1256–1266, 2019. 3, 6, 7

    2019

  31. [39]

    Deep learning for black-box modeling of audio effects.Applied Sciences, 10(2):638, 2020

    Marco A Martínez Ramírez, Emmanouil Benetos, and Joshua D Reiss. Deep learning for black-box modeling of audio effects.Applied Sciences, 10(2):638, 2020. 2

    2020

  32. [40]

    Localizing visual sounds the easy way.arXiv preprint arXiv:2203.09324, 2022

    Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way.arXiv preprint arXiv:2203.09324, 2022. 2

    2022 arXiv

  33. [41]

    Query-dependent video representation for moment retrieval and highlight detection

    WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23023–23033, 2023. 2

    2023

  34. [42]

    Self-supervised generation of spatial audio for 360 video.Advances in neural information processing systems, 2018

    Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video.Advances in neural information processing systems, 2018. 2

    2018

  35. [43]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. InPro- ceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018. 2

    2018

  36. [44]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Tor- ralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. InProceedings of the IEEE conference on computer vision and pattern recognition, 2016. 2

    2016

  37. [45]

    Physical modeling using recurrent neural networks with fast convolutional layers.arXiv preprint arXiv:2204.10125, 2022

    Julian D Parker, Sebastian J Schlecht, Rudolf Rabenstein, and Maximilian Schäfer. Physical modeling using recurrent neural networks with fast convolutional layers.arXiv preprint arXiv:2204.10125, 2022. 2

    2022 arXiv

  38. [46]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720,

    arXiv

  39. [47]

    Multiple sound sources localization from coarse to fine

    Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. InEuropean Conference on Computer Vision, pages 292–308. Springer, 2020. 2

    2020

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  41. [49]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 4

    2020

  42. [50]

    Modeling nonlinear audio effects with end-to-end deep neural networks

    Marco A Martínez Ramírez and Joshua D Reiss. Modeling nonlinear audio effects with end-to-end deep neural networks. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 171–175. IEEE, 2019. 2

    2019

  43. [51]

    Dynamic storyboard generation in an engine-based virtual environment for video production

    Anyi Rao, Xuekun Jiang, Yuwei Guo, Linning Xu, Lei Yang, Libiao Jin, Dahua Lin, and Bo Dai. Dynamic storyboard generation in an engine-based virtual environment for video production. InACM SIGGRAPH 2023 Posters, pages 1–2

    2023

  44. [52]

    U- net: Convolutional networks for biomedical image segmenta- tion

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmenta- tion. InMedical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Par...

    2015

  45. [53]

    Separate and reconstruct: Asymmetric encoder- decoder for speech separation

    Ui-Hyeop Shin, Sangyoun Lee, Taehan Kim, and Hyung- Min Park. Separate and reconstruct: Asymmetric encoder- decoder for speech separation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 6, 7

    2024

  46. [54]

    Freeu: Free lunch in diffusion u-net

    Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. InCVPR, 2024. 2

    2024

  47. [55]

    Steinmetz and Joshua D

    Christian J. Steinmetz and Joshua D. Reiss. pyloudnorm: A simple yet flexible loudness meter in python. In150th AES Convention, 2021. 5

    2021

  48. [56]

    Style transfer of audio effects with differentiable signal pro- cessing.arXiv preprint arXiv:2207.08759, 2022

    Christian J Steinmetz, Nicholas J Bryan, and Joshua D Reiss. Style transfer of audio effects with differentiable signal pro- cessing.arXiv preprint arXiv:2207.08759, 2022. 2

    2022 arXiv

  49. [57]

    Attention is all you need in speech separation

    Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. InICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 21–25. IEEE, 2021. 3, 7

    2021

  50. [58]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. InProceedings of the European Conference on Com- puter Vision (ECCV), pages 247–263, 2018. 2

    2018

  51. [59]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. InEuropean Conference on Computer Vision, pages 436–454. Springer, 2020

    2020

  52. [60]

    Cyclic co-learning of sounding object visual grounding and sound separation

    Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2745–2754, 2021. 2, 3

    2021

  53. [61]

    Diff-mst: Differen- tiable mixing style transfer.arXiv preprint arXiv:2407.08889,

    Soumya Sai Vanka, Christian Steinmetz, Jean-Baptiste Rol- land, Joshua Reiss, and George Fazekas. Diff-mst: Differen- tiable mixing style transfer.arXiv preprint arXiv:2407.08889,

    arXiv

  54. [62]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 4

    2017

  55. [63]

    Audiobox: Unified au- dio generation with natural language prompts.arXiv preprint arXiv:2312.15821, 2023

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified au- dio generation with natural language prompts.arXiv preprint arXiv:2312.15821, 2023. 6

    2023 arXiv

  56. [64]

    Lave: Llm-powered agent assistance and lan- guage augmentation for video editing

    Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, and Raj Sodhi. Lave: Llm-powered agent assistance and lan- guage augmentation for video editing. InProceedings of the 29th International Conference on Intelligent User Interfaces, pages 699–714, 2024. 1

    2024

  57. [65]

    Re- mastering divide and remaster: A cinematic audio source separation dataset with multilingual support

    Karn N Warcharasupat, Chih-Wei Wu, and Iroro Orife. Re- mastering divide and remaster: A cinematic audio source separation dataset with multilingual support. In2024 IEEE 5th International Symposium on the Internet of Sounds (IS2), pages 1–10. IEEE, 2024. 5, 6

  58. [66]

    Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. InCVPR, 2024. 1

    2024

  59. [67]

    Visually informed binaural audio generation without binaural audios

    Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15485–15494, 2021. 6

    2021

  60. [68]

    Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discrimi- nators

    Ryuichi Yamamoto, Eunwoo Song, Min-Jae Hwang, and Jae- Min Kim. Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discrimi- nators. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2021

  61. [69]

    Don’t separate, learn to remix: End-to-end neural remixing with joint optimization

    Haici Yang, Shivani Firodiya, Nicholas J Bryan, and Minje Kim. Don’t separate, learn to remix: End-to-end neural remixing with joint optimization. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 116–120. IEEE, 2022. 1, 6

    2022

  62. [70]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 4

    2023

  63. [71]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. InProceedings of the European conference on computer vision (ECCV), pages 570–586, 2018. 2, 3 Learning to Highlight Audio by Watching Movies Supplementary Material

    2018

  64. [72]

    github.io/VisAH/) to illustrate our method and show- case our results.We strongly encourage readers to visit this webpage and use headphones

    Project Page We have created a project page ( https://wikichao. github.io/VisAH/) to illustrate our method and show- case our results.We strongly encourage readers to visit this webpage and use headphones. Please note that the webpage may not be fully compatible with the Safar...

  65. [73]

    Here, we provide case studies to illustrate the conditions under which such failures occur

    Failure Case Analysis While our VisAH model is effective at highlighting audio guided by video content, there are scenarios where it might fail. Here, we provide case studies to illustrate the conditions under which such failures occur. In Fig. 7, the video captures a natural ...

  66. [74]

    Subjective Test Design We illustrate the interface design of our subjective test in Fig. 9. The instructions emphasize that users should evaluate whether the speech, music, and sound effects in the videos are well-balanced and acoustically pleasing, and whether the audio align...

  67. [75]

    As illustrated in Fig

    Network Details We detail the design of the magnitude and waveform en- coders, along with their input and output dimensions. As illustrated in Fig. 10, each encoder consists of five layers, and the output shapes for both branches after the fifth layer are identical. At each la...

  68. [76]

    The MR-STFT loss is implemented by computing theℓ1 distance between the am- plitude spectrograms of the predicted signalˆsand the ground truth signals

    Loss Function Details Here, we give a more detailed illustration on the MR-STFT (Multi-Resolution Short-Time Fourier Transform) loss func- tion used for training the model. The MR-STFT loss is implemented by computing theℓ1 distance between the am- plitude spectrograms of the ...

  69. [77]

    a dark, elegant outfit

    Motivation for Text Condition. Textual captions supplement video frames by leveraging strong reasoning capabilities of MLLMs. In Fig. 11, the caption generated by InternVL2-8B captures not only visual content, such as the appearance of individuals and room dec- orations, but a...

  70. [78]

    While our method requires more time, it remains efficient for practical applications

    Inference Time Comparison The inference times for VisAH, LCE, and L2R audio back- bone are 0.028s, 0.017s, and 0.018s, respectively. While our method requires more time, it remains efficient for practical applications

  71. [79]

    12 across differ- ent levels of dataset difficulty, as discussed in Sec 5.3.2 and shown in Tab

    Analysis of Dataset Difficulty We visualize the improvement trends in Fig. 12 across differ- ent levels of dataset difficulty, as discussed in Sec 5.3.2 and shown in Tab. 4. The magnitude of improvement is similar for the high and moderate difficulty levels, demonstrating that...

  72. [80]

    Limitations and Future Works Our method leverages versatile temporal conditions as guid- ance for audio highlighting, outperforming baseline methods and demonstrating applicability to real-world scenarios, in- cluding transferring knowledge from movies to daily and generated v...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.