Pith. sign in

REVIEW 2 major objections 6 minor 69 references

Learning from Silence and Noise for Visual Sound Source Localization

T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By pairing every training image with silence and Gaussian noise and penalizing any localization response to them, the paper's fully self-supervised model filters negative audio completely while improving accuracy on real sounds.

desk verdict A plausible training recipe for silencing false positives in VSSL, but the baseline comparison is not fully controlled; the IS3+ benchmark and separability metric are worth having. read the letter →

arxiv 2508.21761 v1 pith:XVQ4CON7 submitted 2025-08-29 cs.CV cs.MM

Xavier Juanola , Giovana Morais , Magdalena Fuentes , Gloria Haro This is my paper
classification cs.CVcs.MM
keywords visualsoundsourcelocalizationself-supervisedlearningnegativeaudiosilenceandnoisecontrastivecross-modalretrievalaudio-visualsimilaritymapssyntheticbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual sound source localization systems learn to point at the object making a sound, but most are trained and tested only on 'positive' cases where the audio is real, visible, and matches the image. The paper argues this hides a failure mode: when the audio is silence, pure noise, or a sound coming from offscreen, these models hallucinate activations on whatever object dominates the scene. Its central proposal is to make negative audio part of training — for every image in a batch, the model also sees a silent clip and a Gaussian-noise clip, and two new loss terms force the resulting similarity maps to zero. The resulting self-supervised model, SSL-SaN, reports better localization on positive sounds than previous self-supervised models while driving activation on silence and noise to effectively zero across four test sets. The paper also contributes a curated benchmark, IS3+, that fixes wrong audio-image pairs in the existing IS3 synthetic set, and a 'separability' metric that measures how cleanly a model's similarity scores separate positive from negative audio.

What carries the argument

The load-bearing mechanism is a pair of negative-audio training pairs and two matching loss terms. For every image j in the batch, the model computes an audio-visual similarity map S(a, vj) by cosine similarity. The silence loss LS = ||S(aS, vj)||²₂ and the noise loss LN = ||S(aNj, vj)||²₂ — where aS is the embedding of a silent clip and aNj a fresh Gaussian-noise clip — push those maps toward zero. The claim is that this penalty is what makes negative-audio similarity scores fall below the model's universal threshold at test time, and that the forced separation of positive and negative score distributions is what lifts performance on positive sounds as well. A secondary piece of machinery i

What would settle it

Train SSL-TIE from scratch for 120 epochs with the same data, augmentations, batch size, and schedule used for SSL-SaN, omitting only the silence/noise pairs and their loss terms. If that same-epochs control matches SSL-SaN's numbers, the central attribution to silence/noise training collapses; if it stays near the published baseline values, the new losses are doing the work. A confirmatory observation already sits in the paper's ablation: dropping LS and LN while keeping the negative samples raises silence activation from 0.05 to 1.75 percent of image area on AVS-Bench S4.

Watch

Extended reading notes

Core claim

The paper's central claim is that failure on silence, noise, and offscreen audio is a training-signal problem, not an architecture problem. Standard contrastive training only mismatches audio and images across videos, so a model never sees a clip that contains no sound and has no incentive to produce an empty similarity map. The paper adds that incentive: each training image is paired with a silent clip and a Gaussian-noise clip, and two loss terms, LS and LN, penalize the squared L2 norm of the similarity maps those pairs generate. The resulting model, SSL-SaN, is the SSL-TIE architecture plus these losses; it reports gains over prior self-supervised models on the universal-threshold cIoU,

Load-bearing premise

The attribution of the gains to silence/noise training assumes all other differences from the baseline are irrelevant, but SSL-SaN trained for 120 epochs versus the baseline's 100, and the paper's own control without the new losses does not reproduce the published baseline numbers, so re-training details could explain part of the improvement.

Editorial extensions

If this is right

  • A fully self-supervised model can drive activation on silence and noise to effectively zero (pIA down to 0.00) while matching or beating prior self-supervised models on positive sounds, with no architectural change.
  • The training strategy is not tied to one architecture: it applies to any contrastively trained localization model, since it only adds negative audio pairs and their loss terms.
  • Localization performance should be judged on positive and negative audio together; with a universal threshold and FLOC/FAUC, silence, noise, and offscreen behavior become part of the standard evaluation rather than an afterthought.
  • The separability metric gives a single number that tracks both sound localization and cross-modal retrieval quality, so it can replace separate diagnostics for the positive-versus-negative distinction.
  • The corrected IS3+ dataset shows that audio-image pair quality changes measured model performance, making dataset curation itself part of fair benchmarking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'explicit nothing' trick — supplying inputs that should produce empty outputs and penalizing any activation — likely transfers to other audio-visual tasks such as source separation or audio-visual segmentation, where hallucinating on silent or offscreen input is a known failure mode.
  • Training uses one Gaussian noise distribution (zero mean, sigma = 1), so a natural extension is testing whether the suppression generalizes to natural background sounds, music, or differently shaped noise spectra.
  • Part of the reported gain may come from the longer training run (120 epochs versus the baseline's 100) or other retraining details rather than the new losses alone; a same-epochs control run would settle the attribution.
  • The separability metric could serve as a cheap, label-free early-stopping or model-selection signal during training, since it appears to track downstream localization and retrieval without needing annotations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes SSL-SaN, a fully self-supervised visual sound source localization model trained with additional silence and Gaussian-noise negative audio-visual pairs. Two new loss terms, LS and LN (Section 3.1), penalize nonzero similarity maps for these negative inputs, with a weighting parameter lambda_SN. The authors claim this strategy improves positive-case localization and robustness to negative audio, and they report state-of-the-art results among self-supervised models on VGG-SS, IS3, IS3+, and AVS-Bench S4 across localization, cross-modal retrieval, and negative-audio metrics. The paper also introduces IS3+, a curated version of the IS3 synthetic benchmark with corrected audio pairs, and a separability metric (Sep) that measures the gap between positive and negative maximum-similarity distributions.

Significance. If the attribution to the proposed losses is properly controlled, the contribution is practically valuable: a simple, architecture-agnostic training strategy that can be added to any contrastive VSSL method, with code and data released. IS3+ and the separability metric are useful community assets. However, the main positive-case claim is currently not established by a controlled comparison, because the baseline differs in training epochs and possibly other unspecified details. The non-circular part of the negative-case robustness claim (offscreen sounds) is promising and deserves a careful evaluation.

major comments (2)
  1. [Supplementary Table A.1; Tables A.7 and 2] The comparison to the SSL-TIE baseline is not controlled. Table A.1 states that SSL-TIE is trained for 100 epochs while SSL-SaN is trained for 120. The no-silence/noise ablation row in Table A.7 reports S4 cIoU 31.57 and VGG-SS 28.38, whereas the published SSL-TIE values in Table 2 are S4 28.40 and VGG-SS 27.78. If the ablation row is a 120-epoch retrained baseline, the improvement attributable to LS/LN is only about 1.2 cIoU on S4 (32.76 vs 31.57), not the roughly 4.4 implied by the Table 2 comparison. The authors must report the epoch count and all hyperparameters for every ablation row, and rerun the baseline under exactly the same training schedule as SSL-SaN. Without this, the central claim that silence/noise losses improve positive-case performance is not supported.
  2. [Section 3.1; Tables 2, 4, A.7] The near-zero pIA values for silence and noise are largely by construction. LS and LN directly penalize nonzero similarity maps for exactly the same silence and Gaussian-noise input distributions that are later evaluated, so reporting pIAS=0.01 and pIAN=0.00 as a robustness result is circular. The paper should explicitly distinguish this optimized behavior from generalization to unseen negatives. The non-circular negative-case evidence is the offscreen pIAO, which is not trained on, and the cross-modal retrieval results. The statement in Section 4.2 that the model 'completely filters out silence and noise' should be reframed as a consequence of the training objective, not an emergent property.
minor comments (6)
  1. [Section 4.2] The text contains a typo: 's Thanks to the addition of loss terms...' should be 'Thanks...'.
  2. [Table 2 caption] The caption says 'VGG-SS, IS3+ and S4 extended test sets', but the table includes columns for IS3 as well. Please update the caption to include IS3.
  3. [Section 3.1] Please define 'empty audio' concretely (e.g., an all-zero log-mel spectrogram) and specify whether the Gaussian noise is a new random sample per batch element and how the standard deviation sigma=1 is applied after normalization.
  4. [Section 4.1] The definition of AUCN is terse. Clarify the integration interval and the relationship to pIA thresholds; a formula would be helpful.
  5. [Supplementary Table A.2.1] The header 'Total (Epoch)' mixes a duration and a count; consider separating total wall-clock time and number of epochs for clarity.
  6. [General] The empirical claims are based on single runs without variance or multiple seeds. Reporting mean and standard deviation over a few seeds would strengthen the robustness of the comparisons.

Circularity Check

1 steps flagged · score 4.0 of 10

Silence/noise filtering is the training objective itself, so those negative-case results are expected; positive-case and retrieval claims retain independent content, though the baseline comparison is not fully controlled.

  1. fitted input called prediction [Sec. 3.1 (definitions of LS/LN) and Sec. 4.2 / Table 2 (silence/noise pIA results)]
    "First, we pair each image in the batch with these two types of negative audio samples... We define silence as an empty audio and noise as an audio with random values following a Gaussian distribution with zero mean and standard deviation σ=1. Second, we add two new loss terms forcing an empty similarity map for these two negative audio-visual pairs... LS = ||S(aS,v_j)||_2^2 ... LN = ||S(aN_j,v_j)||_2^2. Thanks to the addition of loss terms specifically addressing silence and noise during training, our model completely filters out silence and noise."

    The test-time silence and noise inputs are the same distributions used to define the training losses: silence is an empty audio and noise is Gaussian with zero mean and σ=1, exactly as in LS and LN. Minimizing LS/LN drives S(aS,v_j) and S(aN_j,v_j) toward zero for precisely these inputs, so the reported pIAS≈0 and pIAN≈0 (or AUCN=100) are the training objective itself, not an independent prediction. Any model that fully optimizes these loss terms would exhibit the same near-perfect filtering on silence and noise. The statement 'completely filters out silence and noise' therefore reduces, for those two negative categories, to a check that optimization succeeded. The positive-case cIoU and cross-modal retrieval results are not specified by LS/LN and retain independent content.

full rationale

The only circular reduction I can exhibit is the silence/noise portion of the evaluation: the model is trained to output an empty similarity map on exactly the silence and Gaussian-noise distributions used at test time, so near-zero pIA on those inputs is by construction rather than an empirical discovery. This affects the silence/noise columns and the FLOC/FAUC aggregates that include them, but not the positive-case localization or cross-modal retrieval comparisons, which are not determined by LS/LN. I checked the self-citations to the authors' prior work [27] for the Universal threshold and negative metrics: those are not load-bearing in a circular way because the threshold is computed per model from that model's own negative-case similarity distributions, and the metrics are external evaluation tools, not fitted to make SSL-SaN win. I also note a genuine experimental-control concern that is not circularity: Table A.1 shows SSL-SaN trained for 120 epochs vs 100 for SSL-TIE, and the no-silence/noise ablation row in Table A.7 (e.g., S4 cIoU-Uth 31.57) does not match the published SSL-TIE value in Table 2 (S4 28.40), so part of the reported positive-case gain may be attributable to training length or other unspecified differences. That is a correctness/attribution issue, not a self-referential derivation. Overall, the central positive-case claim has independent content, so the circularity is partial and localized rather than wholesale.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or structural entities. IS3+ is a curated dataset, the separability metric is a descriptive statistic, and the silence/noise losses are objective terms, none of which require independent physical evidence outside the paper.

free parameters (3)
  • lambda_SN loss weight for (LS + LN) = 1
    Selected by ablation in Table A.6, apparently on the S4 test set; no held-out validation split is described, so this risks overfitting the loss weight to the benchmark.
  • Training epochs for SSL-SaN = 120
    SSL-SaN is trained for 120 epochs while the SSL-TIE baseline is reported at 100 epochs in Table A.1; this may confound the effect of the proposed losses with longer training.
  • Noise standard deviation sigma = 1
    Gaussian noise negative samples are fixed at sigma=1 with no ablation of noise type or noise level; the reported robustness to 'noise' is specific to this choice.
assumptions (4)
  • domain assumption VGGSound-144K audio-visual pairs are semantically aligned enough for contrastive learning.
    Training relies on i=j pairs carrying the visible source's sound; stated in Section 4.1.
  • domain assumption Silence (empty audio) and Gaussian noise (sigma=1) adequately represent the class of negative audio.
    Defined in Section 3.1 and used for both training and evaluation, so silence and noise robustness scores partly inherit the training distribution.
  • domain assumption Manually curated IS3+ audio-image pairings are correct, giving a reliable benchmark.
    IS3+ curation described in Section 3.2 relies on manual review and class simplification; errors here would bias all models evaluated on IS3+.
  • domain assumption The Universal threshold from [27] transfers across models and datasets.
    Metrics cIoU-Uth, pIA, and FLOC use the quartile-based threshold of the authors' prior work; described in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Silence and Noise for Visual Sound Source Localization." pith.science (2026). https://pith.science/paper/XVQ4CON7

@misc{pith2026250821761,
  author       = {Pith},
  title        = {Pith review of: Learning from Silence and Noise for Visual Sound Source Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVQ4CON7}},
  note         = {Machine review of arXiv:2508.21761}
}
read the original abstract

Visual sound source localization is a fundamental perception task that aims to detect the location of sounding sources in a video given its audio. Despite recent progress, we identify two shortcomings in current methods: 1) most approaches perform poorly in cases with low audio-visual semantic correspondence such as silence, noise, and offscreen sounds, i.e. in the presence of negative audio; and 2) most prior evaluations are limited to positive cases, where both datasets and metrics convey scenarios with a single visible sound source in the scene. To address this, we introduce three key contributions. First, we propose a new training strategy that incorporates silence and noise, which improves performance in positive cases, while being more robust against negative sounds. Our resulting self-supervised model, SSL-SaN, achieves state-of-the-art performance compared to other self-supervised models, both in sound localization and cross-modal retrieval. Second, we propose a new metric that quantifies the trade-off between alignment and separability of auditory and visual features across positive and negative audio-visual pairs. Third, we present IS3+, an extended and improved version of the IS3 synthetic dataset with negative audio. Our data, metrics and code are available on the https://xavijuanola.github.io/SSL-SaN/.

Figures

Figures reproduced from arXiv: 2508.21761 by the authors.

Figure 1
Figure 1. Distribution of the maximum values of the audio-visual similarity maps across [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Localization results of different models in both [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 63 canonical work pages

  1. [1]

    Adobe audition sound effects, 2023

    Adobe. Adobe audition sound effects, 2023. URL https://www.adobe.com/ products/audition/offers/adobeauditiondlcsfx.html. Accessed: [25-Jan-2025]

    work page 2023

  2. [2]

    Self- supervised learning of audio-visual objects from video

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In European Conference on Computer Vision, pages 208–224. Springer, 2020

    work page 2020

  3. [3]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InIEEE International Conference on Computer Vision, pages 609–617, 2017

    work page 2017

  4. [4]

    Objects that sound

    Relja Arandjelovic and Andrew Zisserman. Objects that sound. In European Conference on Computer Vision, pages 435–451, 2018

    work page 2018

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 721–725, 2020

    work page 2020

  6. [6]

    Localizing visual sounds the hard way

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In IEEE/CVF conference on computer vision and pattern recognition, pages 16867–16876, 2021

    work page 2021

  7. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational Conference on Machine Learning, pages 1597–1607. PMLR, 2020

    2020

  8. [8]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF conference on computer vision and pattern recognition, pages 15750–15758, 2021

    work page 2021

Show all 69 references
  1. [9]

    Integrating audio, visual, and semantic information for enhanced multimodal speaker diarization

    Luyao Cheng, Hui Wang, Siqi Zheng, Yafeng Chen, Rongjie Huang, Qinglin Zhang, Qian Chen, and Xihao Li. Integrating audio, visual, and semantic information for enhanced multimodal speaker diarization. arXiv preprint arXiv:2408.12102, 2024

    2024 arXiv

  2. [10]

    Learning a similarity metric discrimi- natively, with application to face verification

    Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discrimi- natively, with application to face verification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, volume 1, pages 539–546, 2005. 12 JUANOLA ET AL.: SSL-SAN: LEARNING FROM SILEN...

    2005

  3. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  4. [12]

    Condi- tional generation of audio from video via foley analogies

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Condi- tional generation of audio from video via foley analogies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2436, 2023

    2023

  5. [13]

    Audio-visual approach for multimodal concurrent speaker detection

    Amit Eliav and Sharon Gannot. Audio-visual approach for multimodal concurrent speaker detection. arXiv preprint arXiv:2407.01774, 2024

    2024 arXiv

  6. [14]

    Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

    Stefan Fichna, Thomas Biberger, Bernhard U Seeber, and Stephan D Ewert. Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments. In 2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA), page...

    2021

  7. [15]

    Learning joint sta- tistical models for audio-visual fusion and segregation

    John W Fisher III, Trevor Darrell, William Freeman, and Paul Viola. Learning joint sta- tistical models for audio-visual fusion and segregation. Advances in neural information processing systems, 13, 2000

    2000

  8. [16]

    Visualvoice: Audio-visual speech separation with cross-modal consistency

    Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15490–15500. IEEE, 2021

    2021

  9. [17]

    Cyclip: Cyclic contrastive language-image pretraining

    Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704–6719, 2022

    2022

  10. [18]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural i...

    2020

  11. [19]

    chirp" from the

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Sep- arating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13117–13127, 2024

    2024

  12. [20]

    Canonical correlation analysis: An overview with application to learning methods

    David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16 (12):2639–2664, 2004

    2004

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  14. [22]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

    2020

  15. [23]

    Audio vision: Using audio-visual synchrony to locate sounds

    John Hershey and Javier Movellan. Audio vision: Using audio-visual synchrony to locate sounds. Advances in neural information processing systems, 12, 1999. JUANOLA ET AL.: SSL-SAN: LEARNING FROM SILENCE AND NOISE FOR VSSL 13

    1999

  16. [24]

    Discriminative sounding objects localization via self-supervised audiovisual matching

    Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33:10077–10087, 2020

    2020

  17. [25]

    Mix and localize: Localizing sound sources in mixtures

    Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022

    2022

  18. [26]

    You said that?: Synthesising talking faces from audio

    Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. International Journal of Computer Vision, 127:1767–1779, 2019

    2019

  19. [27]

    A critical assessment of visual sound source localization models including negative audio

    Xavier Juanola, Gloria Haro, and Magdalena Fuentes. A critical assessment of visual sound source localization models including negative audio. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1–5, 2025

    2025

  20. [28]

    Pixels that sound

    Einat Kidron, Yoav Y Schechner, and Michael Elad. Pixels that sound. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, volume 1, pages 88–95, 2005

    2005

  21. [29]

    Learning to visually localize sound sources from mixtures without prior source knowledge

    Dongjin Kim, Sung Jin Um, Sangmin Lee, and Jung Uk Kim. Learning to visually localize sound sources from mixtures without prior source knowledge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26467–26476, 2024

    2024

  22. [30]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  23. [31]

    Cooperative learning of audio and video models from self-supervised synchronization

    Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018

    2018

  24. [32]

    Recent advances in multi-modal 3d scene understanding: A comprehensive survey and evaluation

    Yinjie Lei, Zixuan Wang, Feng Chen, Guoqing Wang, Peng Wang, and Yang Yang. Recent advances in multi-modal 3d scene understanding: A comprehensive survey and evaluation. arXiv preprint arXiv:2310.15676, 2023

    2023 arXiv

  25. [33]

    Do audio-visual segmentation models truly segment sounding objects? arXiv preprint arXiv:2502.00358, 2025

    Jia Li, Wenjie Zhao, Ziru Huang, Yunhui Guo, and Yapeng Tian. Do audio-visual segmentation models truly segment sounding objects? arXiv preprint arXiv:2502.00358, 2025

    2025 arXiv

  26. [34]

    Av-nerf: Learning neural fields for real-world audio-visual scene synthesis

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems, 36:37472–37490, 2023

    2023

  27. [35]

    Exploiting transformation invariance and equivariance for self-supervised sound localisation

    Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3742–3753, 2022

    2022

  28. [36]

    Visual sound localization in the wild by cross-modal interference erasing

    Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou. Visual sound localization in the wild by cross-modal interference erasing. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 1801–1809, 2022. 14 JUANOLA...

    2022

  29. [37]

    Image segmentation using text and image prompts

    Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022

    2022

  30. [38]

    T-vsl: Text-guided visual sound source localization in mixtures

    Tanvir Mahmud, Yapeng Tian, and Diana Marculescu. T-vsl: Text-guided visual sound source localization in mixtures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26742–26751, 2024

    2024

  31. [39]

    Localizing visual sounds the easy way

    Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In European Conference on Computer Vision, pages 218–234. Springer, 2022

    2022

  32. [40]

    A closer look at weakly-supervised audio-visual source localization

    Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Neural Information Processing Systems, 35:37524– 37536, 2022

    2022

  33. [41]

    Audio-visual grouping network for sound localization from mixtures

    Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10565–10574, 2023

    2023

  34. [42]

    V ovit: Low latency graph-based audio-visual voice separation transformer

    Juan F Montesinos, Venkatesh S Kadandale, and Gloria Haro. V ovit: Low latency graph-based audio-visual voice separation transformer. In European Conference on Computer Vision, pages 310–326. Springer, 2022

    2022

  35. [43]

    Speech inpainting: Context-based speech synthesis guided by video

    Juan F Montesinos, Daniel Michelsanti, Gloria Haro, Zheng-Hua Tan, and Jesper Jensen. Speech inpainting: Context-based speech synthesis guided by video. In Interspeech, pages 4459–4463, 2023

    2023

  36. [44]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  37. [45]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In European Conference on Computer Vision, pages 631–648, 2018

    2018

  38. [46]

    Do we need sound for sound source localization? In Asian Conference on Computer Vision, 2020

    Takashi Oya, Shohei Iwase, Ryota Natsume, Takahiro Itazuri, Shugo Yamaguchi, and Shigeo Morishima. Do we need sound for sound source localization? In Asian Conference on Computer Vision, 2020

    2020

  39. [47]

    Marginnce: Robust sound localization with a negative margin

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Marginnce: Robust sound localization with a negative margin. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1–5, 2023

    2023

  40. [48]

    Can clip help sound source localization? In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5711–5720, 2024

    Sooyoung Park, Arda Senocak, and Joon Son Chung. Can clip help sound source localization? In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5711–5720, 2024

    2024

  41. [49]

    Multiple sound sources localization from coarse to fine

    Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In European Conference on Computer Vision, pages 292–308. Springer, 2020

    2020

  42. [50]

    See the sound, hear the pixels

    Janani Ramaswamy and Sukhendu Das. See the sound, hear the pixels. In IEEE/CVF winter conference on applications of computer vision, pages 2970–2979, 2020. JUANOLA ET AL.: SSL-SAN: LEARNING FROM SILENCE AND NOISE FOR VSSL 15

    2020

  43. [51]

    Sound source localization

    Michael Risoud, J-N Hanson, Fanny Gauvrit, Christian Renard, P-E Lemesre, N-X Bonne, and Christophe Vincent. Sound source localization. European annals of otorhinolaryngology, head and neck diseases, 135(4):259–264, 2018

    2018

  44. [52]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

    2022

  45. [53]

    Multimodal emotion recognition based on a fusion of audiovi- sual information with temporal dynamics

    José Salas-Cáceres, Javier Lorenzo-Navarro, David Freire-Obregón, and Modesto Castrillón-Santana. Multimodal emotion recognition based on a fusion of audiovi- sual information with temporal dynamics. Multimedia Tools and Applications, pages 1–17, 2024

    2024

  46. [54]

    Learn- ing to localize sound source in visual scenes

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learn- ing to localize sound source in visual scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018

    2018

  47. [55]

    Learn- ing to localize sound sources in visual scenes: Analysis and applications

    Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learn- ing to localize sound sources in visual scenes: Analysis and applications. IEEE transac- tions on pattern analysis and machine intelligence, 43(5):1605–1619, 2019

    2019

  48. [56]

    Learning sound localization better from semantically similar samples

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4863–4867, 2022

    2022

  49. [57]

    Less can be more: Sound source localization with a classification model

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Less can be more: Sound source localization with a classification model. In IEEE/CVF conference on computer vision and pattern recognition, pages 3308–3317, 2022

    2022

  50. [58]

    Aligning sight and sound: Advanced sound source localization through audio-visual alignment

    Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, and Joon Son Chung. Aligning sight and sound: Advanced sound source localization through audio-visual alignment. arXiv preprint arXiv:2407.13676, 2024

    2024 arXiv

  51. [59]

    A survey on audio synthesis and audio-visual multimodal processing

    Zhaofeng Shi. A survey on audio synthesis and audio-visual multimodal processing. arXiv preprint arXiv:2108.00443, 2021

    2021 arXiv

  52. [60]

    En- hancing sound source localization via false negative elimination

    Zengjie Song, Jiangshe Zhang, Yuxi Wang, Junsong Fan, and Zhaoxiang Zhang. En- hancing sound source localization via false negative elimination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  53. [61]

    Learning audio-visual source localization via false negative aware contrastive learning

    Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. Learning audio-visual source localization via false negative aware contrastive learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  54. [62]

    Sound to visual scene generation by audio-to-visual latent alignment

    Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. Sound to visual scene generation by audio-to-visual latent alignment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440, 2023

    2023

  55. [63]

    Sound2vision: Gener- ating diverse visuals from audio through cross-modal latent alignment

    Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, and Tae-Hyun Oh. Sound2vision: Gener- ating diverse visuals from audio through cross-modal latent alignment. arXiv preprint arXiv:2412.06209, 2024. 16 JUANOLA ET AL.: SSL-SAN: LEARNING FROM SILENCE AND NOISE FOR VSSL

    2024 arXiv

  56. [64]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In European Conference on Computer Vision, pages 247–263, 2018

    2018

  57. [65]

    Phrasecut: Language-based image segmentation in the wild

    Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language-based image segmentation in the wild. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020

    2020

  58. [66]

    How to listen? rethinking visual sound localization

    Ho-Hsiang Wu, Magdalena Fuentes, Prem Seetharaman, and Juan Pablo Bello. How to listen? rethinking visual sound localization. Interspeech, 2022

    2022

  59. [67]

    Acoustic and visual knowledge distillation for contrastive audio-visual localization

    Ehsan Yaghoubi, Andre Peter Kelm, Timo Gerkmann, and Simone Frintrop. Acoustic and visual knowledge distillation for contrastive audio-visual localization. In Proceed- ings of the 25th International Conference on Multimodal Interaction , pages 15–23, 2023

    2023

  60. [68]

    Diagnosing and rectifying vision models using language

    Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. arXiv preprint arXiv:2302.04269, 2023

    2023 arXiv

  61. [69]

    chicken clucking

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation with semantics. International Journal of Computer Vision, pages 1–21, 2024. JUANOLA ET AL.: SSL-SAN: L...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.