Pith. sign in

REVIEW 3 major objections 5 minor 69 references

ViSAGe: Video-to-Spatial Audio Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ViSAGe generates four-channel first-order ambisonics directly from silent video frames and a camera-direction input, and reports that this end-to-end route beats two-stage pipelines that first generate mono audio and then spatialize it.

desk verdict The paper's central spatial-outperformance claim rests on a mathematically invalid energy map; the new task and dataset are real, but the evaluation needs a fundamental rework. read the letter →

arxiv 2506.12199 v1 pith:EJ7SFRV7 submitted 2025-06-13 cs.SD cs.AIeess.AS

Jaeyeon Kim , Heeseung Yun , Gunhee Kim This is my paper
classification cs.SDcs.AIeess.AS
keywords first-orderambisonicsvideo-to-audiogenerationspatialaudioneuralcodecautoregressiveenergymapCLIPvisualfeaturesdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that spatial audio can be generated end-to-end from silent, ordinary field-of-view video: specifically the four-channel first-order ambisonics format (W, X, Y, Z) used by YouTube and VR platforms. Its claim is that modeling all four channels jointly, as discrete audio codes, outperforms the natural two-stage alternative of first generating mono audio from video and then spatializing that mono into ambisonics using the visual scene. To make this trainable and measurable, the paper contributes YT-Ambigen, a dataset of 102K five-second YouTube clips paired with ground-truth ambisonics, and a pair of spatial metrics (correlation coefficient and area-under-curve over spherical audio energy maps) adapted from visual saliency evaluation. If the claim is right, immersive soundtrack production for film, virtual reality, and AI-generated video could bypass expensive microphone arrays and multi-stage production pipelines entirely.

What carries the argument

The load-bearing machinery has four pieces. First, first-order ambisonics represent a sound field by four spherical-harmonic channels, and the audio energy map $$G(\phi,\$\theta$)=\frac{1}{L}\sum_{t=1}^{L}\left[$Y^{0}$_0 W(t)+$Y^{{-1}}$_1 Y(t)+$Y^{0}$_1 Z(t)+$Y^{1}$_1 X(t)\right]$$ maps the channels' energy onto the sphere, serving as training signal, evaluation instrument, and dataset-curation tool at once. Second, the Descript Audio Codec (DAC), a neural audio codec that compresses each waveform channel into discrete residual codes with nine codebooks per channel, supplies the token vocabulary. Third, the interleaved code-generation pattern reorganizes the four channels' code matrices into $2L_c+1$ sequential steps so that a single autoregressive transformer models both the residual dependency and the spatial dependency without quadrupling the sequence length. Fourth, conditioning on CLIP features, a patchwise energy map computed from spatial and temporal embedding distances, and an MLP-projected camera-direction unit vector, trained with azimuth rotation augmentation and joint classifier-free guidance on both visual and directional conditions, steers the decoder. The energy map appears on both sides of the argument: it tells the model where sound should come from, and it is the ruler used to judge whether the sound is placed correctly.

What would settle it

Two concrete checks would settle the spatial claim. First, recompute the CC/AUC metrics on ground-truth and generated FOA after zeroing the directional channels (setting X=Y=Z=0, keeping only W): if the resulting scores approach the full-channel values, the metric is dominated by the omnidirectional content and is not measuring spatial placement. Second, run a listening experiment with enough participants (well beyond 12) that judges sound-source localization or spatial preference, and test whether clips with higher CC/AUC are actually preferred or more accurately localized; if human spatial judgments do not track the metric, the claimed improvement over two-stage baselines lacks perceptual grounding.

Watch

Extended reading notes

Core claim

The central discovery is that all four first-order ambisonics channels can be generated jointly and directly from video, without any reference mono audio. ViSAGe encodes each channel with the Descript Audio Codec into residual-vector-quantized token sequences, then uses an autoregressive transformer to predict an interleaved schedule of omnidirectional and spatial codebooks ($W_p \to (W_r, S_p) \to (W_p, S_r) \to \dots$) that captures both residual dependencies and cross-channel spatial dependencies in $2L_c+1$ steps instead of $4L_c$. Conditioned on CLIP frame embeddings, a patchwise energy map that highlights spatially and temporally distinct regions, and a camera-direction embedding, ViSAGe finetuned on YT-Ambigen reaches a decoded Fr\'echet audio distance of 3.86 (against 7.24 for the best two-stage baseline) and full-clip spatial agreement of CC 0.635 / AUC 0.846 (against 0.577 / 0.826). With directional guidance alone it beats every two-stage baseline on both semantic and spatial metrics (CC 0.721, AUC 0.890); adding visual guidance improves semantic quality further at a modest spatial cost, matching the stronger baseline spatially while clearly outperforming it semantically.

Load-bearing premise

The reported spatial gains depend on the assumption that CC and AUC agreement between generated and ground-truth audio energy maps genuinely measures spatial audio quality — an assumption currently backed only by a 12-participant listening test with no significance values — so if those scores do not track human spatial perception, the spatial-accuracy comparison is not evidence of better spatial audio.

Editorial extensions

If this is right

  • Two-stage pipelines — video-to-mono generation followed by audio spatialization — accumulate errors; the paper's numbers indicate that generating all four ambisonics channels jointly avoids the training-inference gap that appears when spatializers are fed generated rather than real mono audio.
  • The interleaved codebook schedule is a general recipe for multi-channel neural-codec generation: four channels cost about as many autoregressive steps as two mono sequences, so the same pattern should extend to binaural, 5.1, or higher-order ambisonics without redesign.
  • Camera direction becomes a user-facing control: because the model is conditioned on an explicit direction vector and trained with rotation augmentation, changing the input viewpoint should rotate the perceived sound field, giving creators directable spatial audio for interactive media.
  • YT-Ambigen and the energy-map CC/AUC metrics give the field a shared benchmark for video-to-ambisonics, so future methods can be compared on spatial coherence rather than only on mono audio quality.
  • Joint classifier-free guidance on visual and directional conditions outperforms guiding either condition separately, evidence that the two conditions are interdependent and should be modeled as one coupled signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the spatial metrics compare whole energy maps that include the W (omnidirectional) channel, a spurious driver of agreement is conceivable: if W dominates the map energy, two audios could score high CC/AUC even when their X/Y/Z placements differ. A cheap diagnostic is recomputing the metrics on a W-only rendering (X=Y=Z=0); if the gap over two-stage baselines shrinks sharply, the spatial cl
  • The patchwise energy map encodes a testable hypothesis: patch-level spatial and temporal distinctness in CLIP embeddings predicts the location of sounding objects. Comparing energy-map peaks against explicit tracking or sound-source localization on YT-Ambigen would test this directly, and the same map could serve as a weakly supervised localizer.
  • The interleaved W/S, primary/residual schedule should transfer to other multichannel codec tasks (stereo music, binaural, higher-order ambisonics); the cheapest test is applying it to the stereo-music setting and comparing against the sequential-delay pattern the paper ablates.
  • The small in-person listening test (12 participants, 30 videos, no significance values) is the only perceptual evidence; a larger preregistered localization study correlating human judgments with CC/AUC would determine whether the metric gains are perceptually audible or mostly a proxy artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a new task of generating first-order ambisonics (FOA) directly from silent videos, along with a large-scale dataset YT-Ambigen (102K five-second clips of field-of-view videos paired with FOA recordings), a set of proposed spatial evaluation metrics based on audio energy maps and saliency (CC and AUC), and an end-to-end model called ViSAGe that uses CLIP visual features, patchwise energy maps, camera-direction embeddings, and autoregressive generation of Descript Audio Codec codes for the four ambisonic channels. The experiments compare ViSAGe against two-stage baselines (video-to-audio generation followed by audio spatialization) and include ablations on model components, code-generation patterns, and classifier-free guidance, together with a small in-person listening test.

Significance. If the spatial evaluation were valid, this would be a significant contribution: ViSAGe is apparently the first end-to-end approach to synthesize first-order ambisonics directly from silent videos, and the introduced dataset and code-generation pattern (interleaving omnidirectional and spatial codebooks while modeling both residual and spatial dependencies) are potentially useful for the community. The model design is reasonable and the paper is clearly written. However, the central quantitative claims about spatial quality rest on an evaluation metric whose mathematical definition does not measure energy, and the reported spatial improvements are therefore not currently supported by the evidence; the dataset curation also relies on the same flawed quantity.

major comments (3)
  1. [Section 3.3, Eq. (1)] Equation (1) defines the audio energy map G(phi, theta) as a temporal average of a linear combination of the waveform channels W, X, Y, and Z. This is the DC component of a virtual cardioid microphone signal, not an energy or power quantity. Natural audio waveforms are zero-mean and AC-coupled, so G is near zero in all directions and is dominated by any residual DC offset; moreover, G can be negative, which makes the application of CC and AUC, saliency metrics designed for nonnegative maps, mathematically uncalibrated. Because all spatial results in Tables 3 and 4 (the CC and AUC columns) are computed from these maps, the reported spatial superiority of ViSAGe over the two-stage baselines is not supported by the quantitative evidence as written. The metric should be redefined using a time-averaged energy or power quantity, for example the mean of s(phi, theta)^2 with s as defined in Eq. (2), and all spatial experiments must be rerun. Note that Section 4 also uses the argmax of this map for FoV cropping during dataset curation, so the flawed definition propagates into the dataset construction as well.
  2. [Section 6.2, Table 3 (code-generation-pattern ablation)] In the code-generation-pattern ablation, the row labeled 'Ours' reports FAD_dec = 4.44, KLD_dec = 1.89, FAD_avg = 4.49, CC = 0.544, and AUC = 0.794, which coincides exactly with an intermediate configuration in the model-components ablation (the row with FAD_dec = 4.44, KLD_dec = 1.89, CC = 0.544) rather than with the full ViSAGe model reported in the same table (FAD_dec = 3.86, KLD_dec = 1.71, FAD_avg = 4.20, CC = 0.635, AUC = 0.846). The manuscript does not state that the pattern ablation was run with a different configuration than the final model, so the reader cannot verify that the proposed pattern is compared with the alternatives at the same setting, and the label 'Ours' is misleading. Please disclose the exact configuration used for this ablation and, if it was a reduced configuration, repeat the comparison at the full-model setting.
  3. [Section 6.2 and Appendix E] The in-person listening test is based on only 12 participants, with no significance tests, confidence intervals, or per-participant variability reported. The preference margins are modest, for example for Spatiality against SpecVQGAN the result is 52.22% vs. 31.11% with 16.67% ties, and against Diff-Foley it is 42.22% vs. 30.00% with 27.78% ties. Given that the automatic spatial metrics are invalid as currently defined, this test is too weak to independently corroborate the spatial coherence claims. The authors should report statistical significance, confidence intervals, and ideally a larger participant pool, or at least per-participant agreement measures.
minor comments (5)
  1. [Section 3.3, Eq. (1)] The notation reuses the symbol Y for both the ambisonic channel Y(t) and the spherical harmonic Y_l^m, which may confuse readers; consider using a different symbol for the spherical harmonic (e.g., S_l^m or Y_l^m with a different font).
  2. [Section 4] The dataset curation description says 'We calculate the audio energy map for each clip to identify the argmax coordinate'; if the energy map is redefined as suggested in the major comments, this sentence needs to be checked and the cropping procedure may need to be reconsidered.
  3. [Abstract and Section 1] There are a few typographical issues: 'the interaction betweeen objects' in Section 1 should read 'between', and 'confuses' in the paper's abstract should be 'comprises' in Section 4 ('YT-Ambigen dataset comprises').
  4. [Appendix A] In Appendix A, 'For computing the compute energy map E' should be 'For computing the energy map E'.
  5. [Tables 3 and 4] No error bars, confidence intervals, or statistical significance are reported for any of the quantitative metrics; even after the energy-map definition is corrected, the comparisons should be accompanied by uncertainty estimates.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; the central training objective is independent of the proposed spatial metrics, and baselines are external. The only self-citation is a non-load-bearing methodological pointer.

full rationale

ViSAGe's training objective (Eq. 8) is cross-entropy on DAC code predictions for the four FOA channels, conditioned on video and camera direction. The proposed spatial metrics (Section 3.3) compare CC/AUC between ground-truth and generated audio energy maps defined by Eq. 1; these metrics do not appear in the loss, so the model is not trained to optimize them. The comparison baselines (SpecVQGAN and Diff-Foley) are external pretrained models, and the two-stage spatialization baselines are constructed with standard encoders or trained U-Nets; no fitted parameter is renamed as a prediction. The dataset curation uses the audio energy map to crop FoV clips and the evaluation uses the same energy map; this is a task-design choice that may bias the benchmark, but it does not make the reported result equal to the input by construction, since both ViSAGe and the baselines are evaluated under the same protocol. The only self-citation is PAVER (Yun et al., 2022) for the patchwise saliency score in Eq. 3; it is a methodological pointer, not a load-bearing uniqueness or correctness argument. The skeptic's concern that Eq. 1 computes a signed time-average of FOA channels rather than an energy quantity is a validity and calibration issue for the proposed metric, not a circularity of the derivation; likewise, the 12-participant listening test without significance values is an evidentiary weakness. Under the circularity rubric, no step reduces by construction to its inputs, so the appropriate finding is no significant circularity, with a score of 2 only for the minor non-load-bearing self-citation of PAVER.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its free parameters are hyperparameters chosen by hand or sweep. The key load-bearing axioms are the validity of the energy-map-based spatial metrics and the sufficiency of visual features plus camera direction to determine the sound field.

free parameters (7)
  • classifier-free guidance scale omega = 2.5
    Selected by hyperparameter sweep on validation; affects the balance between semantic and spatial quality (Section 5.2, Table 4).
  • softmax temperature for patchwise energy map = 0.1
    Chosen by hand for converting patch scores to probabilities (Appendix A).
  • top-p threshold for energy map filtering = 0.7
    Chosen by hand after averaging probabilities (Appendix A).
  • rotation augmentation probability = 0.5
    Chosen by hand for azimuth rotation during training (Section 5.2).
  • null embedding dropout probability = 0.1
    Used for classifier-free guidance training (Section 5.2).
  • spatial and temporal window sizes (N, T) for patchwise energy map = 1, 1
    Set to 1 for computing patch scores (Appendix A).
  • CLIP frame rate = 4 FPS
    Chosen by hand for visual feature extraction (Appendix A).
assumptions (5)
  • standard math Spherical harmonics decomposition of first-order ambisonics (Eq. 1) is a valid representation of spatial audio energy.
    Used to define audio energy maps and the proposed spatial metrics (Sections 3.1, 3.3).
  • domain assumption CLIP visual features and patchwise energy maps contain sufficient information to infer the spatial audio field of the scene.
    Core conditioning of ViSAGe; if false, the model cannot generate meaningful spatial audio (Section 5.1).
  • domain assumption Camera direction D=(phi,theta) is available at inference and uniquely specifies where the FoV is placed within the full 3D sound field.
    Task definition includes D as input (Section 3.2); without it, FoV-based ambisonics is ambiguous.
  • domain assumption The audio energy map ground truth used for evaluation (computed from FOA via Eq. 1) is a valid proxy for spatial auditory perception.
    Proposed spatial metrics rely on this (Section 3.3); only a small subjective test supports it.
  • ad hoc to paper The YT-Ambigen curation pipeline (amplitude filtering, AudioSet selection, FoV cropping around energy argmax, audio-visual relevance filtering) yields videos whose sound sources are visible and semantically meaningful.
    Dataset construction choices are specific to this paper and shape the task (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViSAGe: Video-to-Spatial Audio Generation." pith.science (2026). https://pith.science/paper/EJ7SFRV7

@misc{pith2026250612199,
  author       = {Pith},
  title        = {Pith review of: ViSAGe: Video-to-Spatial Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJ7SFRV7}},
  note         = {Machine review of arXiv:2506.12199}
}
read the original abstract

Spatial audio is essential for enhancing the immersiveness of audio-visual experiences, yet its production typically demands complex recording systems and specialized expertise. In this work, we address a novel problem of generating first-order ambisonics, a widely used spatial audio format, directly from silent videos. To support this task, we introduce YT-Ambigen, a dataset comprising 102K 5-second YouTube video clips paired with corresponding first-order ambisonics. We also propose new evaluation metrics to assess the spatial aspect of generated audio based on audio energy maps and saliency metrics. Furthermore, we present Video-to-Spatial Audio Generation (ViSAGe), an end-to-end framework that generates first-order ambisonics from silent video frames by leveraging CLIP visual features, autoregressive neural audio codec modeling with both directional and visual guidance. Experimental results demonstrate that ViSAGe produces plausible and coherent first-order ambisonics, outperforming two-stage approaches consisting of video-to-audio generation and audio spatialization. Qualitative examples further illustrate that ViSAGe generates temporally aligned high-quality spatial audio that adapts to viewpoint changes.

Figures

Figures reproduced from arXiv: 2506.12199 by the authors.

Figure 1
Figure 1. Video-to-Spatial Audio Generation. Given a silent video and the camera direction, the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Overall architecture of ViSAGe and (b) its ambisonics generation with DAC codes. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. (a) Qualitative examples of generated audios and (b) audio energy visualization. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of different code generation patterns from Section 6.2. For (a), (c), and (d), [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Qualitative examples of generated audios from ViSAGe and two-stage approaches. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Qualitative example of patchwise energy map and generated audio in rapidly changing [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Qualitative example of the camera direction parameter. (a) Camera direction is set to front: [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Distribution statistics of YT-Ambigen. (a) The top-50 AudioSet labels distribution predicted [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 55 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Musiclm: Generating music from text

    Andrea Agostinelli, Timo I Denk, Zal \'a n Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv:2301.11325, 2023

    arXiv 2023

  3. [3]

    The Foley grail: The art of performing sound for film, games, and animation

    Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation. Routledge, 2014

    work page 2014

  4. [4]

    Audiolm: A language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. IEEE/ACM TASLP, 2023

    work page 2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/video-generation-models-as-world-simulators

    work page 2024

  6. [6]

    What do different evaluation metrics tell us about saliency models? IEEE TPAMI, 2018

    Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Fr \'e do Durand. What do different evaluation metrics tell us about saliency models? IEEE TPAMI, 2018

    work page 2018

  7. [7]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022

    2022

  8. [8]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020 a

    work page 2020

Show all 69 references
  1. [9]

    Generating visually aligned sound from videos

    Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE TIP, 2020 b

    2020

  2. [10]

    Cube padding for weakly-supervised saliency prediction in 360 videos

    Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. Cube padding for weakly-supervised saliency prediction in 360 videos. In CVPR, 2018

    2018

  3. [11]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D \'e fossez. Simple and controllable music generation. In NeurIPS, 2023

    2023

  4. [12]

    Proc \'e d \'e s et syst \`e mes d'enregistrement et de reproduction sonores en trois dimensions

    Daniel Courville and Ambisonic Studio. Proc \'e d \'e s et syst \`e mes d'enregistrement et de reproduction sonores en trois dimensions . Universit \'e du Qu \'e bec \`a Montr \'e al, 1994

    1994

  5. [13]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024

    2024

  6. [14]

    High fidelity neural audio compression

    Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. TMLR, 2023

    2023

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021

    2021

  9. [17]

    2.5d visual sound

    Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019

    2019

  10. [18]

    Visually-guided audio spatialization in video with geometry-aware multi-task learning

    Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually-guided audio spatialization in video with geometry-aware multi-task learning. IJCV, 2023

    2023

  11. [19]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017

    2017

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  13. [21]

    Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson. Cnn architectures for large-scale audio classification. In ICASSP, 2017

    2017

  14. [22]

    Immersediffusion: A generative spatial audio latent diffusion model

    Mojtaba Heydari, Mehrez Souden, Bruno Conejo, and Joshua Atkins. Immersediffusion: A generative spatial audio latent diffusion model. In ICASSP, 2025

    2025

  15. [23]

    Spatial audio in 360° videos: does it influence visual attention? In ACM MMSys, 2022

    Amit Hirway, Yuansong Qiao, and Niall Murray. Spatial audio in 360° videos: does it influence visual attention? In ACM MMSys, 2022

    2022

  16. [24]

    Evaluating visual attention and qoe for 360° videos with non-spatial and spatial audio

    Amit Hirway, Yuansong Qiao, and Niall Murray. Evaluating visual attention and qoe for 360° videos with non-spatial and spatial audio. In MMSys, 2024

    2024

  17. [25]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022

    2022 arXiv

  18. [26]

    Imagen video: High definition video generation with diffusion models

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02303, 2022 a

    arXiv

  19. [27]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NeurIPS, 2022 b

    2022

  20. [28]

    a \" a n\

    Jukka Holm, Kaisa V\" a \" a n\" a nen, and Anas Battah. User experience of stereo and spatial audio in 360° live music videos. In AcademicMindtrek, 2020

    2020

  21. [29]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. In BMVC, 2021

    2021

  22. [30]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2025

    2025

  23. [31]

    Efficient training of audio transformers with patchout

    Khaled Koutini, Jan Schl \" u ter, Hamid Eghbal - zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Interspeech, 2022

    2022

  24. [32]

    Audiogen: Textually guided audio generation

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D \'e fossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In ICLR, 2023

    2023

  25. [33]

    High-fidelity audio compression with improved rvqgan

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. In NeurIPS, 2024

    2024

  26. [34]

    Diff-sage: End-to-end spatial audio generation using diffusion models

    Saksham Singh Kushwaha, Jianbo Ma, Mark RP Thomas, Yapeng Tian, and Avery Bruni. Diff-sage: End-to-end spatial audio generation using diffusion models. In ICASSP, 2025

    2025

  27. [35]

    Voiceldm: Text-to-speech with environmental context

    Yeonghyeon Lee, Inmo Yeon, Juhan Nam, and Joon Son Chung. Voiceldm: Text-to-speech with environmental context. In ICASSP, 2024

    2024

  28. [36]

    Generating stereophonic music with single-stage language models

    Xingda Li, Fan Zhuo, Dan Luo, Jun Chen, Shiyin Kang, Zhiyong Wu, Tao Jiang, Yang Li, Han Fang, and Yahui Zhou. Generating stereophonic music with single-stage language models. In ICASSP, 2024 a

    2024

  29. [37]

    Cyclic learning for binaural audio generation and localization

    Zhaojian Li, Bin Zhao, and Yuan Yuan. Cyclic learning for binaural audio generation and localization. In CVPR, 2024 b

    2024

  30. [38]

    Enhancing spatial audio generation with source separation and channel panning loss

    Wootaek Lim and Juhan Nam. Enhancing spatial audio generation with source separation and channel panning loss. In ICASSP, 2024

    2024

  31. [39]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Doll \'a r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017

    2017

  32. [40]

    A udio LDM : Text-to-audio generation with latent diffusion models

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. A udio LDM : Text-to-audio generation with latent diffusion models. In ICML, 2023

    2023

  33. [41]

    Visually guided binaural audio generation with cross-modal consistency

    Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. Visually guided binaural audio generation with cross-modal consistency. In ICASSP, 2024

    2024

  34. [42]

    Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In NeurIPS, 2023

    2023

  35. [43]

    Foleygen: Visually-guided audio generation

    Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. arXiv:2309.10537, 2023

    2023 arXiv

  36. [44]

    Self-supervised generation of spatial audio for 360 video

    Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), NeurIPS, 2018

    2018

  37. [45]

    Learning representations from audio-visual spatial alignment

    Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurIPS, 2020

    2020

  38. [46]

    Spatial audio in youtube vr videos and its impacts on audience engagement

    Huyen Nguyen and Madeline Willson. Spatial audio in youtube vr videos and its impacts on audience engagement. In I3DA, 2023

    2023

  39. [47]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In CVPR, 2016

    2016

  40. [48]

    Masked generative video-to-audio transformers with enhanced synchronicity

    Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, and Joan Serr \`a . Masked generative video-to-audio transformers with enhanced synchronicity. In ECCV, 2024

    2024

  41. [49]

    Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence

    Sandra Poeschl, Konstantin Wall, and Nicola Doering. Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence. In IEEE VR, 2013

    2013

  42. [50]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  43. [51]

    Towards generating ambisonics using audio-visual cue for virtual reality

    Aakanksha Rana, Cagri Ozcinar, and Aljosa Smolic. Towards generating ambisonics using audio-visual cue for virtual reality. In ICASSP, 2019

    2019

  44. [52]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurIPS, 2019

    2019

  45. [53]

    Fr 'echet audio distance: A reference-free metric for evaluating music enhancement algorithms

    Dominik Roblek, Kevin Kilgour, Matt Sharifi, and Mauricio Zuluaga. Fr 'echet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Interspeech, 2019

    2019

  46. [54]

    I hear your true colors: Image guided audio generation

    Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP, 2023

    2023

  47. [55]

    Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events

    Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal an...

    2024

  48. [56]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023

    2023

  49. [57]

    Semantic object prediction and spatial sound super-resolution with binaural sounds

    Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Semantic object prediction and spatial sound super-resolution with binaural sounds. In ECCV, 2020

    2020

  50. [58]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv:2301.02111, 2023

    2023 arXiv

  51. [59]

    V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models

    Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In AAAI, 2024

    2024

  52. [60]

    Codec-superb: An in-depth analysis of sound codec models

    Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander H Liu, and Hung-yi Lee. Codec-superb: An in-depth analysis of sound codec models. arXiv:2402.13071, 2024

    2024 arXiv

  53. [61]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023

    2023

  54. [62]

    Visually informed binaural audio generation without binaural audios

    Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021

    2021

  55. [63]

    Dualspeech: Enhancing speaker-fidelity and text-intelligibility through dual classifier-free guidance

    Jinhyeok Yang, Junhyeok Lee, Hyeong-Seok Choi, Seunghoon Ji, Hyeongju Kim, and Juheon Lee. Dualspeech: Enhancing speaker-fidelity and text-intelligibility through dual classifier-free guidance. In Interspeech, 2024

    2024

  56. [64]

    Panoramic vision transformer for saliency detection in 360 videos

    Heeseung Yun, Sehun Lee, and Gunhee Kim. Panoramic vision transformer for saliency detection in 360 videos. In ECCV, 2022

    2022

  57. [65]

    Soundstream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE TASLP, 2021

    2021

  58. [66]

    Sep-stereo: Visually guided stereophonic audio generation by associating source separation

    Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020

    2020

  59. [67]

    Visual to sound: Generating natural sound for videos in the wild

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In CVPR, 2018

    2018

  60. [68]

    Masked audio generation using a single non-autoregressive transformer

    Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Jade Copet, Alexandre D \'e fossez, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. In ICLR, 2024

    2024

  61. [69]

    Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality

    Franz Zotter and Matthias Frank. Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.