Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

FoleySpace: Vision-Aligned Binaural Spatial Audio Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A silent 2D video can drive binaural audio whose perceived source position moves with the visible object, via a trajectory-conditioned diffusion model.

desk verdict FoleySpace is a genuinely novel video-to-binaural system, but the missing trajectory ablation leaves its central spatial-guidance claim undemonstrated. read the letter →

arxiv 2508.12918 v2 pith:BCSIUNXU submitted 2025-08-18 cs.SD

Lei Zhao , Rujin Chen , Chi Zhang , Xiao-Lei Zhang , Xuelong Li This is my paper · ORCID
classification cs.SD
keywords video-to-audiogenerationbinauralaudiospatialdiffusionmodelsoundsourcelocalizationhead-relatedimpulseresponseaudio-visualconsistencyopen-vocabularyobjectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FoleySpace is a framework for turning silent, ordinary 2D video into binaural stereo audio in which the perceived direction of the sound follows the visible source. The paper argues that current video-to-audio methods mostly produce monaural audio, and that stereo methods either ignore visual spatial alignment or only handle static sources. To close that gap, FoleySpace estimates the sound source's position in each frame, converts those positions into a 3D trajectory, and feeds that trajectory together with monaural audio into a diffusion model that renders left- and right-ear signals. The paper reports that the generated audio carries interaural level and time cues matching the on-screen source, and that user studies rate its spatial perception and spatial alignment above existing spatial-audio baselines. If correct, the framework would let creators obtain spatially consistent sound for video without capture hardware or per-video engineering.

What carries the argument

The load-bearing mechanism is the mapping from the 2D visual plane to a listener-centered 3D sound field. With $\delta = 2 S_y / W$, the source's 3D coordinate at frame $k$ is $x_k = \delta \tilde d_k$, $y_k = \delta (w_k - W/2)$, $z_k = -\delta (h_k - H/2)$, where $\tilde d_k$ is the min-max normalized depth scaled by $\gamma = W/2$ and $S_y = 1.47$ m is the hand-set maximum lateral distance. This mapping turns a bounding-box center $(w_k, h_k)$ and a depth value $d_k$ into a trajectory $T = \{(x_k,y_k,z_k)\}_{k=1}^K$ that aligns the image center with the listener and simulates a reasonable listening distance. The trajectory is smoothed by removing frames whose motion exceeds the 95% threshold and linearly interpolating the gaps, then it is replicated along time, concatenated with the monaural audio, and passed through a condition block into the diffusion backbone. This mechanism is what is supposed to make the left- and right-ear cues match the visual position at each moment.

What would settle it

Remove the trajectory condition and keep only the monaural audio, then repeat the user study on the same videos; if spatial perception and spatial alignment scores do not drop, the visual trajectory is not carrying the claimed spatial consistency.

Watch

Extended reading notes

Core claim

The paper's central claim is that binaural spatial audio for dynamic scenes can be synthesized from a silent video by conditioning a waveform diffusion model on a trajectory derived from the video itself. The trajectory is obtained by locating the sound source with an open-vocabulary object detector, reading its depth from a monocular depth estimator, and mapping the normalized pixel coordinates and depth into a listener-centered 3D sound field through a linear camera-like transformation. That 3D trajectory, together with monaural audio from a pretrained video-to-audio model, is the conditioning signal for the diffusion model, which outputs left- and right-ear signals whose interaural level and time differences encode direction. Training data are synthesized by convolving monaural audio clips with recorded head-related impulse responses along randomly generated moving-source trajectories, including distance-varying time delays. Experiments compare FoleySpace with monaural, stereo-VAE, and RIR-based baselines and report higher user-study scores for spatial perception and spatial alignment while keeping semantic, temporal, and audio-quality metrics close to monaural generation.

Load-bearing premise

The entire spatial alignment depends on the estimated 3D trajectory—the bounding-box center plus min-max normalized depth through hand-set constants—being the true sound-source position, and on the diffusion model actually using that trajectory rather than ignoring it.

Editorial extensions

If this is right

  • If the claim holds, a video of an object moving across the screen yields binaural audio whose dominant ear channel shifts accordingly, because the trajectory condition enters the diffusion model at every frame.
  • The framework removes the need for external spatial metadata: any ordinary 2D video can be spatialized, provided the source can be detected and depth estimated.
  • Because training uses simulated moving sources with head-related impulse responses that include distance-induced time delays, the model can render dynamic sound fields rather than only fixed positions.
  • The coarse-grained variant, which estimates the source once per second on a 15-cell grid, scores nearly as well in user ratings, suggesting that low-rate trajectory cues are enough for perceived spatial consistency.
  • Objective audio-quality metrics place FoleySpace between monaural generation and the stereo-VAE baselines, so spatialization is achieved without a large sacrifice in audio fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported experiments do not include an ablation that removes the trajectory condition while keeping the monaural audio, so the causal role of the trajectory in producing spatial consistency is not directly demonstrated.
  • Editorial inference: with the estimator's azimuth error near 40 degrees on the evaluation set, the rendered interaural cues can point well away from the true source even when the model is faithful to its input, so applications requiring accurate localization would inherit this error.
  • Editorial inference: the geometric-center assumption treats the source as a compact point inside the detected box; large, occluded, or off-screen sound sources would produce a trajectory that misrepresents the audible source, so the method is best suited to single, visible, compact sources.
  • Editorial inference: the same trajectory-conditioned diffusion recipe could naturally extend to multiple simultaneous sources by conditioning on several trajectories, but the single-source dataset and 5×3 grid evaluation do not yet test that scenario.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FoleySpace, a framework for generating binaural spatial audio from silent 2D video. It first estimates the sound source's 2D position via YOLO-World object detection and depth via DepthMaster monocular depth estimation. A mapping mechanism (Eqs. (3)-(5)) converts these per-frame pixel/depth coordinates into a 3D trajectory T in a listener-centered sound field, with hand-set parameters S_y=1.47m and gamma=W/2, followed by outlier-based smoothing. A diffusion model (DiffWave backbone) is conditioned jointly on monaural audio generated by a pre-trained MMAudio model and on the trajectory T~; the model is trained on binaural audio synthesized by convolving VGGSound mono clips with HUTUBS HRIRs, with distance-varying resampling and moving-source simulation via segmentation and smoothing. Two variants are implemented: fine-grained (per-frame) and coarse-grained (5x3 grid, 1 Hz). Experiments include a 24-participant user study across 32 videos (PSS, SA, TA, SC, AQ), objective metrics (FD, KL, IS, IB-score, DeSync) on a newly constructed VGGSound-Solo evaluation set, and source-estimation MAE. The authors report better spatial perception consistency than See2Sound, AudioX, and ThinkSound, with preserved audio quality.

Significance. If the trajectory condition is truly causally effective, the framework provides a practical modular solution: off-the-shelf detection and depth models plus a synthetic HRIR-based training set, and a publicly released evaluation dataset. The user study is a genuine external validation, and the comparison to monaural MMAudio provides a useful sanity check. However, the central mechanism is not isolated, and the quantitative source-localization metric is partially self-referential. The work is therefore valuable but requires stronger causal evidence before the core claim, that visual trajectories drive spatial output, can be accepted.

major comments (3)
  1. [§III-D, §V] No experiment in Section V isolates the trajectory condition T~ (Section III-D): the diffusion model is always trained and evaluated with the full condition (monaural audio plus estimated trajectory), so the paper does not demonstrate that the trajectory, rather than the monaural audio or the conditioning architecture, is causally responsible for the reported spatial improvements in Table I and Table II. A direct test would replace T~ with a constant or zero trajectory, or with a trajectory from a different video, and measure PSS/SA or a spatial objective; without such a test, the claim that the visual trajectory guides spatial consistency is unsupported.
  2. [§IV-B, §V-D] The diffusion model is trained on simulated perfect trajectories (Section IV-B) but at inference uses estimated trajectories whose azimuth MAE is 39.62 degrees (Table III); no experiment shows how the model's spatial output degrades with estimation error, and the reported end-to-end results do not reveal whether the system is robust to this error or relies on the monaural audio alone. The authors should either train or fine-tune with noisy/estimated trajectories, or evaluate the generated audio's spatial accuracy as a function of trajectory estimation error.
  3. [§V-C2, Eqs. (3)-(5)] The MAE metric in Section V-C2 is defined by mapping both estimated and manually annotated pixel positions through the same hand-set linear camera model (Eqs. (3)-(5)) with S_y=1.47m and gamma=W/2; thus it measures consistency with the authors' own mapping rather than physical source direction, and the paper does not establish that 39.62 degrees of azimuth error is perceptually negligible. Because the evaluation set VGGSound-Solo is constructed and annotated by the authors, an independent or at least a more detailed validation of the mapping and annotation procedure (e.g., inter-annotator agreement, comparison against a geometric calibration) is needed to support the quantitative claims.
minor comments (4)
  1. [§IV-B, Eq. (8)] The summation in Eq. (8) uses index i in the loop but the formula uses subscript n; the notation is inconsistent and should be unified.
  2. [§V-A] The word 'construted' appears in the sentence beginning 'The training dataset for the diffusion models construted in Section IV'; it should be 'constructed'.
  3. [§IV-B] The word 'segement' appears twice near Eq. (8); it should be 'segment'.
  4. [Fig. 6] The caption begins 'Ilustration of binaural audio waveforms'; 'Ilustration' should be 'Illustration'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central spatial-audio claim rests on external human evaluations and recorded HRIR data, not on the framework's own outputs.

full rationale

The paper's core claim is that FoleySpace generates binaural audio whose perceived direction tracks the video source. That claim is supported by a 24-participant user study (Table I) comparing full pipelines on PSS and SA, which is external to the model's training objective and not derived from Eqs. (3)-(5). The HRIR training data are recorded impulse responses from the HUTUBS database, an external source; the synthesized training clips are constructed by convolving VGGSound monaural clips with those HRIRs along random trajectories, so the diffusion training signal is not generated by the model itself. The only self-referential elements are (i) the self-constructed VGGSound-Solo evaluation set (Ref. [32]) and (ii) the fact that both estimated and ground-truth angles are computed through the same 2D-to-3D mapping. Neither is circular: the annotation is manual and independent of the estimator, and using a common mapping for predicted and reference coordinates is standard for evaluating a localization pipeline. The MAE_alpha value of 39.62 degrees in Table III confirms that the estimator is not trivially equal to the labels. The lack of an ablation isolating the trajectory condition is a genuine experimental gap, but absence of a causal test is not the same as a claim reducing to its inputs, so it does not constitute circularity. No load-bearing step in the derivation chain is equivalent to an input by construction.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim leans on a chain of hand-set mapping constants and domain assumptions connecting pixels and depth to sound direction; none of these are derived from first principles, and the causal role of the trajectory is not separately tested. All training and evaluation use publicly available or author-created data, and no new physical entities are introduced.

free parameters (7)
  • S_y lateral sound-field extent = 1.47 m
    Hand-set maximum lateral distance in Eq. (3); fixes the scale of every mapped 3D trajectory and all derived azimuth/elevation errors.
  • gamma depth scaling factor = W/2, frame-width dependent
    Hand-chosen in Eq. (5) to simulate a reasonable listening distance; no perceptual or physical justification is given.
  • Trajectory smoothing outlier threshold = 95th percentile of frame-wise motion magnitudes
    Section III-C discards frames above this threshold; the threshold is fixed without sensitivity analysis and directly alters the conditioning trajectory.
  • Coarse spatial grid dimensions = 5 x 3 cells and 5 depth levels
    Section III-E defines 75 discrete positions; chosen ad hoc for the coarse scheme and for labeling VGGSound-Solo.
  • Coarse depth set = {1,2,3,4,5} m
    Section IV-A fixes five distances for coarse HRIR resampling; no justification beyond convenience.
  • Direction variation rate M = 200 for fine, 8 for coarse
    Section IV-B sets how often the source direction changes per audio sample; affects smoothness and perceptual continuity of training targets.
  • HRIR azimuth/elevation selection range = Azimuth [90,270], elevation [-40,40] for fine; 15 discrete directions for coarse
    Section IV dataset construction; the choice constrains what spatial cues the model can learn and may not match the mapping in Eq. (4).
assumptions (6)
  • domain assumption Bounding-box center is the sound source location.
    Section III-A uses YOLO-World boxes; the geometric center is treated as the source position, which fails when sound comes from a limb, mouth, or an unmodeled object.
  • domain assumption Monocular depth after min-max normalization gives reliable absolute distance.
    Section III-B and Eq. (5): DepthMaster returns relative depth; normalizing by the frame's min/max makes the mapped depth depend on scene content rather than calibrated physical distance.
  • domain assumption The linear mapping in Eqs. (3)-(4) preserves visual-spatial alignment without camera calibration.
    Assumes the audience is at the image center and a fixed physical scale S_y per video; no camera intrinsics, zoom, or moving camera are modeled.
  • domain assumption HRIR convolution plus cross-fade simulates perceptually valid moving binaural audio.
    Section IV-B: training target audio is synthesized this way; no room reflections, occlusion, Doppler, or dynamic HRTF updates are modeled.
  • ad hoc to paper The diffusion model uses the trajectory condition in a causal way.
    The user study and objective tables compare full systems; no ablation with removed or shuffled trajectories is reported (Sections III-D, V-E).
  • domain assumption The pretrained MMAudio monaural output corresponds to the detected visual source.
    Section III-D uses MMAudio as the mono generator; if the generated mono content does not match the detected object, the spatialization is anchored to the wrong semantic source.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FoleySpace: Vision-Aligned Binaural Spatial Audio Generation." pith.science (2026). https://pith.science/paper/BCSIUNXU

@misc{pith2026250812918,
  author       = {Pith},
  title        = {Pith review of: FoleySpace: Vision-Aligned Binaural Spatial Audio Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCSIUNXU}},
  note         = {Machine review of arXiv:2508.12918}
}
read the original abstract

Recently, with the advancement of AIGC, deep learning-based video-to-audio (V2A) technology has garnered significant attention. However, existing research mostly focuses on mono audio generation that lacks spatial perception, while the exploration of binaural spatial audio generation technologies, which can provide a stronger sense of immersion, remains insufficient. To solve this problem, we propose FoleySpace, a framework for video-to-binaural audio generation that produces immersive and spatially consistent stereo sound guided by visual information. Specifically, we develop a sound source estimation method to determine the sound source 2D coordinates and depth in each video frame, and then employ a coordinate mapping mechanism to convert the 2D source positions into a 3D trajectory. This 3D trajectory, together with the monaural audio generated by a pre-trained V2A model, serves as a conditioning input for a diffusion model to generate spatially consistent binaural audio. To support the generation of dynamic sound fields, we constructed a training dataset based on recorded Head-Related Impulse Responses that includes various sound source movement scenarios. Experimental results demonstrate that the proposed method outperforms existing approaches in spatial perception consistency, effectively enhancing the immersive quality of the audio-visual experience.

Figures

Figures reproduced from arXiv: 2508.12918 by the authors.

Figure 1
Figure 1. (a) The existing V2A model generates audio without [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed FoleySpace framework, consisting of three stages. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of coarse-grained spatio-temporal sound [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Construction of binaural audio for moving sound [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Ilustration of binaural audio waveforms compared with baselines. Red represents the left-ear audio, and blue represents [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An example of the user study questionnaire. The index [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A new large audio-image-text dataset and a FiLM-based audio-to-image model are presented, with strong in-house metrics but evaluation caveats on the external set.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Visual to sound: Generating natural sound for videos in the wild,

    Y . Zhou, Z. Wang, C. Fang, T. Bui, and T. L. Berg, “Visual to sound: Generating natural sound for videos in the wild,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3550–3558

    work page 2018

  2. [2]

    Generating visually aligned sound from videos,

    P. Chen, Y . Zhang, M. Tan, H. Xiao, D. Huang, and C. Gan, “Generating visually aligned sound from videos,”IEEE Trans- actions on Image Processing, vol. 29, pp. 8292–8302, 2020

    work page 2020

  3. [3]

    Taming visually guided sound gener- ation,

    V . Iashin and E. Rahtu, “Taming visually guided sound gener- ation,”arXiv preprint arXiv:2110.08791, 2021

    arXiv 2021

  4. [4]

    Diff-foley: Synchro- nized video-to-audio synthesis with latent diffusion models,

    S. Luo, C. Yan, C. Hu, and H. Zhao, “Diff-foley: Synchro- nized video-to-audio synthesis with latent diffusion models,” Advances in Neural Information Processing Systems, vol. 36, pp. 48 855–48 876, 2023

    work page 2023

  5. [5]

    Frieren: Efficient video-to-audio generation network with rectified flow matching,

    Y . Wang, W. Guo, R. Huang, J. Huang, Z. Wang, F. You, R. Li, and Z. Zhao, “Frieren: Efficient video-to-audio generation network with rectified flow matching,”Advances in Neural Information Processing Systems, vol. 37, pp. 128 118–128 138, 2024

    2024

  6. [6]

    Flow straight and fast: Learning to generate and transfer data with rectified flow,

    X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,”arXiv preprint arXiv:2209.03003, 2022

    arXiv 2022

  7. [7]

    V2a- mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,

    H. Wang, J. Ma, S. Pascual, R. Cartwright, and W. Cai, “V2a- mapper: A lightweight solution for vision-to-audio generation by connecting foundation models,” inProceedings of the AAAI 10 Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 492–15 501

    work page 2024

  8. [8]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

Show all 42 references
  1. [9]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  2. [10]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,”arXiv preprint arXiv:2301.12503, 2023

    2023 arXiv

  3. [11]

    Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis,

    H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y . Mitsufuji, “Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 28 901–28 911

    2025

  4. [12]

    Audiox: Diffusion transformer for anything-to-audio generation,

    Z. Tian, Y . Jin, Z. Liu, R. Yuan, X. Tan, Q. Chen, W. Xue, and Y . Guo, “Audiox: Diffusion transformer for anything-to-audio generation,”arXiv preprint arXiv:2503.10522, 2025

    2025 arXiv

  5. [13]

    Thinksound: Chain-of-thought reasoning in multi- modal large language models for audio generation and editing,

    H. Liu, J. Wang, K. Luo, W. Wang, Q. Chen, Z. Zhao, and W. Xue, “Thinksound: Chain-of-thought reasoning in multi- modal large language models for audio generation and editing,” arXiv preprint arXiv:2506.21448, 2025

    2025

  6. [14]

    On our perception of the direotion of a source of sound,

    L. Rayleigh, “On our perception of the direotion of a source of sound,”Proceedings of the Musical Association, vol. 2, no. 1, pp. 75–84, 1875

  7. [15]

    Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis,

    Y . Leng, Z. Chen, J. Guo, H. Liu, J. Chen, X. Tan, D. Mandic, L. He, X. Li, T. Qinet al., “Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis,”Advances in Neural Information Processing Systems, vol. 35, pp. 23 689–23 700, 2022

    2022

  8. [16]

    See-2-sound: Zero-shot spatial environment-to-spatial sound,

    R. Dagli, S. Prakash, R. Wu, and H. Khosravani, “See-2-sound: Zero-shot spatial environment-to-spatial sound,”arXiv preprint arXiv:2406.06612, 2024

    2024 arXiv

  9. [17]

    2.5 d visual sound,

    R. Gao and K. Grauman, “2.5 d visual sound,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 324–333

    2019

  10. [18]

    Visually-guided audio spa- tialization in video with geometry-aware multi-task learning,

    R. Garg, R. Gao, and K. Grauman, “Visually-guided audio spa- tialization in video with geometry-aware multi-task learning,” International Journal of Computer Vision, vol. 131, no. 10, pp. 2723–2737, 2023

    2023

  11. [19]

    Visually guided binaural audio generation with cross-modal consistency,

    M. Liu, J. Wang, X. Qian, and X. Xie, “Visually guided binaural audio generation with cross-modal consistency,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 7980–7984

    2024

  12. [20]

    Cross-modal generative model for visual-guided binaural stereo generation,

    Z. Li, B. Zhao, and Y . Yuan, “Cross-modal generative model for visual-guided binaural stereo generation,”Knowledge-Based Systems, vol. 296, p. 111814, 2024

    2024

  13. [21]

    Immersedif- fusion: A generative spatial audio latent diffusion model,

    M. Heydari, M. Souden, B. Conejo, and J. Atkins, “Immersedif- fusion: A generative spatial audio latent diffusion model,”arXiv preprint arXiv:2410.14945, 2024

    2024 arXiv

  14. [22]

    Diff-sage: End-to-end spatial audio generation using diffusion models,

    S. S. Kushwaha, J. Ma, M. R. Thomas, Y . Tian, and A. Bruni, “Diff-sage: End-to-end spatial audio generation using diffusion models,”arXiv preprint arXiv:2410.11299, 2024

    2024 arXiv

  15. [23]

    Visage: Video-to-spatial audio generation,

    J. Kim, H. Yun, and G. Kim, “Visage: Video-to-spatial audio generation,” inICLR, 2025

    2025

  16. [24]

    Omniaudio: Generating spatial audio from 360-degree video,

    H. Liu, T. Luo, K. Luo, Q. Jiang, P. Sun, J. Wang, R. Huang, Q. Chen, W. Wang, X. Li, S. Zhang, Z. Yan, Z. Zhao, and W. Xue, “Omniaudio: Generating spatial audio from 360-degree video,”arXiv preprint arXiv:2504.14906, 2025

    2025 arXiv

  17. [25]

    Yolo-world: Real-time open-vocabulary object detection,

    T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan, “Yolo-world: Real-time open-vocabulary object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 901–16 911

    2024

  18. [26]

    Depthmaster: Taming diffusion models for monocular depth estimation,

    Z. Song, Z. Wang, B. Li, H. Zhang, R. Zhu, L. Liu, P.-T. Jiang, and T. Zhang, “Depthmaster: Taming diffusion models for monocular depth estimation,”arXiv preprint arXiv:2501.02576, 2025

    2025 arXiv

  19. [27]

    Diffwave: A versatile diffusion model for audio synthesis,

    Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” arXiv preprint arXiv:2009.09761, 2020

    2009 arXiv

  20. [28]

    Vggsound: A large-scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2020, pp. 721–725

    2020

  21. [29]

    The hutubs hrtf database,

    F. Brinkmann, M. Dinakaran, R. Pelzer, J. J. Wohlgemuth, F. Seipl, and S. Weinzierl, “The hutubs hrtf database,”DOI, vol. 10, p. 14279, 2019

    2019

  22. [30]

    Interpolation of head-related transfer functions,

    R. Martin and K. McAnally, “Interpolation of head-related transfer functions,” Tech. Rep., 2007

    2007

  23. [31]

    gpurir: A python library for room impulse response simulation with gpu acceleration,

    D. Diaz-Guerra, A. Miguel, and J. R. Beltran, “gpurir: A python library for room impulse response simulation with gpu acceleration,”Multimedia Tools and Applications, vol. 80, no. 4, pp. 5653–5671, 2021

    2021

  24. [32]

    VGGSound-Solo,

    L. Zhao, “VGGSound-Solo,” IEEE DataPort, 2025. [Online]. Available: https://dx.doi.org/10.21227/m57x-cr16

    2025 doi

  25. [33]

    Image method for efficiently simulating small-room acoustics,

    J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustics,”The Journal of the Acoustical Society of America, vol. 65, no. 4, pp. 943–950, 1979

    1979

  26. [34]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880– 2894, 2020

    2020

  27. [35]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp...

    2017

  28. [36]

    Effi- cient training of audio transformers with patchout,

    K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, “Effi- cient training of audio transformers with patchout,”Interspeech 2022, 2022

    2022

  29. [37]

    Improved techniques for training gans,

    T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Rad- ford, and X. Chen, “Improved techniques for training gans,” Advances in neural information processing systems, vol. 29, 2016

    2016

  30. [38]

    Temporally aligned audio for video with autoregression,

    I. Viertola, V . Iashin, and E. Rahtu, “Temporally aligned audio for video with autoregression,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  31. [39]

    Dualspec: Text-to-spatial-audio generation via dual-spectrogram guided diffusion model,

    L. Zhao, S. Chen, L. Feng, J. Zhang, X.-L. Zhang, C. Zhang, and X. Li, “Dualspec: Text-to-spatial-audio generation via dual-spectrogram guided diffusion model,”arXiv preprint arXiv:2502.18952, 2025

    2025 arXiv

  32. [40]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inEuropean conference on computer vision. Springer, 2014, pp. 740–755

    2014

  33. [41]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  34. [42]

    Pyroomacoustics: A python package for audio room simulation and array pro- cessing algorithms,

    R. Scheibler, E. Bezzam, and I. Dokmani ´c, “Pyroomacoustics: A python package for audio room simulation and array pro- cessing algorithms,” in2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 351–355

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.