Pith. sign in

REVIEW 4 major objections 7 minor 44 references

In-the-wild Audio Spatialization with Flexible Text-guided Localization

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text prompts alone can guide where mono audio sounds like it comes from, and the controls generalize to real recordings.

desk verdict A solid applied spatialization paper with a new large dataset and a cleanly defined latent-diffusion method, but its 'in-the-wild' and 'semantic consistency' claims rest on evaluation metrics that do not actually validate spatial accuracy. read the letter →

arxiv 2506.00927 v1 pith:F2SBVUO4 submitted 2025-06-01 cs.SD cs.AIeess.AS

Tianrui Pan , Jie Liu , Zewen Huang , Jie Tang , Gangshan Wu This is my paper
classification cs.SDcs.AIeess.AS
keywords binauralaudiospatializationtext-guidedgenerationlatentdiffusionmonaural-to-binauralspatialreasoningsimulateddatasetinteractivecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that plain-language text descriptions of sound-source locations are enough to turn monaural audio into believable binaural audio, giving users interactive control over where sounds appear to come from. To do this, it builds a 376,000-sample simulated training set of binaural clips paired with detailed spatial captions, then trains a latent diffusion model to predict the difference between the left and right channels from a mono mix plus the text prompt. The authors claim the resulting model outperforms earlier visually guided and text-guided methods on both simulated data and two real-world recorded binaural datasets, and that a separate LLM-based spatial-reasoning test confirms the generated sounds carry the requested spatial semantics. If true, this would make spatial audio production a matter of typing where a sound should be, without requiring cameras or manual panning.

What carries the argument

The load-bearing mechanism is a latent diffusion model that predicts the binaural channel difference $A_{lr} = A_l - A_r$ from Gaussian noise, conditioned on a text embedding $T_e$ and a monaural audio embedding $A_e$. During inference the binaural channels are recovered by $\hat{A}_l = (A_{mono} + \hat{A}_{lr})/2$ and $\hat{A}_r = (A_{mono} - \hat{A}_{lr})/2$, which enforces that the spatial information is carried entirely by the learned interaural difference. A second key component is the text spatial coherence module: a classifier $P$ distinguishes whether an audio clip is the true $A_{lr}$ or its flipped version $A_{rl}$, and the resulting binary cross-entropy loss $L_{loc}$ finetunes the text encoder so that embeddings carry directional meaning, complementing the diffusion loss $L_\theta$.

What would settle it

Run a human listening test with head-tracked listeners on the FAIR-Play and YouTube-Binaural test clips: present ground-truth binaural audio and the model's output, and ask listeners to point to or label the perceived direction and distance of each source. If listeners systematically err on the generated clips—especially on elevation, front-back reversal, or distance—while the model's automatic metrics still score high, the claimed perceptual accuracy and generalization are not supported.

Watch

Extended reading notes

Core claim

The central claim is that binaural audio can be generated directly from monaural audio plus a flexible text prompt by learning the channel difference, $A_{lr} = A_l - A_r$, in a latent diffusion framework, rather than generating the full waveform or relying on visual frames. Training on the simulated SpatialTAS dataset—sourced from SpatialSoundQA and refined with GPT-4o captions covering 3D locations, distances, and relative positions between sources—the model conditions on FLAN-T5 text embeddings and an embedded mono audio signal, and is augmented by a flipped-channel coherence module that finetunes the text encoder to reason about left-right spatial cues. The paper reports that this approach outperforms PseudoBinaural and other baselines on generation metrics (FD, FAD, KL, IS) and spatial-understanding metrics (DOA, distance, relative-direction reasoning) on the SpatialTAS test set, and also achieves the best STFT/ENV scores on FAIR-Play and 360-degree YouTube-Binaural, thereby demonstrating that text-only guidance can transfer to real recordings of music, speech, and natural sounds.

Load-bearing premise

The simulated binaural audio in SpatialTAS, rendered with its spatial labels and GPT-4o captions, faithfully represents how real people perceive spatial sound, so that a model trained on it transfers to real recordings without a measurable simulation-to-real gap.

Editorial extensions

If this is right

  • If the claims hold, audio spatialization no longer requires visual frames or source-separation preprocessing, so sounds outside a camera's field of view can still be placed correctly.
  • Text prompts that specify both 3D locations and relative relationships between sources give fine-grained, interactive control, enabling a user to move a source from left to right or from 3m to 9m just by editing a sentence.
  • Training on a large simulated dataset with captions can transfer to real-world recordings of music, speech, and natural sounds, as shown by the reported gains on FAIR-Play and YouTube-Binaural.
  • The proposed LLM-based spatial reasoning evaluation provides a unified measure of both audio quality and spatial semantic fidelity, which could become a standard tool for assessing binaural generation.
  • Predicting only the channel difference in latent space is computationally more efficient than waveform-domain diffusion and still yields high-quality waveforms via a frozen VAE and HiFi-GAN vocoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is moving sound sources: since the model handles static 3D positions only, conditioning on time-varying text or on trajectory descriptors could allow it to render approaching or circling sounds, a limitation the authors acknowledge.
  • Because the spatial coherence module is trained solely on left-right flips, it may under-represent vertical and distance cues; a testable extension would be to add up-down or near-far flip tasks to the same BCE loss and measure whether directional reasoning accuracy on the vertical axis improves.
  • The reported gains on real datasets rely on GPT-4o-generated captions from video frames; a stricter evaluation would use ground-truth source positions from the videos or head-tracked listener judgments, which could reveal how much of the apparent generalization is due to caption quality.
  • The framework could plausibly be adapted to ambisonic or multi-channel input by replacing the scalar channel difference with a higher-dimensional spatial representation, connecting this work to scene-based audio rendering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a text-guided audio spatialization framework (TAS) that uses a latent diffusion model to predict the binaural difference signal Alr = Al - Ar from a monaural downmix Amono = Al + Ar, conditioned on FLAN-T5 text embeddings and an audio embedding of Amono. The authors construct SpatialTAS, a simulated dataset of ~376K binaural samples with GPT-4o-refined text prompts describing 3D sound-source locations and relative positions, and train the generator on it. Evaluation uses generation metrics (FD, FAD, KL, IS) and, on real datasets, waveform-similarity metrics (STFT, ENV, WAV, SNR), plus a spatial QA-based 'understanding' metric produced by a Llama-3.1-8B judge fine-tuned on SpatialTAS. The paper claims superior performance on both simulated and real-recorded datasets and that text prompts give flexible, semantically consistent spatial control.

Significance. If the results hold, the contribution would be practically useful: a large text-annotated simulated binaural dataset, a well-defined conditional generative objective (Eqs. 1-2), and a training scheme that includes a flipped-channel spatial-coherence module. The algebraic formulation is clear, the method is reproducible in principle, and the authors report ablations showing the value of the text condition and the flipper module. However, the evaluation as presented does not support the strongest claims. The understanding metric is circular because the judge and the generator are trained on the same simulated distribution. The real-data metrics are full-waveform distances that do not isolate interaural spatial cues, so they cannot substantiate 'superior generalization and accuracy' in spatial terms. The paper also lacks human listening validation and any statistical significance testing. These are load-bearing issues: they concern precisely the headline claims of the abstract. I therefore agree with the stress-test concern that the real-data and semantic-consistency claims rest on unvalidated metrics.

major comments (4)
  1. [§3.3 and Table 2] The spatial-understanding metric is not independent evidence of spatial semantics. The Llama-3.1-8B judge is fine-tuned on SpatialTAS (Sec. 3.3), and the generator is trained on the same simulated dataset (Secs. 3.1-3.2). Consequently, high DOA/DE and reasoning accuracy on the SpatialTAS test set may reflect the judge's familiarity with the simulation's HRTF and label distribution rather than perceptually valid spatial cues. The paper reports no human listening study and no agreement check between the LLM judge and human listeners. To support the abstract's 'semantic consistency' claim, please add a human listening test on a subset, or validate the judge on real binaural recordings with human-verified spatial labels, and report the human-LLM agreement.
  2. [§5.2, Tables 3-4] The real-dataset evaluation does not measure spatialization accuracy. STFT, ENV, WAV, and SNR are full-waveform similarity metrics between the generated and ground-truth binaural signals. Because the input is Amono = Al + Ar (Eq. 1), which already contains most of the energy of both target channels, a model can achieve low waveform distance by approximately preserving Amono while producing incorrect interaural time/level differences (ITD/ILD). The paper reports no spatial-specific metric on FAIR-Play or YouTube-Binaural (e.g., ITD/ILD error, localization accuracy, or a human listening test). Therefore, the claim that the method 'outperforms existing methods ... on real-recorded datasets, demonstrating superior generalization and accuracy' is not supported by the presented evidence. Please add spatial-specific evaluation or temper the claim.
  3. [§5.2 and Appendix A] The text prompts for the real datasets are generated by GPT-4o from video frames (Appendix A), but sound sources in FAIR-Play and YouTube-Binaural can be off-screen or visually ambiguous. The model is then conditioned on these potentially inaccurate prompts, and the generated audio is compared against ground-truth binaural audio whose actual source locations may differ from the visual caption. This mismatch makes the real-data results hard to interpret. The paper should either verify the accuracy of the generated captions (e.g., by human annotation) or discuss the effect of prompt inaccuracy on the metric values.
  4. [Tables 2-4 (general)] None of the reported metric values are accompanied by error bars, confidence intervals, or significance tests. With the small margins in some comparisons (e.g., Table 3: STFT 0.773 vs. 0.787 and ENV 0.126 vs. 0.128), it is unclear whether the differences are statistically meaningful. Please report means and standard deviations over multiple evaluation runs or across dataset subsets, and, where appropriate, perform paired significance tests (e.g., bootstrap or Wilcoxon).
minor comments (7)
  1. [§5.1, paragraph after Table 2] The reported 'improvement of 5.80% and 7.17%' are absolute percentage-point reductions in error rate (12.79-6.99 and 15.33-8.16), not relative improvements; please report these consistently (e.g., as percentage points or as relative percentages).
  2. [Figure 2 caption] The caption says 'adds noise to the monaural audio Amono' but the method adds noise to the latent of Alr, with Amono used as conditioning via Ae; please correct the caption to match the description in Sec. 3.2.
  3. [§3.2, Eq. (5)] The expression P(Alr|Arl,Te) is confusing; it appears to denote the classifier's output probability rather than a conditional distribution. Also, the definition of the ground-truth indicator g as 'g = P(Alr|Arl,Te)' is circular; g should be defined as the binary flip label. Please clarify the notation.
  4. [Table 3] The baseline 'TAS (2024b)' shares the same acronym as the proposed TAS framework, which is confusing; rename the baseline (e.g., 'TAS (Li et al., 2024b)') or use the authors' full method name.
  5. [§4.1] The guidance scale is called λ in the text but γ in Eq. (4); please unify the notation.
  6. [Figure 3] The label 'Loundness' is a typo for 'Loudness'; also, the caption for (c) should specify whether the spectrograms show the binaural difference or the individual channels.
  7. [§3.2] The latent shape is introduced as T/r × F/r × C, but later the text mentions a 'latent dimension d of 8'; please clarify whether d refers to C or another quantity and define all symbols.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generation objective is a direct supervised regression on the binaural channel difference, and the concerns about the LLM judge and waveform metrics are evaluation-validity limitations, not derivation loops.

full rationale

The paper's core derivation is self-contained rather than circular. The model is trained to predict the binaural channel difference Alr = Al − Ar (Eq. 1) from monaural audio and text embeddings, with a diffusion loss Lθ (Eq. 3) and an auxiliary flip-classification loss Lloc (Eq. 5). None of these losses is defined in terms of the evaluation metrics or the downstream judge; the generator is not optimized to fool the Llama-based assessment model. The SpatialTAS dataset and the fine-tuned Llama-3.1-8B judge do share the same simulated distribution, which weakens the independence of the semantic-consistency evidence, but this is a validity and generalization concern rather than a logical circularity: the judge is a separate model, not a fitted parameter renamed as a prediction, and the paper never claims the judge was used in training. Likewise, the STFT/ENV/WAV/SNR comparisons on FAIR-Play and YouTube-Binaural are external waveform-level metrics against real recordings; even if they are not sensitive to spatial accuracy, they are not constructed from the model's own outputs. There are no load-bearing self-citations and no uniqueness theorems imported from the authors' prior work. The legitimate weaknesses—lack of human listening tests and possible simulation-to-real mismatch—are correctness risks, not circular reasoning.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that simulated binaural audio with LLM-generated text labels is a faithful proxy for real spatial audio, and that a language model judge trained on the same simulation can measure spatial semantics. Standard diffusion and VAE machinery is inherited from cited work (AudioLDM2, FLAN-T5, SpatialSoundQA) rather than derived here. No free scalar parameters are fitted to the target result beyond standard training hyperparameters; the model weights themselves are learned from data.

free parameters (2)
  • Classifier-free guidance scale gamma = 2.5
    Hand-chosen in Sec. 4.1; controls the strength of text/audio conditioning in Eq. (4) and affects all reported generation metrics.
  • CFG condition dropout probability = 0.1
    Chosen in Sec. 3.2; with probability 0.1 the condition pair (Te, Ae) is replaced by zeros during training, enabling classifier-free guidance.
assumptions (5)
  • domain assumption Simulated binaural audio in SpatialTAS, from SpatialSoundQA, is a valid proxy for real binaural audio.
    The model trains entirely on this simulation (Sec. 3.1) and is claimed to generalize to real recordings (Sec. 5.2); no HRTF or room-modeling fidelity analysis is provided.
  • domain assumption GPT-4o-generated captions from a single video frame correctly describe sound source locations in FAIR-Play and YouTube-Binaural.
    These captions are the evaluation prompts for real datasets (Sec. 4.2, Appendix A); caption errors directly bias the real-data results.
  • domain assumption The Llama-3.1-8B judge fine-tuned on SpatialTAS is a valid measure of spatial semantic coherence.
    The understanding metrics depend on this judge (Sec. 3.3); it is trained on the same distribution as the generator and is not validated against human listeners.
  • standard math Alr is a sufficient target: with Alr known, Eq. (2) reconstructs Al and Ar exactly.
    Algebraically true given Amono = Al + Ar (Eq. 1), but whether the model can infer Alr from monaural audio and text is the learned task.
  • domain assumption AudioSet sound events cover acoustic diversity needed for music, speech, and nature sounds.
    Training events are drawn from AudioSet (Sec. 3.1); no coverage analysis against the real-evaluation distribution is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-the-wild Audio Spatialization with Flexible Text-guided Localization." pith.science (2026). https://pith.science/paper/F2SBVUO4

@misc{pith2026250600927,
  author       = {Pith},
  title        = {Pith review of: In-the-wild Audio Spatialization with Flexible Text-guided Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F2SBVUO4}},
  note         = {Machine review of arXiv:2506.00927}
}
read the original abstract

To enhance immersive experiences, binaural audio offers spatial awareness of sounding objects in AR, VR, and embodied AI applications. While existing audio spatialization methods can generally map any available monaural audio to binaural audio signals, they often lack the flexible and interactive control needed in complex multi-object user-interactive environments. To address this, we propose a Text-guided Audio Spatialization (TAS) framework that utilizes flexible text prompts and evaluates our model from unified generation and comprehension perspectives. Due to the limited availability of premium and large-scale stereo data, we construct the SpatialTAS dataset, which encompasses 376,000 simulated binaural audio samples to facilitate the training of our model. Our model learns binaural differences guided by 3D spatial location and relative position prompts, augmented by flipped-channel audio. It outperforms existing methods on both simulated and real-recorded datasets, demonstrating superior generalization and accuracy. Besides, we develop an assessment model based on Llama-3.1-8B, which evaluates the spatial semantic coherence between our generated binaural audio and text prompts through a spatial reasoning task. Results demonstrate that text prompts provide flexible and interactive control to generate binaural audio with excellent quality and semantic consistency in spatial locations. Dataset is available at \href{https://github.com/Alice01010101/TASU}

Figures

Figures reproduced from arXiv: 2506.00927 by the authors.

Figure 1
Figure 1. We propose the Text-guided Audio Spatializa [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The detailed structure for the text-guided audio spatialization model. The dashed lines indicate processes that occur only during training. We train a latent diffusion model that adds noise to the monaural audio Amono based on the concatenation of the encoded text embedding Te and audio embedding Ae. During inference, the model predicts the binaural difference Alr from the Gaussian noise. Additionally, we finetune a… view at source ↗
Figure 3
Figure 3. Ablations for text prompts. We systemati￾cally alter the direction, distance, and relative position in the text prompts, and present the differences observed before and after these changes. differences in spectrograms when changing the rel￾ative position from "is nearer to" to "is farther away from". Given the significant frequency differences between the sounds of a baby crying and dance mu￾sic, we can analyze the … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization for binaural difference pre￾diction. We present the binaural difference results using various spatial text prompts, including 3D sound source descriptions and relative position descriptions for music, speech, and natural sounds. closely aligns with the gr…
Figure 5
Figure 5. Figure 5: illustrates the GPT-4o prompt we use for image caption engineering. 360∘ Video Normal Video Prompt Task instruction Prompt Based on a frame of video, return a caption about sound source’s attributes Identify sound sources description based on frame, you can …… Sound So…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 23 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  4. [4]

    Rishit Dagli, Shivesh Prakash, Robert Wu, and Houman Khosravani. 2024. See-2-sound: Zero-shot spatial environment-to-spatial sound. arXiv preprint arXiv:2406.06612

    arXiv 2024

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  6. [6]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    work page 2023

  7. [7]

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. 2024 a . Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301

    arXiv 2024

  8. [8]

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. 2024 b . Stable audio open. arXiv preprint arXiv:2407.14358

    arXiv 2024

Show all 44 references
  1. [9]

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2021. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829--852

    2021

  2. [10]

    Ruohan Gao and Kristen Grauman. 2019. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324--333

    2019

  3. [11]

    Rishabh Garg, Ruohan Gao, and Kristen Grauman. 2023. Visually-guided audio spatialization in video with geometry-aware multi-task learning. International Journal of Computer Vision, 131(10):2723--2737

    2023

  4. [12]

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal proces...

    2017

  5. [13]

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, s...

    2017

  6. [14]

    Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598

    2022 arXiv

  7. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [16]

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019 a . Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

    2019

  9. [17]

    Hansung Kim, Luca Remaggi, Philip JB Jackson, and Adrian Hilton. 2019 b . Immersive spatial audio reproduction for vr/ar using room acoustic modelling from 360 images. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 120--126. IEEE

    2019

  10. [18]

    Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  11. [19]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020 a . Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022--17033

    2020

  12. [20]

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020 b . Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880--2894

    2020

  13. [21]

    Yeonghyeon Lee, Inmo Yeon, Juhan Nam, and Joon Son Chung. 2023. https://arxiv.org/abs/2309.13664 Voiceldm: Text-to-speech with environmental context . Preprint, arXiv:2309.13664

    2023 arXiv

  14. [22]

    Dingzeyu Li, Timothy R Langlois, and Changxi Zheng. 2018. Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG), 37(4):1--12

    2018

  15. [23]

    Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024 a . Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26669--26678

    2024

  16. [24]

    Zhaojian Li, Bin Zhao, and Yuan Yuan. 2024 b . Tas: Personalized text-guided audio spatialization. In ACM Multimedia 2024

    2024

  17. [25]

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503

    2023 arXiv

  18. [26]

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2024 a . Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing

    2024

  19. [27]

    Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. 2024 b . Visually guided binaural audio generation with cross-modal consistency. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7980--7984. IEEE

    2024

  20. [28]

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. 2024 c . Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886

    2024 arXiv

  21. [29]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  22. [30]

    Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. 2018. Self-supervised generation of spatial audio for 360 video. Advances in neural information processing systems, 31

    2018

  23. [31]

    Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. 2022. Beyond mono to binaural: Generating binaural audio from mono audio with depth and cross modal attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3347--3356

    2022

  24. [32]

    Kranthi Kumar Rachavarapu, Vignesh Sundaresha, AN Rajagopalan, et al. 2021. Localize to binauralize: Audio spatialization from visual sound source localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1930--1939

    2021

  25. [33]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  26. [34]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695

    2022

  27. [35]

    Saksham Singh Kushwaha, Jianbo Ma, Mark RP Thomas, Yapeng Tian, and Avery Bruni. 2024. Diff-sage: End-to-end spatial audio generation using diffusion models. arXiv e-prints, pages arXiv--2410

    2024

  28. [36]

    Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502

    2020 arXiv

  29. [37]

    Peiwen Sun, Sitong Cheng, Xiangtai Li, Zhen Ye, Huadai Liu, Honggang Zhang, Wei Xue, and Yike Guo. 2024. Both ears wide open: Towards language-driven spatial audio generation. arXiv preprint arXiv:2410.10676

    2024 arXiv

  30. [38]

    Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821

    2023 arXiv

  31. [39]

    Xudong Xu, Dejan Markovic, Jacob Sandakly, Todd Keebler, Steven Krenn, and Alexander Richard. 2024. Sounding bodies: modeling 3d spatial sound of humans using body pose and audio. Advances in Neural Information Processing Systems, 36

    2024

  32. [40]

    Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. 2021. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15485--15494

    2021

  33. [41]

    Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watanabe, and Helen Meng. 2023. https://arxiv.org/abs/2310.00704 Uniaudio: An audio foundation model toward universal audio gener...

    2023 arXiv

  34. [42]

    Wen Zhang and Jie Shao. 2021. Multi-attention audio-visual fusion network for audio spatialization. In Proceedings of the 2021 International Conference on Multimedia Retrieval, pages 394--401

    2021

  35. [43]

    Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. 2024. Bat: Learning to reason about spatial sounds with large language models. arXiv preprint arXiv:2402.01591

    2024 arXiv

  36. [44]

    Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. 2020. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XII 16,...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.