BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models

Alexander Richard; Chenliang Xu; Dejan Markovic; Frank Yu; Israel D. Gebru; Jacob Sandakly; Samuel Hassel; Steven Krenn; Susan Liang; Todd Keebler

REVIEW 5 major objections 7 minor 24 references

BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BinauralFlow claims that conditional flow matching over complex spectrograms, run through a causal U-Net with a buffer-bank streaming pipeline, renders binaural speech from mono audio that listeners mistake for real recordings 42% of the…

desk verdict Solid flow-matching binaural rendering with a real streaming mechanism, but the streaming pipeline is an unmeasured approximation and the public-results overclaim. read the letter →

arxiv 2505.22865 v1 pith:O7KVEGAC submitted 2025-05-28 cs.SD cs.AIeess.AS

Susan Liang , Dejan Markovic , Israel D. Gebru , Steven Krenn , Todd Keebler , Jacob Sandakly , Frank Yu , Samuel Hassel

show 2 more authors

Chenliang Xu Alexander Richard

This is my paper · ORCID

classification cs.SDcs.AIeess.AS

keywords binauralaudiosynthesisflowmatchingstreaminginferencecausalU-Netspatialgenerativespeechrenderingreal-time

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that binaural speech rendering should be treated as a generative problem rather than a regression problem. Instead of predicting one binaural signal, a conditional flow matching model learns a distribution of plausible two-channel renderings of a mono signal, conditioned on the poses of the speaker and listener. The authors also build a fully causal U-Net and a continuous inference pipeline, so the same generative model can run chunk-by-chunk without losing binaural cues, room reverberation, or ambient noise. If correct, this would give VR, AR, and live audio applications a high-fidelity binaural renderer that runs in six function evaluations with a real-time factor of 0.239 on a single 4090 GPU, and a perceptual study reports that listeners confuse its output with real recordings 42% of the time, where 50% is chance.

What carries the argument

The central object is a conditional flow matching model over complex spectrograms: the flow $\phi_t(z)=ty+(1-t)z$ linearly interpolates between a Gaussian-noised mono spectrogram $z\sim\mathcal{N}(x,\sigma^2 I)$ and the binaural spectrogram $y$, and the network learns the vector field $v_t=y-z$ by minimizing $\mathcal{L}_{\mathrm{CFM}}=\mathbb{E}\|u_t(\phi_t(z),p_{\mathrm{rx}},p_{\mathrm{tx}},x;\theta)-(y-z)\|$. To make this streamable, the paper replaces the usual U-Net with a causal 2D U-Net whose convolution blocks use one-side padding to keep the receptive field in the past, and it adds a continuous inference pipeline: streaming STFT/ISTFT with buffer padding, a buffer bank that stores each denoising step's hidden states, a midpoint ODE solver, and an early skip schedule that drops the first half of the time interval. The machinery's job is to let an iterative generative model run chunk-by-chunk in real time without destroying the binaural cues, reverb, or ambient noise that the model was trained to generate.

What would settle it

A forced-choice listening test comparing the model's output under the early skip schedule with the same model run on the standard full time schedule at equal function evaluations would settle the weakest assumption. If listeners reliably prefer or distinguish the full-schedule output, the early-skip premise fails, and the streaming real-time factor of 0.239 would need to be recomputed at a higher number of function evaluations.

Watch

Extended reading notes

Core claim

The paper's central claim is that casting binaural rendering as conditional flow matching, rather than direct regression, is what makes high-quality rendering possible. The model is trained to push a noisy version of the mono spectrogram along straight-line paths to the binaural spectrogram while conditioning on transmitter and receiver poses, so it can synthesize the room reverb and ambient noise that are absent from the mono input and that behave stochastically. The authors further claim that with a fully causal U-Net and a continuous inference pipeline, this generative model runs streamably at six function evaluations and produces audio that a perceptual study finds nearly indistinguishable from real recordings: a 42% confusion rate in the A-B test, against a 50% chance ceiling, with environment and spatialization MUSHRA scores of 68.4 and 83.1 versus 87.4 and 89.9 for ground truth. On their dataset it also reports better L2, magnitude-L2, and phase error than SGMSE, BinauralGrad, and regression baselines, and it does so with the fewest function evaluations among generative baselines.

Load-bearing premise

The load-bearing premise is that skipping the first half of the multi-step noise-removal process leaves the final audio quality unchanged; the paper supports this only with a speculation that the second half can correct errors from the first half.

Editorial extensions

If this is right

Streaming binaural rendering becomes feasible in live settings: with six function evaluations the model renders 0.683 seconds of audio in 0.163 seconds, a real-time factor of 0.239 on a single 4090 GPU.
Because the model is generative rather than regression-based, it can synthesize room reverberation and ambient noise that are absent from the mono input, which regression baselines cannot do.
The early skip schedule halves the number of denoising steps without a noticeable quality loss, while the authors report that skipping the second half degrades background-noise modeling.
Pretraining on 7,700 hours of loudspeaker-to-artificial-head data lets the model perform well when fine-tuned on only 1–5% of real human binaural data, reducing the cost of collecting real recordings.
The buffer bank preserves per-step hidden states so chunk boundaries do not create audible discontinuities, which the paper demonstrates by comparing continuous versus non-streaming spectrograms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 42% confusion rate transfers beyond the fixed room and speakers used in the perceptual study, binaural rendering could become a drop-in component for live telepresence and AR audio where latency budgets are tight.
The early-skip result suggests that, for flow-matching audio models, the second half of the trajectory carries the perceptually critical refinement; applying the same schedule to speech enhancement or audio codecs would test whether this is a general property of the flow trajectory.
Because the buffer bank is not specific to flow matching, iterative diffusion models could adopt the same per-step buffering mechanism to become streamable, potentially extending the approach beyond the method presented here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

BinauralFlow is a solid applied-ML paper, but the headline claim about streamable high-quality rendering is weaker than it looks. The genuinely new bits are the conditional flow matching formulation for binaural speech and the causal U-Net with a per-step buffer bank; the soft spot is that the streaming pipeline is a heuristic the paper never measures.

What is actually new and good: applying flow matching to binaural rendering instead of diffusion is a sensible step, and the causal U-Net with buffer banks for streaming generative inference is a real mechanism beyond CosyVoice 2's chunk-aware masking. The CFM derivation is standard and internally consistent, and the in-house dataset with real participants, posed tracking, and 10 hours of audio is valuable. The perceptual study with 23 listeners showing 42% AB confusion rate against 3-21% for baselines is a strong result, and the large-scale pretraining on 7,700 hours is a nice addition. I believe the core contribution is publishable.

The soft spots, in order of importance. First, the stress-test note is right: the continuous inference pipeline is not solving the global ODE. Each chunk samples fresh noise z ~ N(x, sigma^2 I), so the per-chunk trajectory is statistically independent of the previous chunk, yet the buffer bank caches hidden activations from previous chunks' trajectories. This is a streaming approximation, not an exact chunked solver. The paper only shows a qualitative spectrogram in Figure 6, with no quantitative comparison of streaming vs non-streaming outputs or streaming-vs-GT versus offline-vs-GT. That is a load-bearing gap for the 'streamable high-quality' claim. Second, NFE and the early skip schedule are tuned on test data without a separate validation protocol; this is a minor issue in practice but should be fixed in revision. Third, the public-dataset results in Appendix J are mixed: Wave L2 is 0.192 vs BinauralGrad's 0.128, and Phase L2 is 0.918 vs 0.837. The abstract's blanket 'superiority' overstates this. Fourth, no code, data, or checkpoints are released, so the headline numbers are not independently reproducible. And the 'nearly indistinguishable' phrasing is oversold: 42% is above chance, meaning listeners do distinguish more than half the time, though far less than with the baselines.

Who should read this: people working on binaural or spatial audio rendering, and anyone building streaming generative models. It deserves a serious referee, with the main request being a quantitative evaluation of the streaming approximation and a toned-down set of claims. I would not cite it in my own work in the next year unless I was actively working on streaming generative audio, but I would bring it to a reading group to discuss the buffer-bank mechanism and the unmeasured approximation.

Referee Report

5 major / 7 minor

Summary. The paper introduces BinauralFlow, a conditional flow matching model for binaural speech synthesis from monaural audio and speaker/listener poses. The method defines a linear OT flow between Gaussian noise centered at the mono spectrogram and the binaural spectrogram (Eqs. 2-7), trains a causal U-Net to regress the vector field, and deploys a streaming inference pipeline with streaming STFT/ISTFT, a step-wise buffer bank, a midpoint solver, and an early skip schedule. Experiments on a newly recorded 10-hour in-house dataset report lower L2/Mag/Phase errors than DSP, regression, and generative baselines at NFE=6, and a perceptual study reports a 42% A-B confusion rate against real recordings. The appendix reports additional results on a public dataset, ablation of flow matching variants, solver choices, and a 7,700-hour pretraining strategy.

Significance. If the claims hold, BinauralFlow would be a practical step toward real-time generative binaural rendering: the flow matching formulation is standard and internally consistent, the causal U-Net is a careful adaptation, and the streaming pipeline with buffer bank is a nontrivial engineering contribution. The 7,700-hour pretraining result (Figure 7) and the public demo videos strengthen the paper. However, the central 'streamable high-quality' claim is not yet quantitatively validated: the streaming pipeline's per-chunk noise resampling is an approximation whose error is only shown qualitatively, and the NFE/early-skip choices are not protected against test-set selection bias. The public-dataset results also show two metrics where the method is worse than BinauralGrad, which tempers the 'consistent outperformance' statement.

major comments (5)

[Section 3.4, Figure 2] The streaming pipeline resamples noise independently for each chunk (z ~ N(x, sigma^2 I)), while reusing buffers from previous chunks at the same denoising step. This means the per-chunk ODE trajectories are statistically independent and the global output is not the solution of the flow ODE over the full sequence. The paper reports only a qualitative spectrogram comparison in Figure 6; no quantitative metric (e.g., L2/Mag/Phase between streaming and non-streaming outputs, or streaming-vs-GT versus offline-vs-GT) is given. Because 'streamable' and 'high-quality' are joint contributions (abstract, Section 5), this missing validation leaves the central claim unsupported.
[Sections 3.4 and 4.2] The number of function evaluations (NFE=6) and the early skip schedule are chosen after inspecting results that appear to be on the test set (Figure 3(b), Figure 4, Table 1). No separate validation split or repeated-measure protocol is described, so the reported NFE-6 gains may partly reflect selection bias. The paper should either fix hyperparameters on a validation set or provide error bars over multiple random seeds or test subsets.
[Section 3.4 and Table 4] The speed claim (RTF=0.239 at NFE=6) hinges on the early skip schedule, whose justification is explicitly speculative ('We speculate that flow matching may be able to correct the errors...'). No quantitative comparison of early skip versus full schedule is provided for the streaming setting, and the speculative mechanism is not tested (e.g., by measuring error as a function of skip fraction or chunk length). Without this, the real-time factor does not establish a reliable operating point for continuous inference.
[Section 4.4, Table 2] The headline '42% confusion rate' is reported without confidence intervals or statistical tests across 23 participants. A 42% CR is close to the 50% chance level, but the reader cannot tell whether this is distinguishable from chance or from the 21% of the closest baseline without a binomial confidence interval or a paired significance test. The same applies to the ABX (30%) and MUSHRA (68.4 +/- 23.4, 83.1 +/- 18.9) numbers; the MUSHRA scores do have standard deviations but no test against the Ground Truth row.
[Appendix J, Table 7] The text states the model 'performs on par with it in the Wave and Phase metrics,' but the numbers show BinauralFlow is worse than BinauralGrad on Wave L2 (0.192 vs 0.128) and Phase L2 (0.918 vs 0.837). This contradicts the main-text claim of 'consistently outperforms all baselines' (Section 4.2). The appendix should be reworded, and the main text should acknowledge that the advantage over BinauralGrad does not hold on all metrics of the public dataset.

minor comments (7)

[Section 3.2] The same symbols x and y are used for time-domain signals and STFT spectrograms, which is confusing (e.g., 'x = STFT(x)' and later 'we convert them to real numbers'). Please introduce distinct notation for spectral quantities.
[Section 3.3] The phrase 'one-side padding of size 2' is not self-explanatory; please specify that the padding is applied on the time axis only (past side) and state the padding used on the frequency axis for the 3x3 causal convolution.
[Appendix C, Algorithm 2] The midpoint solver uses delta = 2/n, so the number of outer iterations is n/2 while NFE = n. This is correct, but it should be stated explicitly in the main text to avoid confusing the reader about the number of integration steps versus function evaluations.
[Figure 3(b)] The spectrogram panels have no axis labels or color scale; adding them would make the qualitative claim about the early skip schedule easier to verify.
[Appendix D] The sentence 'with one participant used as a part of the training set' is ambiguous; please clarify whether the test speakers were entirely unseen or whether there is any overlap between training and test participants.
[Appendix F] There is a typo: 'sytem' should be 'system'.
[References] The reference 'Kingma, D. P. Adam: A method for stochastic optimization' is incomplete; use the full author list (Kingma and Ba) for the Adam optimizer citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the flow-matching objective is defined from paired data and all central claims are evaluated on held-out recordings and human listeners.

full rationale

The paper's derivation chain is self-contained. The conditional flow matching objective (Eq. 7) trains the network to regress the vector field y - z on interpolants phi_t(z) = t y + (1-t) z, which is the standard CFM loss; the target y is the recorded binaural spectrogram, not a quantity derived from the model's own output. At inference the model is tested against held-out mono-binaural pairs (Section 4.1) and against human listeners (Section 4.4), so the headline quality claim does not reduce to a training objective or to a fitted constant. The early skip schedule is an empirical inference-time hyperparameter (Section 3.4), not a predicted quantity, and its justification is explicitly speculative. Self-citations to Richard et al. 2021 (WarpNet) are used only as a baseline and a source of evaluation metrics, not as evidence for the central claim; no uniqueness theorem or prior-work conclusion is invoked to force the design. The streaming-continuity approximation (fresh noise per chunk with cached buffers) is an empirical validation gap, not a circular step, because it does not make any evaluated quantity equal to its own input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard flow matching theory and on the empirical validity of several engineering assumptions (causal downsampling, streaming STFT reconstruction, early skip). These are domain assumptions, not invented physical entities. The buffer bank and early skip schedule are mechanisms, not entities in the graviton sense.

free parameters (3)

Noise standard deviation sigma = 0.5
Set by hand (Appendix B); controls the source distribution N(x, sigma^2 I) in the flow, affecting generation randomness and output detail.
NFE (number of function evaluations) = 6
Chosen empirically in Section 4.2 and Table 4 as the quality/speed trade-off point; not derived from first principles.
Early skip schedule = skip first half of [0,1]
Adopted after comparing early vs late skip on the test data (Figure 3); the paper speculates rather than proves why it works (Section 3.4).

assumptions (4)

domain assumption The optimal transport flow phi_t(z) = t y + (1-t) z induces a vector field v_t = y - z that the network can learn (Eqs. 2-6).
Standard conditional flow matching assumption from Lipman et al.; the paper relies on it without new proof.
domain assumption Streaming STFT/ISTFT with prepended buffers yields reconstructions indistinguishable from offline STFT/ISTFT.
Invoked in Section 3.4; the paper does not analyze reconstruction error or phase consistency across chunk boundaries.
domain assumption Causal 2D convolutions, causal down/up-sampling, and per-frame GroupNorm preserve exact temporal causality so that buffers fully capture history.
Stated in Section 3.3; no formal proof or ablation shows that the 4x4 stride-2 downsampling cannot leak future frames.
ad hoc to paper The early skip schedule does not hurt quality because the flow corrects first-half errors in the second half.
Explicitly speculative in Section 3.4: 'We speculate that flow matching may be able to correct the errors from the first half during the second half of inference'.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models." pith.science (2026). https://pith.science/paper/O7KVEGAC

@misc{pith2026250522865,
  author       = {Pith},
  title        = {Pith review of: BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O7KVEGAC}},
  note         = {Machine review of arXiv:2505.22865}
}

abstract

Binaural rendering aims to synthesize binaural audio that mimics natural hearing based on a mono audio and the locations of the speaker and listener. Although many methods have been proposed to solve this problem, they struggle with rendering quality and streamable inference. Synthesizing high-quality binaural audio that is indistinguishable from real-world recordings requires precise modeling of binaural cues, room reverb, and ambient sounds. Additionally, real-world applications demand streaming inference. To address these challenges, we propose a flow matching based streaming binaural speech synthesis framework called BinauralFlow. We consider binaural rendering to be a generation problem rather than a regression problem and design a conditional flow matching model to render high-quality audio. Moreover, we design a causal U-Net architecture that estimates the current audio frame solely based on past information to tailor generative models for streaming inference. Finally, we introduce a continuous inference pipeline incorporating streaming STFT/ISTFT operations, a buffer bank, a midpoint solver, and an early skip schedule to improve rendering continuity and speed. Quantitative and qualitative evaluations demonstrate the superiority of our method over SOTA approaches. A perceptual study further reveals that our model is nearly indistinguishable from real-world recordings, with a $42\%$ confusion rate.

Figures

Figures reproduced from arXiv: 2505.22865 by the authors.

**Figure 1.** Overview of our BinauralFlow framework. (a) shows the causal U-Net architecture. Our causal U-Net takes as input the flow ϕt(z) as well as four conditions t, prx, ptx, and x, and outputs a predicted vector field. The U-Net consists of several Causal 2D Conv Blocks in the contracting and expanding parts. (b) displays the Causal 2D Conv Block. We design fully causal convolution, down/up-sampling, and normalization lay… view at source ↗

**Figure 2.** Continuous inference pipeline. Starting with a mono audio chunk (top left, black solid-line box), we compute its spectrogram via streaming STFT, add noise, and duplicate the channel to form the noisy spectrogram ϕ0(z). The trained model progressively removes the noise with a buffer bank. Finally, streaming ISTFT converts the predicted binaural spectrogram ϕ1(z) into binaural audio. When the next audio chunk appears … view at source ↗

**Figure 3.** The early skip time schedule. The use of an early skip strategy effectively reduces the inference steps and retains the generation performance. the head while sitting on a chair. We split the dataset into training/validation/test subsets with 8.47/0.86/1.33 hours of each subset. The test subset contains two additional speakers, male and female, not seen during training. See the appendix for details on the data coll… view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Performance with respect to the NFE. We evaluate all generative models using the same NFE for a fair comparison. Mono Audio SoundSpaces BinauralGrad SGMSE Ours GT Audio Left Channel Right Channel [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Qualitative comparison between different baselines. We display waveforms of rendered spatial audio. regression-based models underperform the generation-based models. Compared with BinauralGrad, SGMSE exhibits better generation quality in terms of L2 error and Mag erro…

**Figure 6.** Figure 6: Output spectrograms using different inference pipelines. achieves lower L2, Mag, and Phase errors, showing the effectiveness of our conditional flow matching approach. Continuous Inference Pipeline. We compare our continuous inference pipeline and the non-streaming in…

**Figure 7.** Figure 7: Large-scale pretraining strategy. We propose pretraining our model using massive data to improve data efficiency and enhance generalization in downstream tasks. scenarios is costly and labor-intensive. To understand how data quantity affects our model’s performance, w…

**Figure 8.** Figure 8: Capture and evaluation setups. mounted to a VR headset worn by the speaker, with guaranteed phase synchronization. Poses of both the speaker and listener were recorded via an OptiTrack tracking system. The speaker was tracked with IR-reflective markers mounted on the h…

**Figure 9.** Figure 9: The large-scale binaural data capture system with artificial binaural heads. layers and Transformer Encoder layers. Neither the ResNet layers nor the Transformer Encoder layers are causal, which means that Matcha-TTS does not achieve time causality or support streaming…

**Figure 10.** Figure 10: Qualitative comparison between different baselines. We display waveforms of rendered spatial audio. Mono Audio SoundSpaces BinauralGrad SGMSE Ours GT Audio Left Channel Right Channel [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

**Figure 11.** Figure 11: Qualitative comparison between different baselines. We display waveforms of rendered spatial audio. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: Qualitative comparison between different baselines. We display waveforms of rendered spatial audio. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

[1]

A., and Van Waterschoot, T

Antonello, N., De Sena, E., Moonen, M., Naylor, P. A., and Van Waterschoot, T. Room impulse response interpola- tion using a sparse spatio-temporal representation of the sound field.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1929–1941,

work page 1929
[2]

We initialize some variables, including t, ϕt(z), and δ

Algorithm 2Inference Procedure with Midpoint Solver Input:Trained networku t, mono spectrogramx, inference stepsn z∼ N(x, σ2I)//Sample random variable t←0 ϕt(z)←z δ←2/n whilet <1do v′ ←u t(ϕt(z), prx, ptx,x;θ)//Calculate vector field att ϕt′(z)←ϕ t(z) +v ′δ//Calculate flow att+δ v′′ ←u t′(ϕt′(z), prx, ptx,x;θ)//Calculate vector field att+δ v= (v ′ +v ′′)/...

work page 2019
[5]

Jung, C., Lee, S., Kim, J.-H., and Chung, J. S. Flowavse: Efficient audio-visual speech enhancement with condi- tional flow matching.arXiv preprint arXiv:2406.09286,

work page Pith review arXiv
[6]

However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes

Changing the coefficients does not lead to significant changes in the quantitative results. However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes. Specifically, background noise becomes more realistic when the coefficient is increased. These results support the ...

work page 2021
[9]

Av- nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Information Pro- cessing Systems, 36:37472–37490, 2023a

Liang, S., Huang, C., Tian, Y ., Kumar, A., and Xu, C. Av- nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Information Pro- cessing Systems, 36:37472–37490, 2023a. Liang, S., Huang, C., Tian, Y ., Kumar, A., and Xu, C. Neural acoustic context field: Rendering realistic room impulse response with neural fields.ar...

arXiv
[11]

RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction

Liu, P., Dai, D., and Wu, Z. Rfwave: Multi-band rectified flow for audio waveform reconstruction.arXiv preprint arXiv:2403.05010,

work page Pith review arXiv
[12]

Mehta, S., Tu, R., Beskow, J., Sz´ekely, ´E., and Henter, G. E. Matcha-tts: A fast tts architecture with conditional flow matching. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11341–11345. IEEE,

work page 2024
[13]

U-net: Con- volutional networks for biomedical image segmenta- tion

Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234–241. Springer,

work page 2015

Show all 24 references

[18]

Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y ., and Yu, D

URL https:// doi.org/10.7488/ds/2645. Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y ., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation.IEEE/ACM Transactions on Audio, Speech, and Language Processing,

doi
[21]

We evaluate our model with various solvers, include both first-order and second-order solvers to analyze their influence on the generation quality

Impact of different numerical solvers. We evaluate our model with various solvers, include both first-order and second-order solvers to analyze their influence on the generation quality. Solver Type NFE Audio Quality L2↓Mag↓Phase↓ Euler 6 Medium 0.90 0.0066 1.24 Midpoint 6 Hig...

2024
[23]

2.759 1.2780.1280.0300.837 SGMSE (Richter et al.,
[24]

More Qualitative Results We display more rendered waveforms in Figures 10 to

2.256 1.352 0.230 0.033 0.983 BinauralFlow (Ours) 2.806 1.2520.1920.0300.918 K. More Qualitative Results We display more rendered waveforms in Figures 10 to

1920
[2004]

Tx” and the speaker’s trajectory is shown in blue. The listener is denoted as “Rx

12 BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models A. Demo Videos To help understand our work, we have created several demo videos showcasing BinauralFlow’s binaural speech rendering capability. We include the...

2019
[2006]

T., Ben-Hamu, H., Nickel, M., and Le, M

Lipman, Y ., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

arXiv
[2012]

Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499, 12,

Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K., et al. Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499, 12,

arXiv
[2015]

Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

Du, Z., Wang, Y ., Chen, Q., Shi, X., Lv, X., Zhao, T., Gao, Z., Yang, Y ., Gao, C., Wang, H., et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

arXiv
[2017]

T., Hsu, W.-N., Gerkmann, T., Richard, A., and Wu, Y .-C

Welker, S., Le, M., Chen, R. T., Hsu, W.-N., Gerkmann, T., Richard, A., and Wu, Y .-C. Flowdec: A flow-based full- band general audio codec with high perceptual quality. arXiv preprint arXiv:2503.01485,

arXiv
[2018]

Denoising diffusion im- plicit models.arXiv preprint arXiv:2010.02502, 2020a

11 BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models.arXiv preprint arXiv:2010.02502, 2020a. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., ...

2010 arXiv
[2019]

Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv
[2020]

Improving and generalizing flow-based generative models with mini- batch optimal transport.arXiv preprint arXiv:2302.00482,

Tong, A., Malkin, N., Huguet, G., Zhang, Y ., Rector-Brooks, J., Fatras, K., Wolf, G., and Bengio, Y . Improving and generalizing flow-based generative models with mini- batch optimal transport.arXiv preprint arXiv:2302.00482,

arXiv
[2021]

Accelerating high-fidelity waveform generation via adversarial flow matching optimization.arXiv preprint arXiv:2408.08019, 2024a

Lee, S.-H., Choi, H.-Y ., and Lee, S.-W. Accelerating high-fidelity waveform generation via adversarial flow matching optimization.arXiv preprint arXiv:2408.08019, 2024a. Lee, S.-H., Choi, H.-Y ., and Lee, S.-W. Periodwave: Multi- period flow matching for high-fidelity wavefor...

arXiv
[2022]

and Gimpel, K

Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415,

arXiv
[2023]

F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.arXiv preprint arXiv:2410.06885,

Chen, Y ., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., Yu, K., and Chen, X. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.arXiv preprint arXiv:2410.06885,

arXiv
[2024]

and Tiwari, N

Kailas, G. and Tiwari, N. Design for immersive experience: Role of spatial audio in extended reality applications. In Design for Tomorrow—V olume 2: Proceedings of ICoRD 2021, pp. 853–863. Springer,

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A., and Van Waterschoot, T

Antonello, N., De Sena, E., Moonen, M., Naylor, P. A., and Van Waterschoot, T. Room impulse response interpola- tion using a sparse spatio-temporal representation of the sound field.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1929–1941,

work page 1929

[2] [2]

We initialize some variables, including t, ϕt(z), and δ

Algorithm 2Inference Procedure with Midpoint Solver Input:Trained networku t, mono spectrogramx, inference stepsn z∼ N(x, σ2I)//Sample random variable t←0 ϕt(z)←z δ←2/n whilet <1do v′ ←u t(ϕt(z), prx, ptx,x;θ)//Calculate vector field att ϕt′(z)←ϕ t(z) +v ′δ//Calculate flow att+δ v′′ ←u t′(ϕt′(z), prx, ptx,x;θ)//Calculate vector field att+δ v= (v ′ +v ′′)/...

work page 2019

[3] [5]

Jung, C., Lee, S., Kim, J.-H., and Chung, J. S. Flowavse: Efficient audio-visual speech enhancement with condi- tional flow matching.arXiv preprint arXiv:2406.09286,

work page Pith review arXiv

[4] [6]

However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes

Changing the coefficients does not lead to significant changes in the quantitative results. However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes. Specifically, background noise becomes more realistic when the coefficient is increased. These results support the ...

work page 2021

[5] [9]

Av- nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Information Pro- cessing Systems, 36:37472–37490, 2023a

Liang, S., Huang, C., Tian, Y ., Kumar, A., and Xu, C. Av- nerf: Learning neural fields for real-world audio-visual scene synthesis.Advances in Neural Information Pro- cessing Systems, 36:37472–37490, 2023a. Liang, S., Huang, C., Tian, Y ., Kumar, A., and Xu, C. Neural acoustic context field: Rendering realistic room impulse response with neural fields.ar...

arXiv

[6] [11]

RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction

Liu, P., Dai, D., and Wu, Z. Rfwave: Multi-band rectified flow for audio waveform reconstruction.arXiv preprint arXiv:2403.05010,

work page Pith review arXiv

[7] [12]

Mehta, S., Tu, R., Beskow, J., Sz´ekely, ´E., and Henter, G. E. Matcha-tts: A fast tts architecture with conditional flow matching. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11341–11345. IEEE,

work page 2024

[8] [13]

U-net: Con- volutional networks for biomedical image segmenta- tion

Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234–241. Springer,

work page 2015