{"id":"58683ca1-70f7-43d2-bce4-10a00511dfd1","arxiv_id":"2608.12951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"VoxAudio generates audio scenes with intelligible, temporally placed quoted speech by combining chunk-wise causal flow matching with multi-reward fine-tuning and a large transcript-annotated corpus.","lead":"VoxAudio is a text-to-audio model that renders quoted speech intelligibly inside environmental soundscapes and controls when the speech occurs and how long the clip is. It trains one streaming model on speech-plus-sound data with timestamped transcripts, then tunes it with several automated quality scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The same Whisper and PE-A-Frame scorers serve as both RL rewards and headline evaluation metrics, so the intelligibility and timing numbers may partly reflect scorer overfitting rather than human-perceived quality.","rationale":"The reader's weakest_assumption concerns lag-Delta asynchronous states being out-of-distribution during sliding-window inference. That concern is plausible but is partly answered by the paper's own ablations: Table IV sweeps the step lag from 0 to 25 and shows the asynchronous schedule is not catastrophic, with the default lag improving WER over the fully synchronous case, and the chunk-size sweep covers neighboring granularities. The metric-overlap issue is less tested and more directly undermines the two headline capabilities of the paper: lexical intelligibility and temporal placement. Both are scored with the same learned models used as RL rewards, and the authors only report separating the CLAP checkpoints. This is a real soft spot because the largest claimed gains (WER 0.038 vs 0.264) are exactly on the dimension optimized by the Whisper reward. The concern is mitigated by the held-out SEED-TTS-EVAL result and by the subjective MOS-C ratings, so it does not justify rejecting the paper or changing the reader's CONDITIONAL verdict; it does justify adding an external-scorer condition before the headline numbers are treated as established.","tokens_in":21293,"tokens_out":15254,"duration_ms":171687,"concrete_test":"Take the 50 VoxBench-10s clips used for the MOS evaluation, transcribe the quoted speech with an independent ASR that was not used during training or reward computation (e.g., Gemini 2.5 ASR or another Whisper-free system), and compare the resulting WER to the reported 0.038. If the independent WER exceeds roughly 0.10, the same-scorer reward/eval overlap is a confirmed validity threat to the headline intelligibility claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C defines the multi-reward NFT objective with rewards that include Whisper WER (linguistic accuracy) and PE-A-Frame TG-IoU (temporal grounding). Section V-B then evaluates exactly these quantities with the same underlying scorers: Whisper-large-v3 for WER and PE-A-Frame at threshold 0.3 for TG-IoU. The paper explicitly states only that the CLAP checkpoint used for evaluation differs from the one used during RL; no such separation is reported for WER or TG-IoU. The two strongest empirical claims—intelligible quoted speech (WER 0.038 on VoxBench-10s) and temporal placement (TG-IoU 0.654)—are therefore measured by the same learned functions optimized during post-training. This makes the headline numbers vulnerable to scorer-specific reward hacking: the model may learn Whisper's and PE-A-Frame's inductive biases rather than human-perceived intelligibility and timing. The 'w/o Linguistic Reward' and 'w/o Temporal Reward' ablations confirm the rewards move these metrics, but they do not establish that the movement reflects the underlying human property. The SEED-TTS-EVAL result and the MOS-C scores mitigate but do not eliminate the concern, since the SEED WER still relies on Whisper and MOS-C is a holistic rating rather than a forced transcription or human interval annotation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VoxAudio is a causal autoregressive flow-matching model for generating audio in which intelligible, quoted speech is embedded in an environmental soundscape. The paper contributes (i) a chunk-wise causal factorization with independent per-chunk noise levels and randomized chunk boundaries during pretraining, enabling sliding-window streaming inference with KV caching and variable target durations; (ii) a multi-reward Negative-aware FineTuning (NFT) stage that jointly optimizes semantic, linguistic (Whisper WER), aesthetic, and temporal-grounding (PE-A-Frame TG-IoU) rewards; and (iii) two new resources, VoxCorpus and VoxBench. Experiments compare VoxAudio with general T2A, unified audio, and TTS baselines on VoxBench, MECAT-en, AudioCaps, and SEED-TTS-EVAL, reporting large WER improvements over Dasheng-AudioGen (0.038 vs 0.264 on VoxBench-10s), competitive SEED-TTS-EVAL WER (1.61%), and 0.32 RTF streaming.","tokens_in":21567,"tokens_out":13466,"duration_ms":136259,"significance":"If the reported results hold, VoxAudio would establish a plausible path to end-to-end vocalized audio synthesis with lexical and temporal control, a genuinely under-served task. The strengths are substantial: the chunk-agnostic causal AR-FM formulation is conceptually clean and addresses a real limitation of fixed-length non-causal generators; the two-stage training and the ablation set are internally consistent; the use of independent per-chunk noise levels followed by reward-based alignment is a reasonable design; and the efficiency measurements (RTF, latency, first-chunk latency) are reported carefully. The main reservations are evaluative and presentational: the headline WER and TG-IoU metrics overlap with the training rewards, the streaming pseudocode does not transparently match the training objective, and the dataset and benchmark are not yet released, which limits external verification. These issues are fixable with additional experiments and clarification rather than requiring a change in the core approach.","major_comments":[{"comment":"Section III-C defines reward terms using Whisper WER (linguistic accuracy) and PE-A-Frame TG-IoU (temporal grounding), and Section V-B evaluates WER with Whisper-large-v3 and TG-IoU with PE-A-Frame at threshold 0.3. The only stated separation between RL and evaluation is for the CLAP checkpoint; no such separation is reported for Whisper or PE-A-Frame. Since the headline VoxBench-10s claims (WER 0.038, TG-IoU 0.654 in Table I) are measured with the same learned functions that were optimized during NFT, the improvement over Dasheng-AudioGen may partly reflect scorer overfitting rather than human-perceived intelligibility and timing. SEED-TTS-EVAL and MOS-C mitigate but do not eliminate this concern. Please add an independent evaluation: a different ASR not used in training, human transcription WER, and human interval annotations or a temporal detector not used as a reward.","section":"III-C, V-B, Table I"},{"comment":"The training objective in Eq. (2) conditions the prediction for chunk i on all preceding chunks at their own noise levels. In Algorithm 1, the KV cache K is updated only when a chunk is fully decoded (lines 7-9), so active chunks that are still noisy are not available as context. For lag Δ>0, a newly activated chunk must attend to an earlier chunk that is still being denoised (e.g., with Δ=5 and N=25, chunk 2 becomes active when chunk 1 is at t=5δ). As written, the per-chunk loop cannot provide that context, making the inference procedure inconsistent with Eq. (2). Please clarify whether the implementation performs a joint forward pass over the active window, stores current noisy latents in K, or otherwise restores the training-time conditioning; this is load-bearing for the streaming claims in Section III-E and Table IV.","section":"III-A, III-E, Algorithm 1"},{"comment":"Section III-C first names LAION-CLAP as the semantic reward and then says \"we adopt PEAV as the semantic metric in our experiments\"; Section V-A lists the reward weights as (w_CLAP, w_Aes, w_WER, w_TG), so it is unclear which model is actually the semantic reward. This ambiguity matters because Table I shows the 'w/o Semantic Reward' row improving TG-IoU from 0.654 to 0.825 and nearly matching the full model's WER (0.042 vs 0.038), while only PEAV and CLAP drop. That pattern is difficult to reconcile with Section V-D's claim that the semantic reward is an essential anchor that prevents collapse. Please state the reward definition explicitly and explain the ablation behavior.","section":"III-C, V-A, Table I"},{"comment":"Section III-A argues that independent per-chunk noise sampling places the asynchronous states visited by streaming inference inside the training distribution. The specific 'under-represented' phrasing is not literally correct if per-chunk noise levels are iid with full support, since every tuple then has the same density. The load-bearing issue is instead that the iid training distribution and the lag-Δ schedule of Eq. (5)/Algorithm 1 have different joint structure, and the model may underweight the correlated trajectories used at inference. Please either train with a mixture that contains Eq. (5) schedules or add an experiment that varies the training noise-level distribution while holding the inference schedule fixed; the current Table IV varies only inference-time parameters and does not isolate this factor.","section":"III-A, V-D"}],"minor_comments":[{"comment":"In Section II-C, \"Flow-GRPO [36] applys\" should read \"applies\".","section":"II-C"},{"comment":"Table IV is difficult to parse: the \"Default (C=16, Δ=5)\" row is not clearly part of any labeled ablation group, and its WER of 0.076 differs from Table I's VoxAudio WER of 0.038 without explanation. Please clarify the evaluation subset and the relationship between the two tables.","section":"Table IV"},{"comment":"Section V-B says MOS tests use 5 listeners, while Appendix C-B says a pool of 16 independent evaluators ensures each clip receives at least five ratings; please reconcile these descriptions.","section":"V-B, Appendix C-B"},{"comment":"The claim that MOS-C is \"significantly higher\" for VoxAudio is not supported by the reported confidence intervals (4.51±0.26 vs 4.16±0.34 overlap); please rephrase or provide a formal significance test.","section":"V-C, Table I"},{"comment":"In Section V-D, \"Dasheng-AudioGe\" is missing the final 'n'; please correct the typo.","section":"V-D"},{"comment":"The abstract and main text say code and demos are available, but Appendix A says VoxCorpus and VoxBench \"will be released\" under CC BY-NC-SA 4.0; please state the actual availability of the data and benchmark for reproducibility.","section":"IV-A, Appendix A"},{"comment":"In Section VI, \"mono-lingual\" should be \"monolingual\".","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is well within the journal's scope and the method is promising, but the evaluative circularity and the Algorithm 1 mismatch need to be resolved before publication. I would also encourage the authors to release VoxBench and at least a portion of VoxCorpus, since the in-house benchmark and non-released data make independent verification difficult. I do not see grounds for rejection, but the revision must include the additional independent evaluations and the pseudocode clarification described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. The central result holds up: VoxAudio actually does what prior T2A systems could not, producing intelligible quoted speech inside a continuous soundscape with controllable timing. The WER gap versus Dasheng-AudioGen (0.038 vs 0.264 on VoxBench-10s) is large enough to be credible as a qualitative improvement, and the SEED-TTS-EVAL result (1.61% WER without reference audio) gives independent support. The architecture is a straightforward adaptation of diffusion-forcing-style asynchronous denoising, but the randomized chunk-boundary pretraining is a clean and genuinely new idea that closes the train-test gap for streaming. The data work is also real: VoxCorpus with verbatim quoted, timestamped captions addresses a genuine missing-supervision problem, and VoxBench's temporal-grounding metric is a useful addition even if it is in-house.\n\nThe soft spots are the ones the stress-test note flags, and they are not manufactured. Whisper WER and PE-A-Frame TG-IoU serve as both RL rewards and headline evaluation metrics, with no reported separation of scorer checkpoints for those two dimensions. The MOS-C scores and the SEED-TTS-EVAL result mitigate but do not eliminate the concern: the model could be partly optimizing Whisper's and PE-A-Frame's inductive biases rather than human-perceived intelligibility and timing. The VoxBench numbers should be re-read with that caveat until independent human transcription or a different ASR/grounding model is used for evaluation. A second, lesser concern: the motivating decoupled-pipeline baseline is described but never measured, so the paper's framing overstates the comparison. The unreleased corpora are a practical limitation for reproduction, though the stated release plan is at least explicit.\n\nProportionately: the flow-matching math is coherent, the ablations are internally consistent and trace each reward to its expected metric, and the citation pattern is honest. The paper is not over-claiming in its conclusions; the limitations section is candid. The main fix is evaluator separation, ideally with human transcription and human interval annotation on a random subset, plus reporting the decoupled baseline. That is a revision-level issue, not a desk-reject issue. I would send this to referees; it deserves careful review. I would not cite the headline WER numbers as independent evidence until the scorer-overlap point is addressed, but I would cite the chunk-boundary pretraining and the corpus design.","headline":"A serious, well-executed systems paper that makes a real advance in unified vocalized audio synthesis, with a genuine but partial evaluator-overlap concern that should be addressed before publication.","tokens_in":22140,"tokens_out":633,"would_cite":true,"duration_ms":8128,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VoxAudio generates intelligible quoted speech inside environmental soundscapes in real time.","keywords":["vocalized audio synthesis","text-to-audio","speech-in-soundscape","autoregressive flow matching","streaming generation","multi-reward fine-tuning","temporal grounding","audio captioning dataset"],"falsifier":"Run the same checkpoint under two noise-level schedules: the training-time independent per-chunk sampling and the exact lag-$\\Delta$ sliding-window schedule. If the lag-$\\Delta$ stream yields substantially worse word error rate or temporal grounding than the independently sampled schedule on the same captions, the train-test gap the paper claims to close is not closed. A sharper test is to compare the default randomized-boundary checkpoint against a fixed-boundary checkpoint at mismatched inference chunk sizes: the mechanism predicts a sharp degradation for the fixed-boundary model and none for the randomized one.","tokens_in":21060,"feed_emoji":"🎙️","tokens_out":6248,"duration_ms":58699,"temperature":0.7,"pith_summary":"VoxAudio aims to make quoted speech intelligible inside a generated environmental soundscape, so a caption like \"a boy exclaims, 'What a lovely cat!' over jingling bells\" yields audio in which those exact words are audible at the right moment, not vocal murmur. The paper argues that this becomes possible by treating speech-in-scene as one causal generative process: a chunk-wise autoregressive flow-matching model that streams audio chunk by chunk while later chunks are still denoising. It claims that training with randomized chunk boundaries makes the streaming granularity a free choice at inference, and that a multi-reward post-training stage, combining semantic fidelity, word accuracy, aesthetics, and temporal grounding, is what turns a sound-effects generator into a speech-capable one. If true, unified text-to-audio systems could replace the current two-model pipeline of TTS plus post-hoc mixing, with direct control over when speech starts, how long it lasts, and how loud it sits in the scene.","feed_headline":"Quoted speech inside soundscapes: WER drops to 0.038","feed_subtitle":"A streaming audio model places intelligible words at the right moment in ambient scenes, beating unified baselines on word accuracy.","key_machinery":"The load-bearing object is the chunk-wise causal factorization of the flow-matching objective: the latent audio sequence is split into chunks, each with an independent noise level $t_i$, and the vector-field prediction for chunk $i$ is conditioned on all preceding chunks at their own noise levels ($\\{(z^{(j)}_{t_j})\\}_{j<i}$ in Eq. 2). This makes the model autoregressive across chunks and continuous within each chunk. Three mechanisms hang off it: randomized chunk boundaries during pretraining, where every latent frame opens a new chunk with probability $p_b$, so the inference chunk size is arbitrary; chunk-wise causal attention that allows full visibility inside a chunk but strict causality across chunks; and a lag-$\\Delta$ sliding-window schedule (Eq. 5, Algorithm 1) that keeps a fixed denoising step difference between adjacent chunks so streaming emits audio while later chunks refine. The multi-reward Negative-aware FineTuning (NFT) loss (Eq. 4) replays exactly these asynchronous noise snapshots and steers the vector field toward implicit positive and negative targets weighted by group-normalized rewards.","core_discovery":"On its own terms, the paper's central discovery is that vocalized audio generation can be cast as causal autoregressive flow matching over latent chunks, where each chunk carries its own noise level, so the model learns the joint distribution of speech and scene as a single streaming process rather than as two separately generated tracks. The corollary claims are: independent per-chunk noise levels during training put the asynchronous lag-\\$\\Delta$ schedule used by sliding-window inference inside the training distribution, closing the train-test gap; randomizing chunk boundaries during pretraining lets the inference chunk size be chosen freely without retraining; and optimizing a weighted combination of semantic, linguistic, aesthetic, and temporal rewards via negative-aware fine-tuning is what makes the embedded speech intelligible and correctly placed. The paper reports word error rates of 0.038 on its 10-second vocalized-audio benchmark versus 0.264 for the strongest prior unified baseline, 1.61% on a standard English TTS benchmark, and streaming at a 0.32 real-time factor.","pith_inferences":["Beyond the paper's claims, the same chunk-agnostic causal flow-matching design suggests a natural upgrade path for live dubbing and podcast tools: a first-chunk latency near 1.4 seconds already approaches interactive use, and the lag-$\\Delta$ parameter is an explicit knob to trade response time against intelligibility, which I predict will need per-application tuning.","The multi-reward objective is generic enough that speaker-timbre and prosody rewards, named as future work in the paper's limitation section, should be a direct addition rather than a redesign, potentially giving the same architecture fine-grained voice control.","A testable consequence of the paper's mechanism claim: a model trained with fixed chunk boundaries should show a sharp word-error-rate penalty when evaluated at a mismatched chunk size, whereas the randomized-boundary model should stay flat; the paper's ablation hints at this, but a systematic sweep would confirm it.","Because word error rate is computed through an external recognizer, the linguistic reward could in principle be gamed by producing easy-to-recognize but unnatural speech; the paper guards against this with the aesthetic and semantic rewards, but the interaction between those rewards deserves direct study."],"forward_implications":["A single text-to-audio model can now render quoted speech verbatim inside a scene, cutting word error rate from 0.264 to 0.038 on 10-second vocalized audio relative to the prior unified baseline.","Streaming generation with key-value caching makes first audio available in about 1.4 seconds and sustains a 0.32 real-time factor on one GPU, so interactive and variable-duration use becomes practical.","Because chunk boundaries are randomized at training, the operator can pick any inference chunk size or step lag; the ablations show small chunks and small lag favor latency, while larger values favor linguistic accuracy, with both metrics degrading at the extremes.","On pure speech benchmarks the same model reaches 1.61% word error rate, competitive with dedicated TTS systems even without reference-speaker prompts, so vocalized-audio training transfers to clean speech synthesis.","The new VoxCorpus and VoxBench, with quoted, time-stamped transcripts and a temporal-grounding metric, give the field a way to measure whether speech content and placement are actually correct."],"supporting_citations":[{"why":"Supplies the strongest prior unified text-to-audio baseline the paper compares against and outperforms on word error rate and temporal placement.","marker":"[8]"},{"why":"Provides the negative-aware fine-tuning (NFT) paradigm that the paper adapts to causal flow matching for multi-reward preference alignment.","marker":"[9]"},{"why":"Supplies the diffusion-transformer backbone architecture the paper modifies with causal convolutions and chunk-wise causal attention.","marker":"[15]"},{"why":"Establishes the principle of training with per-frame independent noise levels for causal rollout, which the paper adapts to general audio.","marker":"[28]"},{"why":"Introduces asynchronous timestep schedulers for autoregressive diffusion, the direct predecessor of the lag-$\\Delta$ sliding-window schedule.","marker":"[29]"},{"why":"Supplies the frozen text encoder whose multi-depth features carry both the lexical surface of quoted speech and the scene semantics.","marker":"[40]"},{"why":"Provides the external speech recognizer that computes word error rate for both the linguistic reward and the evaluation metrics.","marker":"[44]"},{"why":"Supplies the frame-level speech-span detector used to compute the temporal-grounding IoU reward and metric.","marker":"[46]"},{"why":"Provides the standard general-audio captioning test set and RL prompt source, whose lack of verbatim transcripts motivates the new corpus.","marker":"[47]"}],"fun_headline_variants":["Streaming flow matching embeds speech in soundscapes, WER 0.038","VoxAudio: intelligible speech inside ambient audio, WER 0.038","Causal autoregressive flow: quoted speech placed in scenes, WER 0.038","Multi-reward fine-tuning: streamed speech at 0.038 WER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training with independently sampled per-chunk noise levels puts the lag-$\\Delta$ asynchronous schedule used at streaming inference inside the training distribution, so streamed chunks are not out-of-distribution for the model.","fun_headline_variants_meta":{"raw":{"variants":["Streaming flow matching embeds speech in soundscapes, WER 0.038","VoxAudio: intelligible speech inside ambient audio, WER 0.038","Causal autoregressive flow: quoted speech placed in scenes, WER 0.038","Multi-reward fine-tuning: streamed speech at 0.038 WER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3109,"prompt_tokens":1023,"completion_tokens":2086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":1995}},"tokens_in":639,"tokens_out":2086,"duration_ms":15198,"temperature":1.0,"reasoning_tokens":1995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:51:16.277482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same checkpoint under two noise-level schedules: the training-time independent per-chunk sampling and the exact lag-$\\Delta$ sliding-window schedule. If the lag-$\\Delta$ stream yields substantially worse word error rate or temporal grounding than the independently sampled schedule on the same captions, the train-test gap the paper claims to close is not closed. A sharper test is to compare the default randomized-boundary checkpoint against a fixed-boundary checkpoint at mismatched inference chunk sizes: the mechanism predicts a sharp degradation for the fixed-boundary model and none for the randomized one.","supporting_citations":[{"cited_title":"Diffusion forcing: Next-token prediction meets full- sequence diffusion,","cited_arxiv_id":null,"evidence_quote":"Establishes the principle of training with per-frame independent noise levels for causal rollout, which the paper adapts to general audio."},{"cited_title":"Ar-diffusion: Asynchronous video generation with auto-regressive diffusion,","cited_arxiv_id":null,"evidence_quote":"Introduces asynchronous timestep schedulers for autoregressive diffusion, the direct predecessor of the lag-$\\Delta$ sliding-window schedule."}],"review_version":1}