Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that autoregressive speech synthesis is more robust and faster when a lightweight policy dynamically chooses how many speech tokens to predict at each decoding step, rather than always predicting the next single frame.

desk verdict Worth engaging: the dynamic chunk policy is a real contribution, but the headline 72% improvement is inflated by a weak baseline and the intelligibility claim rests only on ASR WER. read the letter →

arxiv 2506.22023 v1 pith:K5MDRBOU submitted 2025-06-27 cs.SD cs.CLeess.AS

Bohan Li , Zhihan Li , Haoran Wang , Hanglei Zhang , Yiwei Guo , Hankun Wang , Xie Chen , Kai Yu This is my paper
classification cs.SDcs.CLeess.AS
keywords dynamicchunk-wisepredictionautoregressivespeechsynthesismulti-tokenreinforcementlearningpolicyintelligibilityinferenceaccelerationdiscretetokenspreferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive text-to-speech systems usually generate speech one token frame at a time, which makes long utterances slow and unstable. This paper argues that frame-by-frame prediction is not necessary: predicting several speech tokens at once, with a small policy network that decides how many to predict at each step, yields both more intelligible and much faster synthesis. On the test set, the dynamic chunk-wise model reduces word error rate from 9.99% to 2.77%, a 72.27% relative improvement, while running 2.61x faster; on a second token type it improves from 2.31% to 1.84% with a 2.89x speedup. The policy is trained on only 980 utterances, so the extra machinery is cheap. If the claim holds, dynamically scheduled chunk-wise decoding is a viable path toward robust, real-time autoregressive speech synthesis.

What carries the argument

The machinery has three parts. First, the chunk-wise autoregressive decoder: a decoder-only transformer with one base prediction head and several additional heads, each trained with equal cross-entropy weight to predict the next several speech tokens, so the model learns a chunk-to-frame attention pattern instead of a frame-to-frame one. Second, the decoding action: at each step a chunk size is sampled and the model emits that many tokens in one step, skipping several autoregressive steps. Third, the DCPO policy: a causal transformer layer and linear head that reads the history of hidden states and outputs a distribution over chunk sizes, trained with an adapted group-relative preference optimization objective whose reward is the word-error-rate gap between generated and ground-truth speech; half of each sample group follows a hand-profiled fixed chunk range, and chunk sizes outside that range receive a negative process reward.

What would settle it

Score the generated and ground-truth speech from the test set with a second, independently trained speech recognizer and with human transcribers; if DCAR's word-error-rate advantage over frame-level autoregression disappears or reverses on either of those, the claimed 72.27% intelligibility improvement is an artifact of the chosen reward recognizer.

Watch

Extended reading notes

Core claim

The paper's central claim is that in autoregressive speech synthesis, the model should predict a chunk of future speech tokens rather than the next single frame, and that the chunk size should vary from step to step. A decoder trained with multiple prediction heads already lowers word error rate under fixed chunk sizes compared with frame-level prediction, and the paper finds that the most confident prediction for a given position often comes from a later prediction head rather than the base head, so no fixed chunking schedule is optimal. A lightweight policy trained with an on-policy preference optimization objective therefore chooses the chunk length dynamically, and this dynamic schedule gives the best robustness while preserving acoustic quality and speaker similarity and multiplying inference speed. The intended reading is that dynamic chunk-wise prediction is a general and scalable improvement over the standard autoregressive paradigm for speech.

Load-bearing premise

The result rests on treating the word-error-rate gap between generated and ground-truth speech, as measured by one automatic speech recognizer, as a faithful proxy for real intelligibility, and on assuming that improving this score on 980 training utterances transfers to human listening; if the model games that recognizer, the reported intelligibility gain would be illusory.

Editorial extensions

If this is right

  • Autoregressive text-to-speech no longer needs to predict one frame at a time; dynamic chunk-wise decoding reduces word error rate while cutting latency, making long-form synthesis more practical for real-time use.
  • The scheduling policy is cheap to obtain: training on only 980 utterances improves over fixed-chunk decoding, suggesting the chunk-size decision is learnable from very little data.
  • Fixed chunk sizes leave accuracy on the table: the best chunk length varies by decoding step, and the adaptive policy outperforms every fixed length in the tested range.
  • The hand-profiled action guidance range is a required ingredient; a completely free policy degrades robustness, so the allowed chunk-size range is a deliberate design choice rather than an optional detail.
  • Because the approach works on two different semantic token types and on a much larger training set, the principle appears portable across autoregressive speech decoders rather than tied to one token representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same chunk-scheduling-as-reinforcement-learning recipe could apply to other locally coherent autoregressive sequence tasks, such as music or non-speech audio generation, where neighboring tokens are strongly correlated and multi-token prediction is easier than in text.
  • The tiny 980-utterance training set suggests that good chunk-size preferences are mostly a global property of autoregressive speech decoding rather than a per-speaker or per-text phenomenon; if so, the learned policy may transfer to other TTS backbones with little or no retraining.
  • The reward's dependence on a single automatic speech recognizer suggests a direct test of whether the intelligibility gain is genuine: replacing the reward with a perceptual score or an ensemble of recognizers would reveal whether the model is improving speech clarity or merely adapting to one recognizer's quirks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DCAR, a dynamic chunk-wise autoregressive speech synthesis framework. It first trains a CAR TTS model with multiple prediction heads and then trains a lightweight policy with a GRPO-style objective (DCPO) to choose the chunk size at each decoding step, using a WER-based reward computed by a NeMo ASR model plus a 'chase-then-exceed' guidance range. Experiments on HuBERT and S3-Tokenizer tokens report that DCAR achieves lower Whisper WER than frame-level AR (FAR) and fixed-chunk CAR baselines, maintains similar UTMOS and SECS, and provides 2.18--2.89x speedups. Additional experiments on CosyVoice and LibriHeavy support generality and scalability.

Significance. DCAR is an interesting and practical contribution to autoregressive TTS: it combines multi-token prediction with a small RL-trained scheduler, and the paper provides a useful empirical study of chunk-wise prediction (attention visualization, head-preference analysis, ablations of the guidance range). The authors are careful to use different ASR models for training reward (NeMo) and evaluation (Whisper), which mitigates but does not eliminate overfitting concerns, and they evaluate on two tokenizers, a public TTS backbone, and a 50k-hour scaling setting. If the central WER claim holds under closer scrutiny, the dynamic chunk policy is a valuable addition to the AR TTS toolkit. However, the headline improvement is largely inherited from the CAR architecture rather than from the dynamic policy, and the absence of human listening tests makes the 'intelligibility' claim provisional.

major comments (4)
  1. [Section 5.2 / Table 1] The abstract's 'up to 72.27% intelligibility improvement' is computed relative to the FAR baseline (WER 9.99% to 2.77%), but the best fixed-chunk CAR baseline already reaches WER 2.99% on the same HuBERT setup. The additional improvement of DCAR over the best CAR configuration is only from 2.99% to 2.77%, i.e., roughly 7.4% relative. The paper should clearly attribute the dominant gain to the chunk-wise CAR architecture and state the marginal contribution of DCPO; otherwise the headline overstates what is new.
  2. [Eq. (5), Section 5.2, Appendix E] The core intelligibility claim rests entirely on ASR WER, while the DCPO reward in Eq. (5) is computed from a NeMo ASR model on generated versus ground-truth speech. Although evaluation uses Whisper, both are neural ASRs and may share systematic biases. Without human listening tests or at least an analysis of acoustic/prosodic artifacts (e.g., speech rate, pause insertion, hyper-articulation), the paper cannot rule out that the policy learns to satisfy ASR systems rather than improve human-perceived intelligibility. UTMOS and SECS are comparable across methods, so they do not independently confirm improved intelligibility. The authors should add a human listening evaluation or a targeted diagnostic study, and should soften the abstract's 'intelligibility improvement' claim to 'ASR-WER reduction' until such evidence is available.
  3. [Eq. (4) and Algorithm 1 (lines 8--10)] The DCPO objective is written as an expectation over action sequences sampled from π_θold, but the 'chase-then-exceed' strategy inserts fixed-CAR actions by 'traveling the guidance range C' into the group (Algorithm 1, line 8). These actions are not sampled from π_θold, so the importance-weighting term π_θ/π_θold in Eq. (4) is not a valid unbiased estimator for those group members. The paper needs to either derive the appropriate off-policy correction or clearly state that these expert actions are used in a reward-weighted manner and explain why the GRPO-style update remains valid. This is load-bearing because Table 2 shows the chase-then-exceed component contributes one of the main ablative gains.
  4. [Section 5.5 / Table 4 / Appendix B.1] The action guidance range C is selected by WER profiling with the NeMo ASR model (Appendix B.1), and Table 4 shows that performance is sensitive to C: the 'totally free' policy degrades to WER 3.30%, worse than fixed CAR, while the chosen ranges [2,3] and [2,3,4] perform best. The paper does not state whether the UniCATS test-B set was used for this profiling. If so, the reported numbers are optimistically biased. The authors should specify a held-out development split for selecting C, and report the selection procedure explicitly, including how the 'WER top3' rows for CAR were chosen.
minor comments (6)
  1. [Abstract and Section 1] The phrasing '72.27% intelligibility improvement' should be replaced with a more precise wording such as 'relative WER reduction on the tested set', given the metric and baseline caveats.
  2. [Section 5.5] The sentence 'Variation of policy action guidance range Table 3 demonstrates...' should reference Table 4, not Table 3.
  3. [Appendix D] The metric is defined as SECS in Table 1 but 'SCES' in the appendix; make the acronym consistent.
  4. [Section 3.1 / Appendix C.1] There are typos such as 'muti-token prediction' and 'critiron'; these should be corrected throughout.
  5. [Table 1 and Table 3] Please report confidence intervals or significance tests for the WER differences; several of the key comparisons (e.g., CAR chunk 3 WER 2.99 vs. DCAR [2,3] WER 2.82 vs. DCAR [2,3,4] WER 2.77) are small and may lie within statistical noise.
  6. [Section 4.2] The definition of the state function s(ai,<t,φ) is informal; a precise description of what hidden states are fed to the policy, and how the mask on inner-chunk positions is applied, would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DCAR's WER gains are measured on a held-out test set with a different ASR than the training reward; self-citations are non-load-bearing.

full rationale

The paper's central comparison is empirical and self-contained rather than definitional. DCAR is trained with DCPO whose reward (Eq. 5) is computed from NeMo-ASR WER on generated vs. ground-truth speech, but the reported test intelligibility is measured with Whisper-Large-v3 on a held-out UniCATS testset-B of 500 utterances from unseen speakers (Sec. 5.1). The test number 2.77% WER is therefore not the training reward by construction; it is a fresh measurement with a different recognizer. The action-guidance range C is a hyperparameter profiled from fixed-size CAR WER (Appendix B.1) and ablated in Table 4; selecting a hyperparameter by validation performance is not a fitted input renamed as a prediction. The 'WER top3' CAR rows are explicitly selected, and DCAR is compared against those selected CAR numbers, so the comparison is conservative rather than forced. Self-citations [3] and [6] are to a tokenization review and to the authors' VADUSA baseline; neither is invoked as a uniqueness theorem or as an unverified premise that carries the main claim. The one passage that could look circular is Sec. 3.2.3's statement that 'feedback derived directly from audio metrics is unconstrained by the ground-truth data,' which is contradicted by the WER_gt term in Eq. 5; this is an overstatement about the training signal, not a case of the prediction reducing to its input. The absence of human listening tests is a validity risk for the 'intelligibility' wording, but it is not an equation-level circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on ASR WER as a reward proxy, on the transferability of hand-profiled guidance ranges, and on several hand-set hyperparameters. No new physical entities are introduced.

free parameters (5)
  • Action guidance range C = [2,3] and [2,3,4] in main experiments
    Selected by profiling fixed-size CAR chunk sizes with NeMo WER (Appendix B.1); the range strongly affects the speed/robustness trade-off (Table 4).
  • Number of additional prediction heads = 6
    Hand-chosen in Section 5.1; all baselines use the same setting.
  • Reward coefficient lambda = 0.1
    Set by hand in Eq. 5 to keep negative reward magnitude an order of magnitude below WER reward.
  • Reward floor epsilon = -10
    Set in Eq. 5 to avoid reward explosion.
  • KL weight beta schedule = 0.1 * (epoch - 1)
    Warm-up schedule in Algorithm 1, tuned for stable RL training.
assumptions (5)
  • domain assumption ASR WER is a valid proxy for intelligibility
    DCPO reward (Eq. 5) and final evaluation both rely on ASR WER; no human listening tests are reported.
  • domain assumption The profiled guidance range C transfers from the 980-utterance training subset to the test set
    The policy is constrained to C, and a totally free policy degrades to WER 3.30% (Table 4), so the choice of C is load-bearing.
  • domain assumption Speech tokens possess local continuity that makes multi-token prediction easier than in text
    Section 3.2.1 makes this argument to justify CAR training; if false, the additional heads would not learn useful predictions.
  • domain assumption The 980-utterance training set from 20 speakers is representative for policy learning
    The lightweight policy sees only this tiny set; generalization to 37 unseen test speakers is assumed.
  • standard math GRPO's clipped surrogate objective remains valid when adapted to chunk-size actions
    DCPO (Eq. 4) adapts GRPO (Eq. 2) without new convergence guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy." pith.science (2026). https://pith.science/paper/K5MDRBOU

@misc{pith2026250622023,
  author       = {Pith},
  title        = {Pith review of: Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5MDRBOU}},
  note         = {Machine review of arXiv:2506.22023}
}
read the original abstract

Recently, autoregressive (AR) language models have emerged as a dominant approach in speech synthesis, offering expressive generation and scalable training. However, conventional AR speech synthesis models relying on the next-token prediction paradigm often encounter significant challenges when handling long speech sequences. These models often struggle to construct stable frame-to-frame attention, leading to increased latency and degraded synthesis quality, thereby limiting their feasibility for real-time applications. To address these limitations, we introduce a novel dynamic chunk-wise autoregressive synthesis framework, termed DCAR, designed to enhance both efficiency and intelligibility robustness in AR speech generation. DCAR introduces a chunk-to-frame attention mechanism through training with multi-token prediction, enabling dynamic chunk prediction in variable speech contexts using a lightweight module trained on-policy. DCAR dynamically adjusts the token prediction span, significantly reducing the sequence length dependency while obtaining high synthesis quality. Comprehensive empirical evaluations demonstrate that DCAR substantially outperforms traditional next-token prediction models, achieving up to 72.27% intelligibility improvement and 2.61x inference speedup simultaneously on the test set. Furthermore, we conduct comprehensive analysis to support it as a versatile foundation for next-generation speech synthesis systems.

Figures

Figures reproduced from arXiv: 2506.22023 by the authors.

Figure 1
Figure 1. Overview of FAR, CAR, and proposed DCAR. Prediction 1 Prediction 2 Fusion 1 Fusion 2 ... Can be different! [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Attention visualization. The first 119 tokens are for text, followed by 535 audio tokens. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Prediction loss ranking across different heads (teacher forcing mode). Brighter colors [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: WER performance with different frame-rate and strategies. Method WER(%)↓ Speedup↑ VADUSA–τ=2 3.86 3.03× VADUSA–τ=3 4.17 3.12× DCAR–[2,3] 2.82 2.39× DCAR–[2,3,4] 2.77 2.61× [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 8
Figure 8. Figure 8: Synthesis performance of HuBERT token on LibriHeavy. E.3 Implementation on CosyVoice To validate the generality of our approach, we implement DCAR on the open-source TTS framework CosyVoice. Specifically, we augment the base CosyVoice model with 6 additional heads, tra…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

  1. [1]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223 , vol. 1, no. 2, 2023

    arXiv 2023

  2. [2]

    Codec-SUPERB: An In-Depth Analysis of Sound Codec Models,

    H. Wu, H.-L. Chung, Y .-C. Linet al., “Codec-SUPERB: An In-Depth Analysis of Sound Codec Models,” arXiv preprint arXiv:2402.13071, 2024

    arXiv 2024

  3. [3]

    Recent advances in discrete speech tokens: A review,

    Y . Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, “Recent advances in discrete speech tokens: A review,”arXiv preprint arXiv:2502.06490, 2025

    arXiv 2025

  4. [4]

    On the landscape of spoken language models: A comprehensive survey,

    S. Arora, K.-W. Chang, C.-M. Chien, Y . Peng, H. Wu, Y . Adi, E. Dupoux, H.-Y . Lee, K. Livescu, and S. Watanabe, “On the landscape of spoken language models: A comprehensive survey,” arXiv preprint arXiv:2504.08528, 2025

    arXiv 2025

  5. [5]

    Neural Discrete Representation Learning,

    A. Van Den Oord, O. Vinyalset al., “Neural Discrete Representation Learning,” Proc. NeurIPS, vol. 30, 2017

    work page 2017

  6. [6]

    Fast and high-quality auto-regressive speech synthesis via speculative decoding,

    B. Li, H. Wang, S. Zhang, Y . Guo, and K. Yu, “Fast and high-quality auto-regressive speech synthesis via speculative decoding,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2025, pp. 1–5

    work page 2025

  7. [7]

    Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding

    T. D. Nguyen, J.-H. Kim, J. Choi, S. Choi, J. Park, Y . Lee, and J. S. Chung, “Accelerating codec-based speech synthesis with multi-token prediction and speculative decoding,” 2024. [Online]. Available: https://arxiv.org/abs/2410.13839

    work page Pith review arXiv 2024

  8. [8]

    V ocalnet: Speech llm with multi-token prediction for faster and high-quality generation,

    Y . Wang, H. Liu, Z. Cheng, R. Wu, Q. Gu, Y . Wang, and Y . Wang, “V ocalnet: Speech llm with multi-token prediction for faster and high-quality generation,” 2025. [Online]. Available: https://arxiv.org/abs/2504.04060

    arXiv 2025

Show all 50 references
  1. [9]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu et al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  2. [10]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training,

    T. Chu, Y . Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V . Le, S. Levine, and Y . Ma, “Sft memorizes, rl generalizes: A comparative study of foundation model post-training,” 2025. [Online]. Available: https://arxiv.org/abs/2501.17161

    2025 arXiv

  3. [11]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?

    Y . Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y . Yue, S. Song, and G. Huang, “Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?” 2025. [Online]. Available: https://arxiv.org/abs/2504.13837

    2025 arXiv

  4. [12]

    LibriTTS: A Corpus Derived from LibriSpeech for Text-to- Speech,

    H. Zen, V . Dang, R. Clarket al., “LibriTTS: A Corpus Derived from LibriSpeech for Text-to- Speech,” in Proc. ISCA Interspeech, 2019, pp. 1526–1530

    2019

  5. [13]

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsaiet al., “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” IEEE/ACM Trans. ASLP ., vol. 29, pp. 3451– 3460, 2021

    2021

  6. [14]

    CosyV oice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens,

    Z. Du, Q. Chen, S. Zhang et al., “CosyV oice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  7. [15]

    Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,

    S. Chen, C. Wang, Y . Wuet al., “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” IEEE/ACM Trans. ASLP ., pp. 1–15, 2025

    2025

  8. [16]

    V oiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild,

    P. Peng, P.-Y . Huang, S.-W. Liet al., “V oiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild,” in Proc. ACL, Aug. 2024, pp. 12 442–12 462

    2024

  9. [17]

    Seed-TTS: A Family of High-Quality Versatile Speech Generation Models,

    P. Anastassiou, J. Chen, J. Chen et al., “Seed-TTS: A Family of High-Quality Versatile Speech Generation Models,” arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  10. [18]

    BASE TTS: Lessons from Building a Billion-Parameter Text-to-Speech Model on 100K Hours of Data,

    M. Łajszczak, G. Cámbara, Y . Liet al., “BASE TTS: Lessons from Building a Billion-Parameter Text-to-Speech Model on 100K Hours of Data,”arXiv preprint arXiv:2402.08093, 2024. 11

    2024 arXiv

  11. [19]

    ELLA-V: Stable Neural Codec Language Modelling with Alignment-Guided Sequence Reordering,

    Y . Song, Z. Chen, X. Wanget al., “ELLA-V: Stable Neural Codec Language Modelling with Alignment-Guided Sequence Reordering,” Proc. AAAI, 2025

    2025

  12. [20]

    V ALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment,

    B. Han, L. Zhou, S. Liu et al., “V ALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment,” arXiv preprint arXiv:2406.07855, 2024

    2024 arXiv

  13. [21]

    V ALL-T: Decoder-only generative transducer for robust and decoding-controllable text-to-speech,

    C. Du, Y . Guo, H. Wang, Y . Yang, Z. Niu, S. Wang, H. Zhang, X. Chen, and K. Yu, “V ALL-T: Decoder-only generative transducer for robust and decoding-controllable text-to-speech,” in Proc. IEEE ICASSP, 2025, pp. 1–5

    2025

  14. [22]

    RALL-E: Robust Codec Language Modelling with Chain-of- Thought Prompting for Text-to-Speech Synthesis,

    D. Xin, X. Tan, K. Shen et al., “RALL-E: Robust Codec Language Modelling with Chain-of- Thought Prompting for Text-to-Speech Synthesis,”arXiv preprint arXiv:2404.03204, 2024

    2024 arXiv

  15. [23]

    SNAC: Multi-Scale Neural Audio Codec,

    H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer, “SNAC: Multi-Scale Neural Audio Codec,” arXiv preprint arXiv:2410.14411, 2024

    2024 arXiv

  16. [24]

    Speaking from Coarse to Fine: Improving Neural Codec Lan- guage Model via Multi-Scale Speech Coding and Generation,

    H. Guo, F. Xie, D. Yanget al., “Speaking from Coarse to Fine: Improving Neural Codec Lan- guage Model via Multi-Scale Speech Coding and Generation,”arXiv preprint arXiv:2409.11630, 2024

    2024 arXiv

  17. [25]

    UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner,

    D. Yang, H. Guo, Y . Wanget al., “UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner,” inProc. NeurIPS, 2024

    2024

  18. [26]

    SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models,

    X. Zhang, D. Zhang, S. Li et al., “SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models,” in Proc. ICLR, 2024

    2024

  19. [27]

    Moshi: A Speech-Text Foundation Model for Real-Time Dialogue,

    A. Défossez, L. Mazaré, M. Orsini et al. , “Moshi: A Speech-Text Foundation Model for Real-Time Dialogue,” arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  20. [28]

    AudioLM: A Language Modeling Approach to Audio Generation,

    Z. Borsos, R. Marinier, D. Vincentet al., “AudioLM: A Language Modeling Approach to Audio Generation,” IEEE/ACM Trans. ASLP ., vol. 31, pp. 2523–2533, 2023

    2023

  21. [29]

    Speak, Read and Prompt: High-Fidelity Text-to- Speech with Minimal Supervision,

    E. Kharitonov, D. Vincent, Z. Borsos et al., “Speak, Read and Prompt: High-Fidelity Text-to- Speech with Minimal Supervision,” TACL, vol. 11, pp. 1703–1718, 2023

    2023

  22. [30]

    MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer,

    Y . Wang, H. Zhan, L. Liuet al., “MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer,” in Proc. ICLR, 2025

    2025

  23. [31]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads,

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” arXiv preprint arXiv:2401.10774, 2024

    2024 arXiv

  24. [32]

    Better & faster large language models via multi-token prediction,

    F. Gloeckle, B. Y . Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve, “Better & faster large language models via multi-token prediction,” arXiv preprint arXiv:2404.19737, 2024

    2024 arXiv

  25. [33]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  26. [34]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  27. [35]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  28. [36]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  29. [37]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  30. [38]

    Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback,

    G.-T. Lin, P. G. Shivakumar, A. Gouravet al., “Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback,” arXiv preprint arXiv:2411.01834, 2024. 12

    2024 arXiv

  31. [39]

    Robust zero-shot text-to-speech synthesis with reverse inference optimization,

    Y . Hu, C. Chen, S. Wang, E. S. Chng, and C. Zhang, “Robust zero-shot text-to-speech synthesis with reverse inference optimization,” arXiv preprint arXiv:2407.02243, 2024

    2024 arXiv

  32. [40]

    Enhancing zero-shot text-to- speech synthesis with human feedback,

    C. Chen, Y . Hu, W. Wu, H. Wang, E. S. Chng, and C. Zhang, “Enhancing zero-shot text-to- speech synthesis with human feedback,” arXiv preprint arXiv:2406.00654, 2024

    2024 arXiv

  33. [41]

    SpeechAlign: Aligning speech generation to human preferences,

    D. Zhang, Z. Li, S. Li, X. Zhang, P. Wang, Y . Zhou, and X. Qiu, “SpeechAlign: Aligning speech generation to human preferences,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. [Online]. Available: https://openreview.net/forum?id=SKCbZR8Pyd

    2024

  34. [42]

    Fine-grained preference optimization improves zero-shot text-to-speech,

    J. Yao, Y . Yang, Y . Pan, Y . Feng, Z. Ning, J. Ye, H. Zhou, and L. Xie, “Fine-grained preference optimization improves zero-shot text-to-speech,” arXiv preprint arXiv:2502.02950, 2025

    2025

  35. [43]

    V ALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers,

    S. Chen, S. Liu, L. Zhou et al., “V ALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers,”arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  36. [44]

    UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and V ocoding,

    C. Du, Y . Guo, F. Shenet al., “UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and V ocoding,” inProc. AAAI, vol. 38, no. 16, 2024, pp. 17 924– 17 932

    2024

  37. [45]

    LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec,

    Y . Guo, Z. Li, C. Duet al., “LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec,” arXiv preprint arXiv:2410.15764, 2024

    2024 arXiv

  38. [46]

    WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,

    S. Chen, C. Wang, Z. Chen et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,” IEEE JSTSP, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  39. [47]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,

    S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural networks, vol. 107, pp. 3–11, 2018

    2018

  40. [48]

    UTMOS: UTokyo- SaruLab System for V oiceMOS Challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo- SaruLab System for V oiceMOS Challenge 2022,” inInterspeech 2022, 2022, pp. 4521–4525

    2022

  41. [49]

    Robust Speech Recognition via Large-Scale Weak Supervision,

    A. Radford, J. W. Kim, T. Xu et al. , “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proc. ICML. PMLR, 2023, pp. 28 492–28 518

    2023

  42. [50]

    Zipformer: A faster and better encoder for automatic speech recognition,

    Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” arXiv preprint arXiv:2310.11230, 2023. 13 Appendix A Limitations and broader impacts 15 B Details of DCPO algorithm 15...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.