pith. machine review for the scientific record. sign in

arxiv: 2406.02430 · v1 · submitted 2024-06-04 · 📡 eess.AS · cs.SD

Recognition: 3 theorem links

· Lean Theorem

Seed-TTS: A Family of High-Quality Versatile Speech Generation Models

Philip Anastassiou , Jiawei Chen , Jitong Chen , Yuanzhe Chen , Zhuo Chen , Ziyi Chen , Jian Cong , Lelai Deng
show 38 more authors
Chuang Ding Lu Gao Mingqing Gong Peisong Huang Qingqing Huang Zhiying Huang Yuanyuan Huo Dongya Jia Chumin Li Feiya Li Hui Li Jiaxin Li Xiaoyang Li Xingxing Li Lin Liu Shouda Liu Sichao Liu Xudong Liu Yuchen Liu Zhengxi Liu Lu Lu Junjie Pan Xin Wang Yuping Wang Yuxuan Wang Zhen Wei Jian Wu Chao Yao Yifeng Yang Yuanhao Yi Junteng Zhang Qidi Zhang Shuo Zhang Wenjie Zhang Yang Zhang Zilin Zhao Dejian Zhong Xiaobin Zhuang
Authors on Pith no claims yet

Pith reviewed 2026-05-15 12:22 UTC · model grok-4.3

classification 📡 eess.AS cs.SD
keywords text-to-speechspeech synthesisautoregressive modelsdiffusion modelsspeaker similarityemotion controlspeech editingfoundation models
0
0 comments X

The pith

Seed-TTS generates speech that matches human recordings in speaker similarity and naturalness according to objective metrics and listener tests.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Seed-TTS as a family of large-scale autoregressive text-to-speech models that produce speech virtually indistinguishable from human speech. These models match ground truth performance in speaker similarity and naturalness through both objective measures and subjective evaluations, while supporting in-context learning for new speakers and fine control over attributes such as emotion. A non-autoregressive variant called Seed-TTS_DiT uses a fully diffusion-based architecture for end-to-end generation without relying on pre-estimated phoneme durations. The authors add self-distillation for speech factorization and reinforcement learning to improve robustness, similarity, and controllability, positioning the system as a versatile foundation model for expressive speech generation across diverse conditions.

Core claim

Seed-TTS achieves performance in speaker similarity and naturalness that matches ground truth human speech in both objective and subjective evaluations, serving as a foundation model for speech generation with superior controllability over speech attributes such as emotion and the ability to produce highly expressive and diverse speech for speakers in the wild.

What carries the argument

Large-scale autoregressive text-to-speech model enhanced by self-distillation for speech factorization and reinforcement learning for robustness, paired with a fully diffusion-based non-autoregressive architecture that performs end-to-end speech generation without pre-estimated durations.

If this is right

  • Fine-tuning produces even higher subjective scores in naturalness and speaker similarity.
  • The models support effective in-context learning for speakers outside the training set.
  • Seed-TTS_DiT enables speech editing through its end-to-end diffusion process.
  • Reinforcement learning improves robustness and controllability over emotional expression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread use could replace recorded human voices in media and virtual agents if performance holds outside controlled test conditions.
  • The factorization and reinforcement learning steps might transfer to other audio generation tasks such as music or sound effects.
  • Real-time deployment could support dynamic, personalized voice output in interactive systems without per-speaker retraining.

Load-bearing premise

Subjective listener evaluations and chosen objective metrics reliably indicate real-world indistinguishability from human speech and that the models generalize to unseen speakers and conditions without overfitting.

What would settle it

A blind listening test with many participants across varied real-world conditions and unseen speakers where listeners cannot distinguish Seed-TTS outputs from actual human recordings at rates above chance.

read the original abstract

We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech (TTS) models capable of generating speech that is virtually indistinguishable from human speech. Seed-TTS serves as a foundation model for speech generation and excels in speech in-context learning, achieving performance in speaker similarity and naturalness that matches ground truth human speech in both objective and subjective evaluations. With fine-tuning, we achieve even higher subjective scores across these metrics. Seed-TTS offers superior controllability over various speech attributes such as emotion and is capable of generating highly expressive and diverse speech for speakers in the wild. Furthermore, we propose a self-distillation method for speech factorization, as well as a reinforcement learning approach to enhance model robustness, speaker similarity, and controllability. We additionally present a non-autoregressive (NAR) variant of the Seed-TTS model, named $\text{Seed-TTS}_\text{DiT}$, which utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS systems, $\text{Seed-TTS}_\text{DiT}$ does not depend on pre-estimated phoneme durations and performs speech generation through end-to-end processing. We demonstrate that this variant achieves comparable performance to the language model-based variant and showcase its effectiveness in speech editing. We encourage readers to listen to demos at \url{https://bytedancespeech.github.io/seedtts_tech_report}.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces Seed-TTS, a family of large-scale autoregressive TTS models (with a diffusion-based NAR variant Seed-TTS_DiT) that generate speech claimed to be virtually indistinguishable from human speech. It reports matching ground-truth performance in speaker similarity and naturalness via objective and subjective evaluations, strong in-context learning, controllability over attributes like emotion, and further gains from fine-tuning, self-distillation for factorization, and RL for robustness.

Significance. If the central performance claims hold under rigorous scrutiny, the work would constitute a meaningful contribution to speech generation by providing versatile foundation models with high fidelity, expressiveness for in-the-wild speakers, and end-to-end NAR processing without pre-estimated durations. The combination of AR and DiT architectures plus RL enhancements offers practical value, though the current lack of evaluation transparency limits immediate assessment of its standing relative to prior TTS systems.

major comments (3)
  1. [Abstract] Abstract: The claim that Seed-TTS 'matches ground truth human speech in both objective and subjective evaluations' and produces output that is 'virtually indistinguishable' is load-bearing for the central contribution, yet the manuscript provides no details on the subjective protocol (forced-choice discrimination vs. scalar MOS/ABX ratings, number of listeners and utterances, presentation of ground-truth references, or strict held-out test speakers/conditions). Scalar ratings alone can approach ceiling values without proving indistinguishability.
  2. [Section 4] Section 4 (Experiments) and abstract: No information is given on training data scale, exact objective metrics (e.g., specific speaker similarity measures or their computation), chosen baselines, or statistical significance (error bars, p-values). This absence prevents evaluation of whether reported gains are robust or could be explained by data scale or overfitting.
  3. [Section 5] Section 5 (fine-tuning and RL): The post-hoc fine-tuning and RL improvements are presented as achieving 'even higher subjective scores,' but without reporting the base vs. fine-tuned comparison tables, training schedules, or controls for data leakage, it is unclear whether these gains reflect genuine robustness enhancements or simply additional adaptation to the evaluation distribution.
minor comments (2)
  1. [Abstract] The manuscript would benefit from explicit cross-references in the text to specific demo audio examples that illustrate the controllability and editing claims.
  2. [Section 3] Notation for the DiT variant (Seed-TTS_DiT) is introduced without a dedicated equation or diagram clarifying how the diffusion process replaces autoregressive token prediction while remaining end-to-end.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We appreciate the referee's detailed feedback on our manuscript. We will revise the paper to address the concerns regarding evaluation transparency and provide more details on the experimental setup. Below we respond to each major comment.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that Seed-TTS 'matches ground truth human speech in both objective and subjective evaluations' and produces output that is 'virtually indistinguishable' is load-bearing for the central contribution, yet the manuscript provides no details on the subjective protocol (forced-choice discrimination vs. scalar MOS/ABX ratings, number of listeners and utterances, presentation of ground-truth references, or strict held-out test speakers/conditions). Scalar ratings alone can approach ceiling values without proving indistinguishability.

    Authors: We agree that additional details on the subjective evaluation protocol are essential to support the claims. In the revised version, we will expand the abstract and add a dedicated subsection in Section 4 describing the listening test methodology. This will include: the use of ABX or MOS ratings, number of participants (e.g., 20+ native speakers), number of utterances per condition, how ground-truth references were presented, and confirmation that evaluations used held-out speakers and in-the-wild conditions not seen during training. We believe this will demonstrate the indistinguishability more rigorously. revision: yes

  2. Referee: [Section 4] Section 4 (Experiments) and abstract: No information is given on training data scale, exact objective metrics (e.g., specific speaker similarity measures or their computation), chosen baselines, or statistical significance (error bars, p-values). This absence prevents evaluation of whether reported gains are robust or could be explained by data scale or overfitting.

    Authors: We acknowledge this gap in the current draft. The revised manuscript will include: (1) details on the training data scale, such as the total hours of speech data used (noting it is on the order of tens of thousands of hours from diverse sources); (2) exact definitions and computation methods for objective metrics, e.g., speaker similarity via cosine distance on embeddings from a pre-trained speaker verification model like ECAPA-TDNN; (3) a full list of baselines compared against, including recent TTS systems; and (4) statistical analysis with error bars from multiple runs or bootstrap methods and p-values for key comparisons. This will allow readers to assess the robustness of the results. revision: yes

  3. Referee: [Section 5] Section 5 (fine-tuning and RL): The post-hoc fine-tuning and RL improvements are presented as achieving 'even higher subjective scores,' but without reporting the base vs. fine-tuned comparison tables, training schedules, or controls for data leakage, it is unclear whether these gains reflect genuine robustness enhancements or simply additional adaptation to the evaluation distribution.

    Authors: We will revise Section 5 to include direct comparison tables between the base Seed-TTS model and the fine-tuned/RL versions on the same evaluation sets. We will detail the fine-tuning schedules, hyperparameters, and the RL reward design. To address data leakage concerns, we will clarify that all fine-tuning and RL stages used disjoint data splits from the evaluation sets, with no overlap in speakers or utterances. This will show that the improvements stem from the proposed self-distillation and RL techniques rather than overfitting to the test distribution. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper introduces empirical TTS models (autoregressive Seed-TTS and diffusion-based Seed-TTS_DiT) trained on large-scale data, with proposed techniques like self-distillation for factorization and RL for robustness. All load-bearing claims rest on external objective metrics and subjective listener evaluations compared against ground-truth human speech, not on internal derivations that reduce to fitted inputs by construction. No equations, uniqueness theorems, or self-citations are invoked to force results; performance matching is demonstrated via held-out test comparisons rather than tautological renaming or self-referential fitting. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The central claims rest on standard assumptions in neural speech modeling plus many unspecified training choices; no new entities are postulated.

free parameters (2)
  • model scale and hyperparameters
    Large autoregressive and diffusion architectures require numerous tuned parameters whose values are not reported in the abstract.
  • fine-tuning schedule
    Performance gains after fine-tuning depend on unspecified data selection and optimization choices.
axioms (2)
  • domain assumption Neural networks can accurately model the distribution of natural speech waveforms
    Invoked implicitly by the autoregressive and diffusion architectures.
  • domain assumption Subjective human ratings and standard objective metrics (e.g., similarity scores) are valid proxies for perceptual quality
    Underpins all reported evaluation claims.

pith-pipeline@v0.9.0 · 5723 in / 1266 out tokens · 28066 ms · 2026-05-15T12:22:23.552594+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech

    eess.AS 2026-05 unverdicted novelty 7.0

    GibbsTTS combines a training-free kinetic-optimal scheduler with finite-step moment correction in MI-DFM to deliver top naturalness and strong speaker similarity in zero-shot TTS.

  2. MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech

    eess.AS 2026-04 unverdicted novelty 7.0

    MINT-Bench is a new benchmark using hierarchical taxonomy, multi-stage data pipeline, and hybrid evaluation to assess instruction-following TTS systems, revealing major gaps in compositional and paralinguistic controls.

  3. From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench

    cs.AI 2026-04 unverdicted novelty 7.0

    ProVoice-Bench is the first framework to evaluate proactive voice agents, revealing that state-of-the-art multimodal LLMs struggle with over-triggering and context-aware reasoning.

  4. X-VC: Zero-shot Streaming Voice Conversion in Codec Space

    eess.AS 2026-04 unverdicted novelty 7.0

    X-VC achieves zero-shot streaming voice conversion via one-step codec-space conversion with dual-conditioning acoustic converter and role-assignment training on generated paired data.

  5. CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

    cs.SD 2026-04 unverdicted novelty 7.0

    CapTalk unifies single-utterance and dialogue voice design via utterance- and speaker-level captions plus a hierarchical variational module for stable timbre with adaptive expression.

  6. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models

    cs.SD 2025-07 unverdicted novelty 7.0

    Audio Flamingo 3 introduces an open large audio-language model achieving new state-of-the-art results on over 20 audio understanding and reasoning benchmarks using a unified encoder and curriculum training on open data.

  7. MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    MMControl adds multi-modal controls for identity, timbre, pose, and layout to unified audio-video diffusion models via dual-stream injection and adjustable guidance scaling.

  8. Text-To-Speech with Chain-of-Details: modeling temporal dynamics in speech generation

    eess.AS 2026-04 unverdicted novelty 6.0

    Chain-of-Details (CoD) is a cascaded TTS method that explicitly models temporal coarse-to-fine dynamics with a shared decoder, achieving competitive performance using significantly fewer parameters.

  9. Bridging What the Model Thinks and How It Speaks: Self-Aware Speech Language Models for Expressive Speech Generation

    cs.CL 2026-04 unverdicted novelty 6.0

    SA-SLM uses variational information bottleneck for intent-aware bridging and self-criticism for realization-aware alignment to close the semantic-acoustic gap, outperforming open-source models and nearing GPT-4o-Audio...

  10. OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    OmniVoice introduces a diffusion language model-style non-autoregressive TTS system that directly maps text to multi-codebook acoustic tokens, scaling zero-shot synthesis to over 600 languages with SOTA results on mul...

  11. Qwen3-Omni Technical Report

    cs.CL 2025-09 unverdicted novelty 6.0

    Qwen3-Omni is a unified multimodal model that achieves open-source SOTA on 32 of 36 audio and audio-visual benchmarks and overall SOTA on 22 without degrading performance on text, image, or video relative to single-mo...

  12. JaiTTS: A Thai Voice Cloning Model

    cs.CL 2026-04 unverdicted novelty 5.0

    JaiTTS-v1.0 achieves a character error rate of 1.94% on short Thai speech tasks, surpassing human ground truth of 1.98%, matches humans on long tasks, and wins 283 of 400 human pairwise comparisons against commercial models.

  13. JaiTTS: A Thai Voice Cloning Model

    cs.CL 2026-04 unverdicted novelty 5.0

    JaiTTS-v1.0 achieves 1.94% CER on short Thai speech, beating human ground truth of 1.98%, matches humans on long speech, and wins 283 of 400 human comparisons against commercial systems.

  14. Qwen3.5-Omni Technical Report

    cs.CL 2026-04 unverdicted novelty 5.0

    Qwen3.5-Omni scales an omnimodal model to hundreds of billions of parameters with 256k context, introduces ARIA for stable speech synthesis, and reports SOTA performance on 215 audio-visual benchmarks while adding mul...

  15. Controllable Singing Style Conversion with Boundary-Aware Information Bottleneck

    cs.SD 2026-04 unverdicted novelty 5.0

    A singing voice conversion system with boundary-aware information bottleneck and high-frequency augmentation achieves the best naturalness in SVCC2025 subjective tests while using less extra data than competitors.

  16. Voxtral TTS

    cs.AI 2026-03 unverdicted novelty 5.0

    Voxtral TTS produces expressive multilingual speech from 3-second reference audio with a hybrid autoregressive-plus-flow-matching architecture and a new VQ-FSQ tokenizer, achieving 68.4% win rate over ElevenLabs in hu...

  17. WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

    cs.CL 2026-03 unverdicted novelty 5.0

    WAND adapts AR-TTS models to constant complexity via windowed attention and distillation, cutting KV cache memory by up to 66.2% while preserving quality and achieving length-invariant latency.

  18. Qwen2.5-Omni Technical Report

    cs.CL 2025-03 conditional novelty 5.0

    Qwen2.5-Omni presents a multimodal model with block-wise encoders, TMRoPE position embeddings, and a Thinker-Talker architecture that enables simultaneous text and streaming speech generation while matching text perfo...

  19. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

    cs.SD 2024-12 unverdicted novelty 5.0

    CosyVoice 2 delivers human-parity naturalness and near-lossless streaming speech synthesis by combining finite-scalar quantization, a streamlined pre-trained LLM, and chunk-aware causal flow matching on large multilin...

  20. Empowering Video Translation using Multimodal Large Language Models

    cs.CV 2026-04 unverdicted novelty 4.0

    The paper offers the first focused review of MLLM-based video translation organized by a three-role taxonomy of Semantic Reasoner, Expressive Performer, and Visual Synthesizer, plus open challenges.

Reference graph

Works this paper leans on

45 extracted references · 45 canonical work pages · cited by 19 Pith papers · 12 internal anchors

  1. [1]

    Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance

    Yuanzhe Chen, Ming Tu, Tang Li, Xin Li, Qiuqiang Kong, Jiaxin Li, Zhichao Wang, Qiao Tian, Yuping Wang, and Yuxuan Wang. Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE,

    work page 2023

  2. [2]

    StreamV oice: Streamable context-aware language modeling for real-time zero-shot voice conversion

    Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Zhuo Chen, Lei Xie, Yuping Wang, and Yuxuan Wang. StreamV oice: Streamable context-aware language modeling for real-time zero-shot voice conversion. arXiv preprint arXiv:2401.11053, 2024a. Zhichao Wang, Yuanzhe Chen, Lei Xie, Qiao Tian, and Yuping Wang. LM-VC: Zero-shot voice conversion via speech generation base...

    work page arXiv

  3. [3]

    BASE TTS: Lessons from building a billion-parameter text-to-speech model on 100k hours of data

    Mateusz Łajszczak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín-Cortinas, Ammar Abbas, Adam Michalski, et al. BASE TTS: Lessons from building a billion-parameter text-to-speech model on 100k hours of data. arXiv preprint arXiv:2402.08093,

    work page arXiv

  4. [4]

    Mega-TTS: Zero-shot text-to-speech at scale with intrinsic inductive bias

    Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-TTS: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509,

    work page arXiv

  5. [5]

    Deep Reinforcement Learning: An Overview

    Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Neural codec language models are zero-shot text to speech synthesizers, 2023b

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023b. Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et...

    work page arXiv

  7. [7]

    E3 TTS: Easy end-to-end diffusion- based text to speech

    Yuan Gao, Nobuyuki Morioka, Yu Zhang, and Nanxin Chen. E3 TTS: Easy end-to-end diffusion- based text to speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023a. Zehua Chen, Yihan Wu, Yichong Leng, Jiawei Chen, Haohe Liu, Xu Tan, Yang Cui, Ke Wang, Lei He, Sheng Zhao, et al. ResGrad: Residual denoising di...

    work page arXiv 2023

  8. [8]

    LLaMA: Open and Efficient Foundation Language Models

    In INTERSPEECH, pages 1606–1610, 2022a. Yuanhao Yi, Lei He, Shifeng Pan, Xi Wang, and Yujia Xiao. ProsodySpeech: Towards advanced prosody model for neural text-to-speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7582–7586. IEEE, 2022b. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  9. [9]

    Better speech synthesis through scaling

    James Betker. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07243,

    work page arXiv

  10. [10]

    BigVGAN: A universal neural vocoder with large-scale training

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. BigVGAN: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658,

    work page arXiv

  11. [11]

    Glow-WaveGAN: Learning speech representations from gan-based variational auto-encoder for high fidelity flow-based speech synthesis

    Jian Cong, Shan Yang, Lei Xie, and Dan Su. Glow-WaveGAN: Learning speech representations from gan-based variational auto-encoder for high fidelity flow-based speech synthesis. arXiv preprint arXiv:2106.10831,

    work page arXiv

  12. [12]

    Basis-MelGAN: Efficient neural vocoder based on audio decomposi- tion

    Zhengxi Liu and Yanmin Qian. Basis-MelGAN: Efficient neural vocoder based on audio decomposi- tion. arXiv preprint arXiv:2106.13419,

    work page arXiv

  13. [13]

    Common V oice: A massively- multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common V oice: A massively- multilingual speech corpus. arXiv preprint arXiv:1912.06670,

    work page arXiv 1912

  14. [14]

    DiDiSpeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. DiDiSpeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6968–6972. IEEE,

    work page 2021

  15. [15]

    FunASR: A fundamental end-to-end speech recognition toolkit

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. FunASR: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023b. 14 Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng. Large-scale self-super...

    work page arXiv 2022

  16. [16]

    FastSpeech 2: Fast and high-quality end-to-end text to speech

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558,

    work page arXiv 2006

  17. [17]

    Controllable and lossless non-autoregressive end-to-end text-to-speech

    Zhengxi Liu, Qiao Tian, Chenxu Hu, Xudong Liu, Menglin Wu, Yuping Wang, Hang Zhao, and Yuxuan Wang. Controllable and lossless non-autoregressive end-to-end text-to-speech. arXiv preprint arXiv:2207.06088,

    work page arXiv

  18. [18]

    LibriSpeech: an ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: an ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE,

    work page 2015

  19. [19]

    WeNet 2.0: More productive end-to-end speech recognition toolkit

    Binbin Zhang, Di Wu, Zhendong Peng, Xingchen Song, Zhuoyuan Yao, Hang Lv, Lei Xie, Chao Yang, Fuping Pan, and Jianwei Niu. WeNet 2.0: More productive end-to-end speech recognition toolkit. arXiv preprint arXiv:2203.15455,

    work page arXiv

  20. [20]

    Developing far-field speaker system via teacher-student learning

    Jinyu Li, Rui Zhao, Zhuo Chen, Changliang Liu, Xiong Xiao, Guoli Ye, and Yifan Gong. Developing far-field speaker system via teacher-student learning. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5699–5703. IEEE,

    work page 2018

  21. [21]

    V oxCeleb: A large-scale speaker identifica- tion dataset

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. V oxCeleb: A large-scale speaker identifica- tion dataset. arXiv preprint arXiv:1706.08612,

    work page arXiv

  22. [22]

    Singing Voice Synthesis Using Deep Autoregressive Neural Networks for Acoustic Modeling

    Yuan-Hao Yi, Yang Ai, Zhen-Hua Ling, and Li-Rong Dai. Singing voice synthesis using deep autoregressive neural networks for acoustic modeling. arXiv preprint arXiv:1906.08977,

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  23. [23]

    LiteSing: Towards fast, lightweight and expressive singing voice synthesis

    Xiaobin Zhuang, Tao Jiang, Szu-Yu Chou, Bin Wu, Peng Hu, and Simon Lui. LiteSing: Towards fast, lightweight and expressive singing voice synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7078–7082. IEEE,

    work page 2021

  24. [24]

    Prosody-aware SpeechT5 for expressive neural TTS

    Yan Deng, Long Zhou, Yuanhao Yi, Shujie Liu, and Lei He. Prosody-aware SpeechT5 for expressive neural TTS. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE,

    work page 2023

  25. [25]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

    15 Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. AWQ: Activation- aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978,

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Consistency Models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206,

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    A white paper on neural network quantization

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295,

    work page arXiv

  32. [32]

    decoupleq: Towards 2-bit post-training uniform quantization via decoupling parameters into integer and floating points

    Yi Guo, Fanliu Kong, Xiaoyang Li, Hui Li, Wei Chen, Xiaogang Tian, Jinping Cai, Yang Zhang, and Shouda Liu. decoupleq: Towards 2-bit post-training uniform quantization via decoupling parameters into integer and floating points. arXiv preprint arXiv:2404.12759,

    work page arXiv

  33. [33]

    V oiceShop: A unified speech-to-speech framework for identity- preserving zero-shot voice editing

    Philip Anastassiou, Zhenyu Tang, Kainan Peng, Dongya Jia, Jiaxin Li, Ming Tu, Yuping Wang, Yuxuan Wang, and Mingbo Ma. V oiceShop: A unified speech-to-speech framework for identity- preserving zero-shot voice editing. arXiv preprint arXiv:2404.06674,

    work page arXiv

  34. [34]

    HierSpeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis

    Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, and Seong-Whan Lee. HierSpeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.12454,

    work page arXiv

  35. [35]

    Zero-shot accent conversion using pseudo siamese disentanglement network

    Dongya Jia, Qiao Tian, Kainan Peng, Jiaxin Li, Yuanzhe Chen, Mingbo Ma, Yuping Wang, and Yuxuan Wang. Zero-shot accent conversion using pseudo siamese disentanglement network. arXiv preprint arXiv:2212.05751,

    work page arXiv

  36. [36]

    Diffusion-based voice conversion with fast maximum likelihood sampling scheme

    Vadim Popov, Ivan V ovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint arXiv:2109.13821,

    work page arXiv

  37. [37]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    Diffusion model alignment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12908,

    work page arXiv

  39. [39]

    MusicRL: Aligning music generation to human preferences

    Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, et al. MusicRL: Aligning music generation to human preferences. arXiv preprint arXiv:2402.04229,

    work page arXiv

  40. [40]

    SpeechAlign: Aligning speech generation to human preferences

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechAlign: Aligning speech generation to human preferences. arXiv preprint arXiv:2404.05600,

    work page arXiv

  41. [41]

    Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint arXiv:2402.14740,

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [42]

    Minimum word error rate training for attention-based sequence-to-sequence models

    Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Cheng Chiu, and Anjuli Kannan. Minimum word error rate training for attention-based sequence-to-sequence models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4839–4843. IEEE,

    work page 2018

  43. [43]

    Transforming and combining rewards for aligning large language models

    Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D’Amour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models. arXiv preprint arXiv:2402.00742, 2024b. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint ...

    work page arXiv

  44. [44]

    Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion - tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737,

    work page internal anchor Pith review Pith/arXiv arXiv

  45. [45]

    SpeechX: Neural codec language model as a versatile speech transformer

    Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. SpeechX: Neural codec language model as a versatile speech transformer. arXiv preprint arXiv:2308.06873, 2023c. OpenAI. Navigating the challenges and opportunities of synthetic voices. https://openai.com/index/nav...

    work page arXiv