Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Autoregressive Speech Enhancement via Acoustic Tokens

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that in token-based speech enhancement, acoustic tokens from EnCodec and DAC preserve speaker identity better than semantic tokens from WavLM, that an autoregressive transducer improves intelligibility, and that discrete…

desk verdict A modest but solid empirical study of acoustic-token enhancement; the BSR step is undefined and the acoustic-vs-semantic claim is stronger than a single configuration supports, but the comparison is fair and the paper deserves serious review. read the letter →

arxiv 2507.12825 v1 pith:3OZOLYNQ submitted 2025-07-17 cs.SD cs.LGeess.AS

Luca Della Libera , Cem Subakan , Mirco Ravanelli This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords speechenhancementacoustictokenssemanticautoregressivetransducerspeakeridentityEnCodecDACWavLMdiscrete
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether speech enhancement can be done on discrete acoustic tokens, and whether an autoregressive model improves on the conditionally independent models used so far. On VoiceBank and Libri1Mix it reports that acoustic tokens from EnCodec and DAC preserve speaker identity better than semantic tokens extracted from WavLM, showing higher cosine similarity in every setting. It also introduces the Speech Enhancement Transducer, an encoder–predictor–joiner language model that improves intelligibility in several configurations. The paper's honest bottom line is that discrete representations still fall short of continuous features, especially on intelligibility, which it attributes to the small amount of training data for high-bitrate token language models.

What carries the argument

The load-bearing mechanism is the token conversion and the Speech Enhancement Transducer (SET) language model. The tokenizer maps raw noisy speech into discrete tokens—either residual-vector-quantized acoustic tokens from pretrained codecs or k-means quantized semantic tokens from WavLM—and the detokenizer maps predicted tokens back to waveforms. SET adapts the speech-recognition transducer to enhancement: an encoder summarizes the noisy token sequence, an autoregressive causal predictor summarizes the enhanced tokens produced so far, and a joiner combines both representations by a linear layer plus element-wise sum to predict the next enhanced token. The model is trained end-to-end by cross-entropy with teacher forcing, and at inference uses five-beam search followed by a refinement pass that re-feeds the predicted tokens to the model; this refinement is what mitigates exposure bias in multi-codebook prediction.

What would settle it

Train the WavLM semantic tokenizer with 1024 or 4096 k-means clusters and with features from several transformer layers, then rerun the VoiceBank and Libri1Mix enhancement experiments; if the cosine-similarity gap to EnCodec/DAC disappears or reverses, the claimed acoustic-over-semantic advantage is an artifact of tokenizer configuration.

Watch

Extended reading notes

Core claim

The central claim is empirical and comparative: when a language model is trained to map noisy speech tokens to clean speech tokens, acoustic codec tokens (EnCodec, DAC) preserve speaker identity better than semantic tokens (k-means over WavLM), with higher x-vector cosine similarity across all tested settings. The paper further claims that its autoregressive Speech Enhancement Transducer (SET) improves intelligibility over the non-autoregressive Conformer baseline in several settings, while acoustic codecs such as DAC gain more from autoregression at higher bitrates. It also reports a clear gap between all discrete methods and continuous baselines—WavLM continuous and SepFormer—which is largest for intelligibility; the authors' explanation is that training a language model on high-bitrate tokens needs far more than the roughly one hundred hours used here.

Load-bearing premise

The comparison assumes that a 512-cluster k-means quantizer on WavLM layer 6 is a fair representative of semantic tokens, while EnCodec and DAC are heavily optimized pretrained codecs, so the speaker-identity gap could shrink or reverse if the semantic tokenizer were better tuned.

Editorial extensions

If this is right

  • If acoustic tokens really preserve speaker identity better, token-based enhancement can be used in applications where speaker fidelity matters, such as voice conversion or speaker-verification frontends.
  • Autoregressive modeling of audio tokens is worth pursuing: SET shows consistent intelligibility gains over conditionally independent prediction, especially for DAC at high bitrates and low SNR.
  • The exposure-bias result implies that multi-codebook audio language models need explicit mitigation, such as refinement or rescoring, before beam search is reliable.
  • Vocoder choice matters independently of the quantizer: swapping EnCodec's vocoder for Vocos raises DNSMOS and cosine similarity without retraining the codec.
  • Discrete enhancement cannot yet replace continuous regression; closing the word-error-rate gap likely requires larger training sets or better discrete representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the semantic tokenizer is a 512-cluster k-means quantizer while the acoustic codecs are heavily optimized pretrained systems, the speaker-identity advantage may partly reflect engineering asymmetry; a fair test with larger semantic codebooks or different WavLM layers could narrow it.
  • SET's transducer structure is streaming-compatible and could be combined with an external speech language model to rescore hypotheses and reduce word errors, a direction the paper only lists as future work.
  • The bitrate results suggest an optimum that depends on both codec and language-model capacity, so simply raising bitrate is not a reliable route to better enhancement; hybrid semantic-acoustic tokens might capture both speaker identity and intelligibility.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies speech enhancement with discrete audio tokens. It compares acoustic tokens from EnCodec and DAC against semantic tokens from a k-means quantizer on WavLM layer 6, and it introduces an autoregressive transducer-style architecture (SET) with encoder, predictor, and joiner components. Experiments on VoiceBank and Libri1Mix report DNSMOS, CosSim, and dWER. The paper claims that acoustic tokens preserve speaker identity better than semantic tokens, that the autoregressive SET architecture improves intelligibility on average, and that discrete representations still lag behind continuous ones such as WavLM continuous and SepFormer.

Significance. If the claims hold, the paper provides useful empirical evidence for a relatively underexplored direction: using acoustic codec tokens for speech enhancement rather than semantic tokens. The bitrate and noise-strength analyses in Fig. 2 are useful, and the SET architecture is a plausible contribution with potential for streaming and rescoring extensions. However, the manuscript's central empirical claims currently rest on a single semantic-tokenizer configuration, missing statistical uncertainty, and at least one undefined procedure (the refinement step), so the significance is conditional on addressing these issues.

major comments (4)
  1. [Section III-B, Section IV-C] The claim that acoustic tokens outperform semantic tokens for speaker identity is based on a single semantic-tokenizer configuration: k-means with 512 clusters on WavLM layer 6, trained on LibriSpeech train-clean-100. The acoustic codecs, in contrast, are heavily optimized pretrained RVQ codecs at 3.0 kbps with four codebooks and pretrained vocoders. A more finely tuned semantic quantizer (more clusters, a different WavLM layer, or a dedicated semantic codec) could close or reverse the CosSim gap. This is not a circularity problem, but it is a fairness problem for the categorical conclusion in the abstract. The issue is load-bearing because CosSim is the only metric supporting the 'acoustic beats semantic' claim, and some Libri1Mix differences are small (e.g., 0.894 vs 0.888 for DAC NAR). Please either test additional semantic configurations and report significance/error bars, or reframe the claim to the tested configurations.
  2. [Section IV-A, Table I] The description of the 'refinement step' is missing. Section IV-C says 'the refinement step introduced in Sec. III-B greatly reduces the impact of exposure bias,' but Sec. III-B contains no description of any refinement step. This is not a minor omission: the BSR numbers in Table II and the AR results in Table I depend on this step, and the exposure-bias argument is one of the paper's stated contributions. The step must be defined precisely, including how it operates on the beam output and how it differs from standard beam search.
  3. [Table II] The claim that the autoregressive SET architecture improves intelligibility is not consistent across all configurations. On VoiceBank, DAC AR has a higher dWER (31.69) than DAC NAR (31.18), and WavLM discrete AR also has a higher dWER (22.77) than WavLM discrete NAR (22.39). The paper says SET 'on average' improves intelligibility, but the average is not reported and there are no significance tests. Please provide the average with per-configuration breakdown, or restrict the claim to the configurations that actually improve.
  4. [Table II] The TF column in Table II is not an inference protocol: it appears to be teacher-forced evaluation, where the model receives ground-truth tokens. The paper does not state how TF values are computed, and the claimed exposure-bias gap (BSR vs TF) partly reflects the difference between an oracle procedure and a real decoding procedure. Please clarify how TF is evaluated and whether the conclusion about exposure bias is based on a corresponding decrease from a deployable decoding procedure.
minor comments (3)
  1. [Section IV-A] The text refers to 'WavLM discrete' as a semantic codec, but WavLM is not itself a codec; it is a self-supervised encoder with a separately trained k-means quantizer and vocoder. Rephrasing this as a semantic tokenizer based on WavLM would avoid confusion.
  2. [Table I] Table I says that the best values for each codec are highlighted in bold and the best overall values are framed, but the table as rendered does not show bold or framed entries. Please ensure the formatting is visible in the final version.
  3. [Section III-D] There is a typo in the learning-rate schedule description: 'annealead' should be 'annealed.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claims are empirical comparisons evaluated with external metrics, not derivations from fitted inputs or self-citations.

full rationale

The paper's derivation chain is an empirical benchmarking study, not a theoretical construction. Acoustic and semantic tokens are produced by fixed pretrained codecs (EnCodec, DAC) or by a standard k-means recipe on WavLM features; the language models (NAR and SET) are trained with cross-entropy loss to map noisy tokens to clean tokens; and all conclusions are evaluated with external metrics (DNSMOS, CosSim, dWER) on held-out test sets. No equation in the paper defines an output in terms of the claimed result, and no fitted parameter is renamed as a prediction. The claim that acoustic tokens outperform semantic tokens for speaker identity is a measured difference in Table I, not a consequence of how the tokenizers were constructed. Concerns that the semantic tokenizer uses only 512 clusters and one WavLM layer are comparability or fairness issues, not circular reasoning. The self-citations (SpeechBrain, SepFormer) are used as implementation baselines and are not load-bearing for the paper's conclusions. The conclusion that discrete representations lag continuous ones is likewise an empirical observation. Therefore, no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on benchmark comparisons rather than a formal derivation, so no fitted constant enforces the result. The listed hyperparameters are experimental choices that can influence rankings, and the listed axioms are domain assumptions about tokenizers, metrics, and training. No invented entities are introduced.

free parameters (3)
  • Codec bitrate = 3.0 kbps
    Main comparisons use EnCodec and DAC at 3.0 kbps with 4 codebooks; the relative performance of acoustic versus semantic tokens may change with bitrate, as the authors acknowledge in Fig. 2.
  • k-means cluster count = 512
    The semantic tokenizer uses 512 clusters on WavLM layer 6; speaker identity preservation could depend on this choice, making it a hand-chosen parameter that affects the comparison.
  • Beam search beam count = 5
    Autoregressive inference uses beam search with 5 beams; this value directly affects the reported AR results and is chosen without an ablation.
assumptions (4)
  • domain assumption Pretrained codecs (EnCodec, DAC) and WavLM representations are valid tokenizers and detokenizers for speech enhancement.
    The framework assumes codec tokens retain enough acoustic detail for enhancement and that the pretrained vocoders reconstruct speech faithfully.
  • domain assumption DNSMOS, CosSim, and dWER are reliable proxies for signal quality, speaker identity, and intelligibility.
    All conclusions are based on these metrics without human listening or confidence intervals, so the rankings depend on the metrics being trustworthy.
  • domain assumption Cross-entropy training with teacher forcing produces a language model whose outputs are suitable for beam search decoding.
    The AR model's benefit is evaluated through beam search after refinement, relying on standard language model assumptions about calibrated probabilities.
  • ad hoc to paper k-means quantization on WavLM layer 6 is a representative semantic tokenizer.
    The semantic tokenizer is trained by the authors for this study; the choice of layer and cluster count is a design decision that may not generalize to other semantic tokenizer configurations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Autoregressive Speech Enhancement via Acoustic Tokens." pith.science (2026). https://pith.science/paper/3OZOLYNQ

@misc{pith2026250712825,
  author       = {Pith},
  title        = {Pith review of: Autoregressive Speech Enhancement via Acoustic Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OZOLYNQ}},
  note         = {Machine review of arXiv:2507.12825}
}
read the original abstract

In speech processing pipelines, improving the quality and intelligibility of real-world recordings is crucial. While supervised regression is the primary method for speech enhancement, audio tokenization is emerging as a promising alternative for a smooth integration with other modalities. However, research on speech enhancement using discrete representations is still limited. Previous work has mainly focused on semantic tokens, which tend to discard key acoustic details such as speaker identity. Additionally, these studies typically employ non-autoregressive models, assuming conditional independence of outputs and overlooking the potential improvements offered by autoregressive modeling. To address these gaps we: 1) conduct a comprehensive study of the performance of acoustic tokens for speech enhancement, including the effect of bitrate and noise strength; 2) introduce a novel transducer-based autoregressive architecture specifically designed for this task. Experiments on VoiceBank and Libri1Mix datasets show that acoustic tokens outperform semantic tokens in terms of preserving speaker identity, and that our autoregressive approach can further improve performance. Nevertheless, we observe that discrete representations still fall short compared to continuous ones, highlighting the need for further research in this area.

Figures

Figures reproduced from arXiv: 2507.12825 by the authors.

Figure 1
Figure 1. Overview of our speech enhancement framework, consisting of a tokenizer, a language model, and a detokenizer. The language model is based on the proposed Speech Enhancement Transducer (SET) autoregressive architecture. model that transforms the sequence of noisy tokens into a sequence of enhanced tokens, and a detokenizer that converts the enhanced speech tokens back to the time domain into an enhanced waveform. B. … view at source ↗
Figure 2
Figure 2. (Left): the effect of bitrate on VoiceBank. (Right): the effect of noise strength on Libri1Mix. on high-bitrate tokens. For instance, other studies on discrete audio representations for speech enhancement and separation utilize thousands of hours of data [41], [60], whereas, in our case, we utilize no more than a hundred hours. Furthermore, while architectures like SepFormer are specifically designed for speech enha… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Absorbing Discrete Diffusion for Speech Enhancement

    cs.SD 2026-02 conditional novelty 5.0 of 10

    ADDSE performs speech enhancement by absorbing discrete diffusion over neural audio codec tokens, reaching competitive non-intrusive quality and low-SNR robustness in few sampling steps.

Reference graph

Works this paper leans on

60 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    MetricGAN-U: Unsupervised speech enhancement / dereverberation based only on noisy / reverberated speech,

    S.-W. Fu, C. Yu, K.-H. Hung, M. Ravanelli, and Y . Tsao, “MetricGAN-U: Unsupervised speech enhancement / dereverberation based only on noisy / reverberated speech,” in ICASSP, 2022, pp. 7412–7416

    work page 2022

  2. [2]

    MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement,

    S.-W. Fu, C.-F. Liao, Y . Tsao, and S.-D. Lin, “MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement,” in ICML, 2019

    work page 2019

  3. [3]

    MetricGAN+: An improved version of MetricGAN for speech enhancement,

    S.-W. Fu et al., “MetricGAN+: An improved version of MetricGAN for speech enhancement,” in Interspeech, 2021

    work page 2021

  4. [4]

    Reinforcement learning based speech enhancement for robust speech recognition,

    Y .-L. Shenet al., “Reinforcement learning based speech enhancement for robust speech recognition,” in ICASSP, 2019, pp. 6750–6754

    work page 2019

  5. [5]

    MetaRL-SE: a few-shot speech enhancement method based on meta-reinforcement learning,

    W. Zhou, R. Ji, and J. Lai, “MetaRL-SE: a few-shot speech enhancement method based on meta-reinforcement learning,” Multimedia Tools Appl., pp. 43 903–43 922, 2023

    work page 2023

  6. [6]

    Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in ICASSP, 2001, pp. 749–752

    work page 2001

  7. [7]

    An algorithm for intelligibility prediction of time–frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” IEEE Trans. Audio, Speech and Lang. Proc. , pp. 2125–2136, 2011

    work page 2011

  8. [8]

    Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , pp. 1256–1266, 2019

    work page 2019

Show all 60 references
  1. [9]

    Attention is all you need in speech separation,

    C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention is all you need in speech separation,” in ICASSP, 2021

    2021

  2. [10]

    Real time speech enhancement in the waveform domain,

    A. Defossez, G. Synnaeve, and Y . Adi, “Real time speech enhancement in the waveform domain,” in Interspeech, 2020, pp. 3291–3295

    2020

  3. [11]

    DNN-based enhancement of noisy and reverberant speech,

    Y . Zhao, D. Wang, and I. Merks, “DNN-based enhancement of noisy and reverberant speech,” in ICASSP, 2016, pp. 6525–6529

    2016

  4. [12]

    Phasen: A phase-and-harmonics-aware speech enhancement network,

    D. Yin, C. Luo, and Z. Xiong, “Phasen: A phase-and-harmonics-aware speech enhancement network,” in AAAI, 2020, pp. 9458–9465

    2020

  5. [13]

    Inter-SubNet: Speech enhancement with subband interaction,

    J. Chen et al. , “Inter-SubNet: Speech enhancement with subband interaction,” in ICASSP, 2023, pp. 1–5

    2023

  6. [14]

    Speech enhancement with score- based generative models in the complex STFT domain,

    S. Welker, J. Richter, and T. Gerkmann, “Speech enhancement with score- based generative models in the complex STFT domain,” in Interspeech, 2022, pp. 2928–2932

    2022

  7. [15]

    Speech enhancement and dereverberation with diffusion-based generative models,

    J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , pp. 2351–2364, 2023

    2023

  8. [16]

    Gemini: A family of highly capable multimodal models,

    Gemini Team, Google, “Gemini: A family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  9. [17]

    AudioLM: A language modeling approach to audio generation,

    Z. Borsos et al., “AudioLM: A language modeling approach to audio generation,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 31, pp. 2523–2533, 2023

    2023

  10. [18]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang et al. , “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111 , 2023

    2023 arXiv

  11. [19]

    AudioGen: Textually guided audio generation,

    F. Kreuk et al., “AudioGen: Textually guided audio generation,” in ICLR, 2023

    2023

  12. [20]

    Simple and controllable music generation,

    J. Copet et al., “Simple and controllable music generation,” in NeurIPS, 2023

    2023

  13. [21]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang et al., “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,” in EMNLP, 2023, pp. 15 757–15 773

    2023

  14. [22]

    SpiRit-LM: Interleaved spoken and written language model,

    T. A. Nguyen et al., “SpiRit-LM: Interleaved spoken and written language model,” arXiv preprint arXiv:2402.05755 , 2024

    2024 arXiv

  15. [23]

    NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju et al., “NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” arXiv preprint arXiv:2403.03100 , 2024

    2024 arXiv

  16. [24]

    CLam-TTS: Improving neural codec language model for zero-shot text-to-speech,

    J. Kim, K. Lee, S. Chung, and J. Cho, “CLam-TTS: Improving neural codec language model for zero-shot text-to-speech,” in ICLR, 2024

    2024

  17. [25]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in NeurIPS, 2020, pp. 12 449–12 460

    2020

  18. [26]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu et al. , “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , pp. 3451–3460, 2021

    2021

  19. [27]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al. , “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, pp. 1505–1518, 2022

    2022

  20. [28]

    Exploring speech recognition, translation, and under- standing with discrete speech units: A comparative study,

    X. Chang et al., “Exploring speech recognition, translation, and under- standing with discrete speech units: A comparative study,” in ICASSP, 2024, pp. 11 481–11 485

    2024

  21. [29]

    Towards universal speech discrete tokens: A case study for ASR and TTS,

    Y . Yanget al., “Towards universal speech discrete tokens: A case study for ASR and TTS,” in ICASSP, 2024, pp. 10 401–10 405

    2024

  22. [30]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” TMLR, 2023

    2023

  23. [31]

    High- fidelity audio compression with improved RVQGAN,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- fidelity audio compression with improved RVQGAN,” in NeurIPS, 2023

    2023

  24. [32]

    FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,

    Z. Du, S. Zhang, K. Hu, and S. Zheng, “FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,” arXiv preprint arXiv:2309.07405 , 2023

    2023 arXiv

  25. [33]

    HiFi-Codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang et al., “HiFi-Codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765 , 2023

    2023 arXiv

  26. [34]

    Language-Codec: Reducing the gaps between discrete codec representation and speech language models,

    S. Ji et al. , “Language-Codec: Reducing the gaps between discrete codec representation and speech language models,” arXiv preprint arXiv:2402.12208, 2024

    2024 arXiv

  27. [35]

    WavTokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,

    ——, “WavTokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,” arXiv preprint arXiv:2408.16532 , 2024

    2024 arXiv

  28. [36]

    Neural discrete representation learning,

    A. Van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in NeurIPS, 2017

    2017

  29. [37]

    SoundStream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , pp. 495–507, 2021

    2021

  30. [38]

    A comparison of discrete and soft speech units for improved voice conversion,

    B. van Niekerk et al., “A comparison of discrete and soft speech units for improved voice conversion,” in ICASSP, 2022

    2022

  31. [39]

    SpeechTokenizer: Unified speech tokenizer for speech large language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “SpeechTokenizer: Unified speech tokenizer for speech large language models,” in ICLR, 2024

    2024

  32. [40]

    SemantiCodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu et al., “SemantiCodec: An ultra low bitrate semantic audio codec for general sound,” arXiv preprint arXiv:2405.00233 , 2024

    2024 arXiv

  33. [41]

    SELM: Speech enhancement using discrete tokens and language models,

    Z. Wang et al., “SELM: Speech enhancement using discrete tokens and language models,” in ICASSP, 2024, pp. 11 561–11 565

    2024

  34. [42]

    How should we extract discrete audio tokens from self-supervised models?

    P. Mousavi et al., “How should we extract discrete audio tokens from self-supervised models?” in Interspeech, 2024

    2024

  35. [43]

    DASB - discrete audio and speech benchmark,

    ——, “DASB - discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294, 2024

    2024 arXiv

  36. [44]

    Sequence transduction with recurrent neural networks,

    A. Graves, “Sequence transduction with recurrent neural networks,” in ICML Workshop on Representation Learning , 2012

    2012

  37. [45]

    Exposure bias versus self- recovery: Are distortions really incremental for autoregressive text generation?

    T. He, J. Zhang, Z. Zhou, and J. Glass, “Exposure bias versus self- recovery: Are distortions really incremental for autoregressive text generation?” in EMNLP, 2021, pp. 5087–5102

    2021

  38. [46]

    Attention is all you need,

    A. Vaswani et al. , “Attention is all you need,” in NeurIPS, 2017, pp. 6000–6010

    2017

  39. [47]

    Investi- gating RNN-based speech enhancement methods for noise-robust text-to- speech,

    C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Investi- gating RNN-based speech enhancement methods for noise-robust text-to- speech,” in Speech Synthesis Workshop, 2016, pp. 146–152

    2016

  40. [48]

    LibriMix: An open-source dataset for generalizable speech separation,

    J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, “LibriMix: An open-source dataset for generalizable speech separation,” arXiv preprint arXiv:2005.11262 , 2020

    2005 arXiv

  41. [49]

    Librispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in ICASSP, 2015, pp. 5206–5210

    2015

  42. [50]

    WHAM!: Extending speech separation to noisy environments,

    G. Wichern et al. , “WHAM!: Extending speech separation to noisy environments,” in Interspeech, 2019

    2019

  43. [51]

    V ocos: Closing the gap between time-domain and fourier- based neural vocoders for high-quality audio synthesis,

    H. Siuzdak, “V ocos: Closing the gap between time-domain and fourier- based neural vocoders for high-quality audio synthesis,” arXiv preprint arXiv:2306.00814, 2023

    2023 arXiv

  44. [52]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech, 2020, pp. 5036–5040

    2020

  45. [53]

    HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis,” in NeurIPS, 2020

    2020

  46. [54]

    SDR–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR–half-baked or well done?” in ICASSP, 2019, pp. 626–630

    2019

  47. [55]

    DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP, 2022

    2022

  48. [56]

    Sequential multi-frame neural beamforming for speech separation and enhancement,

    Z.-Q. Wang et al., “Sequential multi-frame neural beamforming for speech separation and enhancement,” in IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 905–911

    2021

  49. [57]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al. , “Robust speech recognition via large-scale weak supervision,” arXiv preprint arXiv:2212.04356 , 2022

    2022 arXiv

  50. [58]

    SpeechBrain: A general-purpose speech toolkit,

    M. Ravanelli et al., “SpeechBrain: A general-purpose speech toolkit,” arXiv preprint arXiv:2106.04624 , 2021

    2021 arXiv

  51. [59]

    Open-source conversational AI with SpeechBrain 1.0,

    ——, “Open-source conversational AI with SpeechBrain 1.0,” arXiv preprint arXiv:2407.00463, 2024

    2024 arXiv

  52. [60]

    TokenSplit: Using discrete speech representations for direct, refined, and transcript-conditioned speech separation and recognition,

    H. Erdogan et al., “TokenSplit: Using discrete speech representations for direct, refined, and transcript-conditioned speech separation and recognition,” in Interspeech, 2023, pp. 3462–3466

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.