Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

GenTSE, a two-stage generative language model for target speaker extraction, outperforms previous LM-based systems by first predicting coarse semantic tokens and then generating fine acoustic tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:15 UTC pith:OMTSAE5T

load-bearing objection Solid two-stage generative LM for TSE with clean ablations; the DPO gain is real but partly an artifact of using UTMOS as both reward and metric. the 3 major comments →

arxiv 2512.20978 v2 pith:OMTSAE5T submitted 2025-12-24 eess.AS cs.AIcs.LG

GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

Haoyang Li , Xuyi Zhuang , Azmat Adnan , Ye Ni , Wei Rao , Shreyas Gopal , Eng Siong Chng , Boon Siew Han
show 1 more author
Yuanjin Zheng
This is my paper
classification eess.AS cs.AIcs.LG
keywords target speaker extractiongenerative language modeltwo-stage decodingsemantic tokensacoustic tokensdirect preference optimizationexposure biasLibri2Mix
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GenTSE recasts target speaker extraction (TSE) as a generative problem: a decoder-only language model first predicts coarse semantic tokens of the target speaker's speech, then a second language model generates fine acoustic tokens conditioned on those semantics. The paper argues that this coarse-to-fine split stabilizes decoding and yields more faithful target speech than single-stage codec-token prediction, and that conditioning on continuous speech and codec embeddings preserves more speaker information than discretized prompts. Two training techniques carry the empirical gains: Frozen-LM Conditioning, which exposes the models to their own earlier predictions to close the teacher-forcing/autoregressive gap, and Direct Preference Optimization, which biases the acoustic model toward outputs rated higher by a DNN-based MOS predictor. On the Libri2Mix benchmark, GenTSE reports the best speech quality, intelligibility, and speaker similarity among LM-based TSE systems, with DNSMOS 3.656 and UTMOS 4.135. The deeper claim is that fully generative language-model pipelines, rather than discriminative mappings, are a viable path to high-fidelity target speaker extraction.

Core claim

The paper's central claim is that separating TSE into a semantic coarse-token stage and an acoustic fine-token stage, both handled by decoder-only LMs conditioned on continuous embeddings, produces higher-quality and more intelligible target speech than end-to-end discriminative models or single-stage LM-based generative models. Stage-1 quantizes self-supervised speech embeddings (WavLM layer 6) with k-means and predicts those tokens; Stage-2 predicts single-codebook neural codec tokens (SimCodec) conditioned on the semantic tokens and on continuous codec embeddings of mixture and reference. The paper also introduces Frozen-LM Conditioning (FLC) to reduce exposure bias by training fresh mode

What carries the argument

The central object is a two-stage decoder-only language-model hierarchy: a semantic extraction LM that autoregressively predicts coarse semantic tokens (k-means-quantized WavLM embeddings) of the target speech, and an acoustic generation LM that autoregressively predicts single-codebook SimCodec acoustic tokens conditioned on those semantic tokens. Both LMs are conditioned on continuous frame-level embeddings — WavLM embeddings for the semantic stage and DAC embeddings for the acoustic stage — from reference and mixture, which the paper argues carry richer speaker context than discretized prompts. Two optimization mechanisms carry the argument: Frozen-LM Conditioning (FLC), which fine-tunes

Load-bearing premise

The perceptual-quality gains from DPO rest on the assumption that the DNN-based MOS predictor used to select preference pairs (UTMOS) is an unbiased proxy for human perception, since the same predictor also provides the reported quality evidence.

What would settle it

Run a paired listening study on the Libri2Mix test set comparing GenTSE trained with DPO-only (Exp 8), FLC-only (Exp 6), and teacher-forcing (Exp 5). If human raters do not consistently prefer the DPO outputs, or prefer them far less strongly than the reported UTMOS gap (4.161 vs 4.111 vs 4.099) implies, the claim that DPO improves perceptual quality fails. Alternatively, select preference pairs with UTMOS but evaluate on a held-out MOS predictor or human ratings; if the quality advantage disappears, the gains are metric-fitting.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If GenTSE's results hold, fully generative TSE is achievable without deterministic or non-generative post-processing components, and a two-stage semantic-then-acoustic hierarchy is a stronger design than single-stage codec-token prediction.
  • Continuous SSL/codec embeddings as LM conditioning preserve more usable target-speaker information than discretized reference prompts, which could improve generalization when reference prompts are short or noisy.
  • FLC meaningfully reduces exposure bias: the paper reports dWER dropping from 0.217 to 0.172 and a shrinking teacher-forcing vs. autoregressive accuracy gap, so LM-based TSE systems should incorporate some form of scheduled-sampling training.
  • DPO lifts perceptual quality metrics after only 400 fine-tuning steps without changing the inference architecture, showing preference alignment transfers to TSE.
  • The paper's ablations show the full setup trades a small amount of intelligibility and speaker similarity for quality when DPO alone is used, so preference signals other than a single MOS predictor may be needed to avoid that trade-off.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The DPO results may partly reflect fitting the UTMOS predictor rather than genuine human preference, since UTMOS is used both to select preference pairs and as the headline quality metric; a human listening test would determine whether the reported quality gap is perceptual or metric-fitting.
  • The same coarse-to-fine LM recipe should transfer to related speaker-conditioned generation tasks — speech enhancement, voice conversion, and codec-based speech restoration — if the two-step decomposition behaves as a general inductive bias.
  • The quality-intelligibility trade-off observed under pure DPO suggests a multi-metric or human-feedback preference signal, rather than a single MOS proxy, could preserve intelligibility while keeping quality gains — an extension the paper itself flags.
  • Because continuous embeddings beat discretized reference prompts in the ablations, future LM-based TSE systems may drop codec-token prompting entirely and feed frame-level continuous features directly into the decoder-only LMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. GenTSE proposes a two-stage decoder-only generative language model for target speaker extraction. Stage 1 autoregressively predicts k-means-discretized WavLM semantic tokens of the target speech conditioned on continuous WavLM embeddings of the reference and mixture; Stage 2 generates SimCodec acoustic tokens conditioned on the predicted semantic tokens and continuous DAC embeddings. The paper introduces Frozen-LM Conditioning (FLC) to reduce exposure bias, and applies Direct Preference Optimization (DPO) to the acoustic stage using UTMOS-ranked candidate pairs. Experiments on Libri2Mix report improvements over prior LM-based TSE systems in DNSMOS, UTMOS, NISQA, SECS, dWER, and SpeechBERT. The central claim is that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.

Significance. The architectural idea is timely and the ablation evidence for the two-stage design is meaningful: the semantic stage reduces dWER from 0.284 to 0.217 (Table 2, Exp. 1 vs. 5), and FLC further reduces dWER to 0.172 (Table 3, Exp. 5 vs. 6). These dWER gains are not confounded by the DPO reward, since Whisper-based dWER is not used as a training signal. If the FLC mechanism is correctly implemented and the DPO quality claim were validated against human perception, GenTSE would be a solid contribution to LM-based generative TSE. However, the paper's headline perceptual-quality claim rests on circular DNN-MOS evaluation, and the FLC equations as written do not match the described mechanism. The comparison with prior work is also not fully controlled. The contribution is therefore plausible but not yet established at the level claimed.

major comments (3)
  1. [Section 2.3, Eq. (5); Table 3, Exps. 8-9] The DPO evaluation is circular. Preference pairs (A+, A-) are selected by UTMOS (Section 2.3), and UTMOS is then reported as the headline quality metric in Table 3. The Exp. 8 gains in UTMOS (4.161 vs. 4.111) and the correlated DNSMOS/NISQA gains may therefore reflect fitting the UTMOS predictor rather than genuine perceptual improvement. The paper itself acknowledges that preference pairs are selected solely by UTMOS and that this causes drops in SECS and intelligibility (Section 4). This is a load-bearing issue for the central claim of improved speech quality. The authors should add a human listening test, or at least a held-out preference/reward signal that was not used for candidate selection, and should report a non-correlated objective anchor such as PESQ or SI-SNR despite their caveats.
  2. [Section 2.2, Eqs. (3)-(4)] As written, the FLC objective does not implement scheduled sampling. In Eq. (3) the target is \bar{s}'_t, defined in the text as a token generated by the trainable model \theta', rather than the ground-truth token s_t. Maximizing the likelihood of the model's own generated tokens is a different and potentially degenerate objective. In Eq. (4), the acoustic LM is conditioned on the ground-truth semantic sequence S instead of the predicted \bar{S}, so the acoustic stage is not exposed to semantic-stage prediction errors during FLC, undercutting the stated exposure-bias mechanism. Please correct the equations or clarify the intended training target; this is essential for interpreting the dWER gain attributed to FLC.
  3. [Section 3.4, Table 1] The headline comparison against prior LM-based systems is not controlled. LLaSE-G1 and Metis are evaluated using official checkpoints trained on substantially larger datasets, and X-TF-GridNet is an author-provided checkpoint trained on WHAMR!, not Libri2Mix; only TSELM-L and USEF-SepFormer are retrained on the paper's training data. The reported margins in Table 1 could reflect training-data scale rather than architectural advantage. In addition, no confidence intervals or significance tests are reported. The discrepancy between Table 2 Exp. 5 (UTMOS 4.065) and Table 3 Exp. 5 (UTMOS 4.099) for the same configuration suggests evaluation noise that is not quantified. Please retrain matched baselines or clearly restrict the claim to 'outperforms published numbers', and report variance or significance for the key comparisons.
minor comments (4)
  1. [Tables 2 and 3] Exp. 5 is described as the final configuration in both tables, but UTMOS is 4.065 in Table 2 and 4.099 in Table 3, with dWER and SECS identical. Clarify whether these are different checkpoints, different evaluation runs, or a typo.
  2. [Section 2.3 / Abstract] The phrase 'human perceptual preferences' (Abstract, Section 2.3) is stronger than what is implemented: the preference pairs are ranked by UTMOS, a DNN MOS predictor. Recommend using 'proxy-MOS preference' or similar throughout.
  3. [Section 3.3 / inference] The inference decoding procedure is not specified. The DPO candidates use top-k=16 and M=32, but it is unclear whether the final evaluations use greedy decoding, sampling, or beam search. This affects all reported metrics and should be stated.
  4. [Figure 2] The caption should specify the validation set, the number of utterances, and the definition of the teacher-forcing vs. autoregressive accuracy curves. Without this, the figure is hard to interpret.

Circularity Check

1 steps flagged

DPO quality gain is circular: UTMOS selects preference pairs and UTMOS is the reported outcome; no human anchor.

specific steps
  1. fitted input called prediction [Section 2.3 (Eq. 5), Section 3.2, Table 3 Exps 6-9; concession in Section 4]
    "Each candidate is then decoded by the SimCodec decoder, and evaluated using UTMOS [27], a DNN-based MOS predictor strongly correlated with human judgments. The candidate with the highest UTMOS score is selected as A+, while the one with the lowest score becomes A−."

    DPO constructs its preference pairs by ranking SimCodec-decoded candidates with UTMOS (Section 2.3), then optimizes Eq. (5) to raise the likelihood of the top-UTMOS candidate. The paper's headline evidence for the resulting quality gain is the same UTMOS metric (Tables 1 and 3). Thus the DPO-induced UTMOS improvement (Exp 8: 4.161 vs 4.111; Exp 9: 4.135) has a by-construction component: the model is trained on the very predictor used to score it. The paper even concedes 'preference pairs are selected solely by UTMOS' and concludes using 'proxy-MOS feedback,' so the perceptual-quality claim for DPO reduces to metric-fitting without a human listening anchor.

full rationale

The architecture, FLC, and cross-entropy baselines are independently evaluated and do not reduce to circularity: FLC's dWER improvement (0.217 to 0.172) and the head-to-head comparisons in Table 1 have content beyond the DPO reward loop. However, the DPO contribution is advertised as aligning outputs with 'human perceptual preferences,' yet both the reward signal and the reported evaluation are UTMOS, a DNN MOS predictor. Any UTMOS gain from DPO is therefore partly forced by training on the evaluation metric; DNSMOS and NISQA are correlated predictors rather than exact rewards, so they provide partial but not independent confirmation. The self-citation [20] for the DPO sampling recipe is not load-bearing because DPO itself comes from [17] and the experiments are reproducible in principle. The absence of PESQ/SI-SNR and any human listening test increases the risk that the quality claim rests on the circular UTMOS loop, but the central system claim retains independent content, warranting a partial-circularity score rather than a higher one.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on pretrained speech representations (WavLM, DAC, SimCodec) and on UTMOS serving as a perceptual oracle; these are adopted as given from prior work. No new entities are invented; no closed-form derivation is attempted. The two-stage hierarchy and FLC/DPO recipes are engineering choices validated only on Libri2Mix.

free parameters (5)
  • WavLM k-means cluster count K = 1024
    Chosen by hand (Section 3.3); sets coarse semantic token granularity; not derived from data in this paper.
  • WavLM layer depth = layer 6
    Chosen for semantic feature extraction (Section 2.1.1); no layer-ablation justification is provided.
  • DPO beta = 0.1
    Eq. (5) / Section 3.3; hand-tuned scalar controlling preference strength.
  • DPO sampling budget (top-k, M) = k=16, M=32
    Section 3.3; candidate generation hyperparameters; no sensitivity analysis reported.
  • Training step counts = semantic 64k, acoustic 78k, FLC 12k/6k, DPO 400
    Section 3.3; stopping rules based on validation convergence; not tied to a principled criterion.
axioms (5)
  • domain assumption UTMOS is a valid stand-in for human perceptual preference (strong correlation with human judgments).
    Load-bearing for DPO: preference pairs are ranked by UTMOS (Section 2.3) and the same metric is reported as a quality outcome (Table 3); if UTMOS disagrees with human listening, the 'perceptual alignment' claim weakens.
  • domain assumption WavLM layer-6 frame embeddings, after k-means discretization (K=1024), are sufficient semantic tokens of the target speech.
    Section 2.1.1 defines the semantic LM target via these tokens; the two-stage design assumes no information needed for target content is lost.
  • domain assumption SimCodec's single-codebook tokens and DAC continuous embeddings preserve the acoustic detail needed to reconstruct target speech.
    Section 2.1.2 uses SimCodec as acoustic tokenizer and DAC embeddings as condition; output quality is bounded by these pretrained codecs.
  • domain assumption FLC with frozen-model predictions as conditioning remains stable and converges at the stated step counts.
    Section 2.2 / Fig. 2 shows the teacher-forcing vs autoregressive gap closing for the semantic LM, but stability is only empirically probed on Libri2Mix.
  • standard math Standard cross-entropy and DPO losses are statistically well-founded objectives for this setting.
    Background ML results, Eqs. (1)-(5); not derived in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 8346 in / 17542 out tokens · 159128 ms · 2026-08-03T14:15:14.334386+00:00 · methodology

0 comments
read the original abstract

Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We propose GenTSE, a two-stage decoder-only generative LM for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more accurate target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further apply DPO to better align outputs with perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.

Figures

Figures reproduced from arXiv: 2512.20978 by Azmat Adnan, Boon Siew Han, Eng Siong Chng, Haoyang Li, Shreyas Gopal, Wei Rao, Xuyi Zhuang, Ye Ni, Yuanjin Zheng.

Figure 1
Figure 1. Figure 1: The overall architecture of GenTSE consisting of a semantic extraction stage (left) and an acoustic generation stage (right) 2.1. Hierarchical Modeling 2.1.1. Semantic extraction stage Given a reference and mixture speech, WavLM [22] serves as the high level feature extractor to extract the corresponding frame-level embeddings from layer 6, denoted as W avlmref , W avlmmix ∈ R T ×H, where T and H are the t… view at source ↗
Figure 2
Figure 2. Figure 2: Semantic LM validation top-1 accuracy during FLC: blue = teacher-forcing, green = autoregressive [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech

    cs.SD 2026-06 unverdicted novelty 6.0

    A code-mixing guided preference-learning method for TTS produces synthetic data that lowers mixed error rate when fine-tuning Whisper on the SEAME Mandarin-English corpus.

Reference graph

Works this paper leans on

40 extracted references · 18 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Introduction Target Speaker Extraction (TSE) aims to recover the voice of a designated speaker from a multi-speaker mixture by condition- ing the extraction process on auxiliary information that uniquely characterizes the target speaker [1]. Most existing TSE meth- ods rely on discriminative, supervised models that learn a direct mapping from the mixture ...

  2. [2]

    G” and “D

    Methodology Fig. 1 illustrates the proposed GenTSE framework, which com- prises a semantic extraction stage followed by an acoustic gen- eration stage. The semantic stage autoregressively predicts coarse semantic tokens for the target speaker, and the acous- tic stage autoregressively generates fine-grained acoustic tokens conditioned on these predictions...

    Pith/arXiv arXiv 2025

  3. [3]

    Experiments 3.1. Dataset We train all LMs using the train-100 and train-360 splits of the clean LibriMix2spk dataset [21], a standard benchmark for TSE, and conduct evaluation on the corresponding test set. For inference, a reference utterance of the target speaker is ran- domly chosen and used consistently in all evaluations. WavLM (WavLM-Large), DAC1 an...

    arXiv

  4. [4]

    GenTSE surpasses prior LM- based baselines in speech quality, speaker similarity, and in- telligibility, demonstrating clear advantage among LM-driven methods

    Results and discussion Table 1 compares GenTSE against recent generative LM-based and discriminative TSE models. GenTSE surpasses prior LM- based baselines in speech quality, speaker similarity, and in- telligibility, demonstrating clear advantage among LM-driven methods. When compared with leading discriminative methods, GenTSE achieves higher perceptual...

  5. [5]

    The semantic stage predicts k-means-discretized WavLM tokens of the target speech, and the acoustic stage generates neural audio codec to- kens conditioned on these predictions

    Conclusion We present GenTSE, a fully generative TSE framework based on a two-stage decoder-only LM architecture. The semantic stage predicts k-means-discretized WavLM tokens of the target speech, and the acoustic stage generates neural audio codec to- kens conditioned on these predictions. This two-stage design reduces modeling complexity compared to dir...

  6. [6]

    Neural target speech extraction: An overview,

    K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. ˇCernock`y, and D. Yu, “Neural target speech extraction: An overview,”IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023

    2023

  7. [7]

    Spex: Multi-scale time domain speaker extraction network,

    C. Xu, W. Rao, E. S. Chng, and H. Li, “Spex: Multi-scale time domain speaker extraction network,”IEEE/ACM transactions on audio, speech, and language processing, vol. 28, pp. 1370–1384, 2020

    2020

  8. [8]

    Spex+: A complete time domain speaker extraction network,

    M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “Spex+: A complete time domain speaker extraction network,” arXiv preprint arXiv:2005.04686, 2020

    Pith/arXiv arXiv 2005

  9. [9]

    Mc-spex: Towards effective speaker extraction with multi-scale interfusion and conditional speaker modulation,

    J. Chen, W. Rao, Z. Wang, J. Lin, Y . Ju, S. He, Y . Wang, and Z. Wu, “Mc-spex: Towards effective speaker extraction with multi-scale interfusion and conditional speaker modulation,” arXiv preprint arXiv:2306.16250, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Hierarchical speaker representation for target speaker extrac- tion,

    S. He, H. Zhang, W. Rao, K. Zhang, Y . Ju, Y . Yang, and X. Zhang, “Hierarchical speaker representation for target speaker extrac- tion,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 361–10 365

    2024

  11. [11]

    Usef-tse: Universal speaker embedding free target speaker extraction,

    B. Zeng and M. Li, “Usef-tse: Universal speaker embedding free target speaker extraction,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  12. [12]

    Solospeech: Enhanc- ing intelligibility and quality in target speech extraction through a cascaded generative pipeline,

    H. Wang, J. Hai, D. Yang, C. Chen, K. Li, J. Peng, T. Thebaud, L. M. Velazquez, J. Villalba, and N. Dehak, “Solospeech: Enhanc- ing intelligibility and quality in target speech extraction through a cascaded generative pipeline,”arXiv preprint arXiv:2505.19314, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement,

    J. Zhang, J. Yang, Z. Fang, Y . Wang, Z. Zhang, Z. Wang, F. Fan, and Z. Wu, “Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement,”arXiv preprint arXiv:2501.15417, 2025

    arXiv 2025

  14. [14]

    Llase-g1: Incentivizing gener- alization capability for llama-based speech enhancement,

    B. Kang, X. Zhu, Z. Zhang, Z. Ye, M. Liu, Z. Wang, Y . Zhu, G. Ma, J. Chen, L. Xiaoet al., “Llase-g1: Incentivizing gener- alization capability for llama-based speech enhancement,”arXiv preprint arXiv:2503.00493, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Unise: A uni- fied framework for decoder-only autoregressive lm-based speech enhancement,

    H. Yan, C. Liu, S. Xue, X. Liang, and Z. Xue, “Unise: A uni- fied framework for decoder-only autoregressive lm-based speech enhancement,”arXiv preprint arXiv:2510.20441, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Tselm: Target speaker extrac- tion using discrete tokens and language models,

    B. Tang, B. Zeng, and M. Li, “Tselm: Target speaker extrac- tion using discrete tokens and language models,”arXiv preprint arXiv:2409.07841, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    LauraTSE: Target speaker extrac- tion using auto-regressive decoder-only language models,

    B. Tang, B. Zeng, and M. Li, “LauraTSE: Target speaker extrac- tion using auto-regressive decoder-only language models,”arXiv preprint arXiv:2504.07402, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Speechx: Neural codec language model as a versatile speech transformer,

    X. Wang, M. Thakker, Z. Chen, N. Kanda, S. E. Eskimez, S. Chen, M. Tang, S. Liu, J. Li, and T. Yoshioka, “Speechx: Neural codec language model as a versatile speech transformer,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 32, pp. 3355–3364, 2024

    2024

  19. [19]

    Metis: A foundation speech generation model with masked gen- erative pre-training,

    Y . Wang, J. Zheng, J. Zhang, X. Zhang, H. Liao, and Z. Wu, “Metis: A foundation speech generation model with masked gen- erative pre-training,”arXiv preprint arXiv:2502.03128, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Scheduled sampling for trans- formers,

    T. Mihaylova and A. F. Martins, “Scheduled sampling for trans- formers,”arXiv preprint arXiv:1906.07651, 2019

    Pith/arXiv arXiv 1906

  21. [21]

    Scheduled sam- pling based on decoding steps for neural machine translation,

    Y . Liu, F. Meng, Y . Chen, J. Xu, and J. Zhou, “Scheduled sam- pling based on decoding steps for neural machine translation,” arXiv preprint arXiv:2108.12963, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in neural informa- tion processing systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  23. [23]

    X-sepformer: End-to-end speaker extraction network with explicit optimization on speaker confusion,

    K. Liu, Z. Du, X. Wan, and H. Zhou, “X-sepformer: End-to-end speaker extraction network with explicit optimization on speaker confusion,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  24. [24]

    Neu- ral speech extraction with human feedback,

    M. Itani, A. Graves, S. E. Eskimez, and S. Gollakota, “Neu- ral speech extraction with human feedback,”arXiv preprint arXiv:2508.03041, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Aligning generative speech enhancement with human preferences via direct preference optimization,

    H. Li, N. Hou, Y . Hu, J. Yao, S. M. Siniscalchi, and E. S. Chng, “Aligning generative speech enhancement with human preferences via direct preference optimization,”arXiv preprint arXiv:2507.09929, 2025

    arXiv 2025

  26. [26]

    Librimix: An open-source dataset for generalizable speech separation,

    J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vin- cent, “Librimix: An open-source dataset for generalizable speech separation,”arXiv preprint arXiv:2005.11262, 2020

    Pith/arXiv arXiv 2005

  27. [27]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  28. [28]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 27 980–27 993, 2023

    2023

  29. [29]

    Gense: Generative speech enhancement via language models using hier- archical modeling,

    J. Yao, H. Liu, C. Chen, Y . Hu, E. Chng, and L. Xie, “Gense: Generative speech enhancement via language models using hier- archical modeling,”arXiv preprint arXiv:2502.02942, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    X-tf-gridnet: A time–frequency do- main target speaker extraction network with adaptive speaker em- bedding fusion,

    F. Hao, X. Li, and C. Zheng, “X-tf-gridnet: A time–frequency do- main target speaker extraction network with adaptive speaker em- bedding fusion,”Information Fusion, vol. 112, p. 102550, 2024

    2024

  31. [31]

    Generalization in generation: A closer look at expo- sure bias,

    F. Schmidt, “Generalization in generation: A closer look at expo- sure bias,”arXiv preprint arXiv:1910.00292, 2019

    Pith/arXiv arXiv 1910

  32. [32]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,”arXiv preprint arXiv:2204.02152, 2022

    Pith/arXiv arXiv 2022

  33. [33]

    Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6493–6497

    2021

  34. [34]

    Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,

    G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,”arXiv preprint arXiv:2104.09494, 2021

    Pith/arXiv arXiv 2021

  35. [35]

    Speechbertscore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,

    T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “Speechbertscore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,”arXiv preprint arXiv:2401.16812, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  37. [37]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  38. [38]

    Selm: Speech enhancement using discrete tokens and language models,

    Z. Wang, X. Zhu, Z. Zhang, Y . Lv, N. Jiang, G. Zhao, and L. Xie, “Selm: Speech enhancement using discrete tokens and language models,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 561–11 565

    2024

  39. [39]

    Whamr!: Noisy and reverberant single-channel speech separa- tion,

    M. Maciejewski, G. Wichern, E. McQuinn, and J. Le Roux, “Whamr!: Noisy and reverberant single-channel speech separa- tion,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 696–700

    2020

  40. [40]

    Multi- metric preference alignment for generative speech restoration,

    J. Zhang, X. Zhang, J. Yang, Y . Wang, F. Fan, and Z. Wu, “Multi- metric preference alignment for generative speech restoration,” arXiv preprint arXiv:2508.17229, 2025

    arXiv 2025