Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Natural-language quality descriptions from an audio LLM can act as a reinforcement-learning reward for audio-visual speech enhancement, beating scalar-metric rewards on both objective and subjective tests.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:11 UTC pith:TFBRISTN

load-bearing objection Genuinely new RL-reward recipe for AVSE, but the evidence is too thin to carry the central claim without a revision. the 4 major comments →

arxiv 2603.13952 v3 pith:TFBRISTN submitted 2026-03-14 cs.SD cs.AIeess.AS

LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement

Chih-Ning Chen , Jen-Cheng Hou , Hsin-Min Wang , Shao-Yi Chien , Yu Tsao , Fan-Gang Zeng This is my paper
classification cs.SD cs.AIeess.AS
keywords audio-visual speech enhancementreinforcement learninglarge language model rewardinterpretable rewardspeech qualityproximal policy optimizationsentiment analysisDNSMOS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that reinforcement-learning fine-tuning of an audio-visual speech enhancer can be guided by an LLM's natural-language quality assessments, and that these textual feedback signals outperform scalar reward signals. The authors report that converting LLM descriptions into sentiment scores and using the score change as a PPO reward improves PESQ, STOI, neural quality metrics, and subjective listener preference over both a supervised baseline and an RL baseline using a scalar predicted-MOS reward. This matters because conventional objectives like SI-SNR correlate imperfectly with perception and provide no explanation of what improved. If the claim holds, natural-language evaluation becomes a practical, interpretable training signal for speech enhancement.

Core claim

The paper's central claim is that descriptive text from an audio LLM, converted by a sentiment model into a 1–5 score, can serve as a reward for reinforcement-learning fine-tuning of an audio-visual speech enhancer and yields better results than scalar metric rewards. The reward is the score difference between the RL output and the base output, which stabilizes training and removes the need for a critic network. The policy is updated with a PPO clipping objective plus the original SI-SNR loss. On the AVSEC-4 test set the method reports the best PESQ (1.57), STOI (0.58), NISQA MOS (1.29), VQscore (0.62), and S-BERT (0.57), and wins 96.7% of A/B preferences over the pretrained baseline and 67.

What carries the argument

The machinery is a reward chain: an audio LLM generates free-text assessments of the enhanced waveform, a sentiment analysis model converts each assessment into a 1–5 score, and the reward is the relative score between the RL output and the base output. Gaussian noise is injected into the predicted time-domain mask to make the deterministic enhancer stochastic enough for policy-gradient updates. Training combines a PPO clipping loss with the original SI-SNR loss, with a KL term controlling deviation from the base policy. The LLM and sentiment model are frozen, keeping the reward criterion fixed during fine-tuning.

Load-bearing premise

The load-bearing premise is that sentiment scores assigned to an LLM's fixed-pattern text descriptions order speech outputs by perceived quality well enough to guide thousands of policy updates, yet no correlation of this reward with human ratings or standard quality metrics is reported.

What would settle it

On a held-out set of enhanced utterances, compute the rank correlation between the LLM-plus-sentiment reward and human MOS or PESQ per utterance. If the correlation is near zero or negative, the reward cannot be the driver of the reported gains; likewise, eliminating the LLM reward while keeping PPO noise and the SI-SNR loss should remove most of the improvements if the LLM signal is causal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, LLM-based textual rewards can serve as a general perceptual training signal for audio-visual speech enhancement, replacing or complementing scalar metrics.
  • The relative-reward design removes the need for a learned critic, simplifying RL fine-tuning to a clipped policy update plus a stability term.
  • Because the reward is tied to interpretable text, each training improvement becomes inspectable: one can read why a sample's score rose, such as reduced distortion versus reduced noise.
  • The framework could be applied to other speech-processing objectives, such as dereverberation or separation, where perceptual quality descriptions are meaningful.
  • The retained SI-SNR loss suggests perceptual LLM rewards are best used as a refinement layer on top of a signal-preserving objective, not as a standalone objective.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported gains are not isolated from the SI-SNR term: an ablation that fine-tunes with the same PPO noise and SI-SNR loss but a random or constant reward is needed to attribute the improvement to the LLM's semantic content.
  • Since the paper admits the LLM descriptions fall into fixed patterns, the reward may currently behave like a coarse quality classifier; using more diverse prompts or stronger LLMs could either sharpen the signal or reveal a ceiling.
  • The single illustrative example in the interpretability analysis is anecdotal; a systematic per-utterance correlation between the LLM-based reward and human MOS would test whether the reward genuinely aligns with perception across the dataset.
  • If the approach is replicated, the natural-language channel could be adapted to personal preference—for example, different descriptions of desired quality could steer enhancement toward individual listeners' tastes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LR-AVSE, an RL fine-tuning framework for audio-visual speech enhancement. A frozen SALMONN audio-LLM generates natural-language descriptions of the enhanced speech, which are converted by BERT sentiment analysis into a 1–5 score (Eq. 7). This score, used as a relative reward against the base policy (Eq. 6), is combined with a clipped PPO objective and the original SI-SNR loss (Eq. 11) to fine-tune a pretrained AVSE model. Experiments on the AVSEC-4 dataset compare the method against a supervised baseline and an RL baseline using DNSMOS as reward, reporting PESQ, STOI, NISQA, VQscore, S-BERT, and an A/B listening test.

Significance. The idea of converting LLM-generated descriptive evaluations into reward signals for AVSE is novel and timely, and the design choices are partly reassuring: the reward model is a frozen, externally released component, and the relative-reward form plus a shared DNSMOS-based RL baseline reduce some circularity. If the reward chain were validated and the results replicated, the framework would offer a meaningful interpretability advantage over scalar-reward RL. However, the current evidence is not sufficient to establish the central claim that natural-language semantic content drives the improvement; several load-bearing validations are missing and the reported margins are small.

major comments (4)
  1. [§2.3, Eqs. (6)–(7)] The reward chain r_phi(yhat)=Sentiment(LLM(yhat)) is the sole mechanism by which natural-language feedback guides PPO, but no calibration is reported. No correlation of r_phi with PESQ, STOI, NISQA, or human ratings is given; Fig. 3 is a single utterance, and §5 concedes the descriptions fall into fixed patterns. Because Eq. (6) subtracts the base-policy score and Eq. (11) retains the SI-SNR loss at gamma=1.0, the reported gains could partly come from continued supervised training or reward noise. Please report reward–metric correlations on the validation set and include an ablation with a random reward matched in mean/variance, or a reward known to be a monotone transform of DNSMOS, to attribute the gain to the semantic content of the text.
  2. [§4.1, Table 1] All objective results are single-run values with no error bars, confidence intervals, or significance tests. The claimed advantage of LR-AVSE over RL-DNSMOS is small: PESQ 1.57 vs 1.54, STOI 0.58 vs 0.57, and VQscore ties at 0.62. The prose 'significantly outperforming' in §4.1 is not supported by any statistical evidence. At least 3–5 random seeds with paired tests are needed to establish that the LLM reward yields a real difference over the DNSMOS reward.
  3. [§4.2, Fig. 4] The A/B preference test uses 21 listeners and only 10 utterances per condition. The reported 67.6% preference rate for LR-AVSE over RL-DNSMOS is presented without confidence intervals or a statistical test; at this sample size it may not be distinguishable from chance. Report per-utterance listener preferences, a binomial or paired test, and confidence intervals, and justify the choice of 10 utterances in terms of coverage of the SNR/noise conditions.
  4. [§2.3, Eqs. (9)–(10)] The policy optimization objective is non-standard and not fully specified. In Eq. (9), L(phi) includes the reward R, which depends on phi through the current sample, and the KL penalty is folded into the clipped-ratio term rather than added as a separate penalty. It is not clear how the density ratio pi_RL_phi / pi_RL_phi- is computed for the continuous Gaussian action in Eq. (5), or whether R is treated as a constant for the sampled trajectory when differentiating. Please clarify the derivation, or use a standard PPO advantage estimator with a separate KL penalty; otherwise the updated objective may not be the intended clipped surrogate.
minor comments (5)
  1. [§2.1, Eq. (2)] Notation inconsistency: the text uses 'Tv' and 'Dv' without subscripts; use T_v and D_v for clarity.
  2. [§2.2] The state/action definitions are loose: S is described as 'all possible noisy waveform distributions' but the policy outputs masks. Clarify how the state space maps to the mask action space in the RL formulation.
  3. [§3.2] Hyperparameters are given, but the number of PPO iterations, batch size, and total fine-tuning compute are not reported. This is needed for reproducibility and for judging the significance of the RL procedure.
  4. [§4.3, Fig. 3] The caption states that rewards from textual descriptions 'align with PESQ and STOI,' but only one sample is shown. Please soften the claim or provide a quantitative correlation over a larger set; the current sentence overstates the evidence.
  5. [§5] The manuscript itself acknowledges that SALMONN's descriptions follow fixed patterns. This limitation should be directly connected to the calibration concern in Eq. (7): if the reward saturates or collapses to a few values, the PPO updates may be driven by noise rather than by fine-grained quality differences.

Circularity Check

0 steps flagged

No significant circularity; the LLM-based reward is an external frozen benchmark and the evaluated metrics are independent of the training objective.

full rationale

The paper's central derivation is the RL fine-tuning objective in Eqs. (6)-(11): r_phi(yhat) = Sentiment(LLM(yhat)) is computed with frozen external models (SALMONN and BERT sentiment), and the relative reward subtracts the base policy's score instead of fitting anything to the test set. The DNSMOS baseline (Eq. 8) is likewise an external pretrained reward. None of these reward definitions reduces by construction to PESQ, STOI, NISQA, VQscore, S-BERT, or the subjective preference results; the evaluation metrics are independently computed on the test set. The paper does cite prior work by the same authors (e.g., [16], [18], [37]), but these citations support only general motivation or the choice of an evaluation metric, not the load-bearing reward-design claim. There is no imported uniqueness theorem, no ansatz smuggled in via self-citation, and no fitted parameter renamed as a prediction. The skeptical concern that the SALMONN+BERT reward is unvalidated and possibly misaligned is a legitimate external validity risk, not circularity: an incorrect or noisy reward is still an independent input to the optimization. Similarly, the absence of a random-reward ablation affects attribution but does not make the derivation equivalent to its inputs. Overall, the derivation is self-contained against external benchmarks, with only minor non-load-bearing self-citations, so the circularity score is low.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or ontological entities; it composes existing models (SALMONN, BERT sentiment, AVSEC-4 backbone). The load-bearing assumptions are the validity of the LLM+sentiment reward mapping, the transferability of the base pretrained model, and the correctness of the modified PPO surrogate. Hyperparameters are hand-chosen; none are fitted to the test set.

free parameters (5)
  • sigma (mask Gaussian noise std) = 0.05
    Hand-chosen exploration scale for the stochasticity requirement of RL (Section 3.2, Eq. 5); no sensitivity analysis.
  • epsilon (PPO clip range) = 0.1
    Hand-chosen PPO hyperparameter (Section 3.2, Eq. 10); standard value but not justified for this setting.
  • beta (KL weight) = 0.0001
    Hand-chosen weight for the KL term between the RL and base policies (Section 3.2, Eq. 9).
  • gamma (SI-SNR loss weight) = 1.0
    Hand-chosen balance between the RL objective and the original supervised loss (Section 3.2, Eq. 11).
  • learning rate = 0.001
    Hand-chosen optimizer learning rate (Section 3.2).
axioms (4)
  • domain assumption SALMONN, frozen, generates natural-language speech-quality descriptions that are perceptually meaningful for AVSEC-4 conditions (SNR -18 to +6.55 dB, real RIR test set).
    Invoked in Section 2.3 and 3.2 as the source of reward text. No calibration or validation of SALMONN's descriptions on the challenge's noise types or SNR range is provided, and Section 5 admits the outputs are repetitive fixed patterns.
  • ad hoc to paper BERT sentiment score on the LLM text is monotonically related to perceived speech quality.
    Eq. (7) makes this the entire reward signal. No correlation analysis against human ratings, PESQ, or STOI is given; Fig. 3 shows a single anecdotal example.
  • domain assumption The AVSEC-4 official pretrained model is a valid base policy whose SI-SNR objective transfers to the test distribution.
    Sections 2.1 and 3.3 initialize from the challenge-provided weights. Table 1 shows the pretrained baseline's STOI (0.48) is below the noisy input (0.55), indicating the base model degrades intelligibility on the real-RIR test set; the paper does not discuss this.
  • standard math Standard PPO clipping guarantees apply to the modified surrogate where the advantage is replaced by L(phi), which itself contains a KL term depending on the current policy.
    Eqs. (9)-(10). The replacement of A_t with L(phi) makes the clipped objective's argument policy-dependent, so the standard unbiasedness guarantee of PPO clipping does not straightforwardly apply; no ablation validates the simplification.

pith-pipeline@v1.3.0-alltime-deepseek · 8604 in / 15530 out tokens · 151735 ms · 2026-08-02T18:11:00.730383+00:00 · methodology

0 comments
read the original abstract

In existing Audio-Visual Speech Enhancement (AVSE) methods, objectives such as Scale-Invariant Signal-to-Noise Ratio (SI-SNR) and Mean Squared Error (MSE) are widely used; however, their correlation with perceived speech quality is often suboptimal and provides limited interpretability for optimization. This work proposes a reinforcement learning-based AVSE framework with a Large Language Model (LLM)-based interpretable reward model. An audio LLM generates natural language descriptions of enhanced speech, which are converted by a sentiment analysis model into a 1-5 rating score serving as the PPO reward for fine-tuning a pretrained AVSE model. Compared with scalar metrics, LLM-generated feedback is semantically rich and explicitly describes speech quality improvements. Experiments on the AVSEC-4 dataset show that the proposed method outperforms a supervised baseline and a DNSMOS-based RL baseline in PESQ, STOI, neural quality metrics, and subjective listening tests.

Figures

Figures reproduced from arXiv: 2603.13952 by Chih-Ning Chen, Fan-Gang Zeng, Hsin-Min Wang, Jen-Cheng Hou, Shao-Yi Chien, Yu Tsao.

Figure 1
Figure 1. Figure 1: The training procedure of the proposed LR-AVSE framework [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline of the LLM-based interpretable reward gen￾eration. model [24]. Let fθ(·) denote the neural network with param￾eters θ, which takes audio and visual inputs and produces the enhanced speech. The noisy speech waveform is denoted as x ∈ R C×T , where C denotes the number of channels and T denotes the temporal length. The corresponding visual input is represented as v ∈ R 1×F ×H×W , where F is the numb… view at source ↗
Figure 3
Figure 3. Figure 3: LR-AVSE inference with SALMONN. Rewards from textual descriptions align with PESQ and STOI, demonstrating LR-AVSE’s interpretability [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A/B preference test results on the AVSEC-4 test set. 4.2. Subjective Evaluation To subjectively evaluate speech quality, we conducted an A/B preference test. A total of 21 participants were recruited for the experiment. Two comparison conditions were designed: LR-AVSE vs. Pretrained Baseline and LR-AVSE vs. RL￾DNSMOS. Each comparison included 10 utterances. As shown in [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 12 linked inside Pith

  1. [1]

    Speech Enhancement (SE) techniques are there- fore widely used to suppress noise and improve speech qual- ity and intelligibility

    Introduction In real-world environments, speech is often corrupted by back- ground noise. Speech Enhancement (SE) techniques are there- fore widely used to suppress noise and improve speech qual- ity and intelligibility. Compared to conventional SE, Audio- Visual Speech Enhancement (A VSE) additionally incorporates visual information, providing complement...

  2. [2]

    the speech is clear but still has slight background noise

    Methodology This section describes in detail the technical architecture of the proposed LR-A VSE approach. We first introduce the conven- tional A VSE model as the base architecture, then reformulate the SE problem as a reinforcement learning problem, and de- scribe the RL-based policy optimization strategy and reward model design. The overall system pipe...

    Pith/arXiv arXiv 2026

  3. [3]

    Give me an assessment of the quality of this speech sample,

    Experimental Setup 3.1. Dataset We evaluate the proposed LR-A VSE method on the 4th COG-MHEAR Audio-Visual Speech Enhancement Challenge (A VSEC-4) dataset [24]. The training set contains 34,524 scenes, with target speakers drawn from 605 TED/TEDx speak- ers in the LRS3 [30] dataset. Noise sources include 405 compet- ing speakers and 7,346 noise files span...

  4. [4]

    dis- torted and muffled,

    Results 4.1. Objective Quality Metrics As shown in Table 1, evaluated on the test set, LR-A VSE achieves a PESQ of 1.57, outperforming the Pretrained Base- line at 1.45 and RL-DNSMOS at 1.54. Regarding the NISQA- predicted MOS, LR-A VSE reaches 1.29, significantly outper- forming the Noisy input at 0.97 and the Pretrained Baseline at 0.99. On the VQscore ...

  5. [5]

    The quality of this speech sample is poor/good

    Discussion Our LLM-based reward demonstrates clear advantages in inter- pretability; however, the current choice of LLM remains rela- tively limited. When generating natural language descriptions, the produced sentences tend to follow fixed patterns—for ex- ample, repeating phrases such as “The quality of this speech sample is poor/good” or “The audio is ...

  6. [6]

    Conclusion This paper proposes LR-A VSE, which, to the best of our knowl- edge, is the first framework that leverages reinforcement learn- ing from LLM feedback for A VSE. The proposed LLM-based interpretable reward achieves notable improvements over the supervised-trained baseline and DNSMOS-based RL A VSE systems across PESQ, STOI, neural speech quality...

  7. [7]

    Generative AI Use Disclosure Generative AI was used only for editing and polishing this manuscript

  8. [8]

    Looking to listen at the cock- tail party: A speaker-independent audio-visual model for speech separation,

    A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cock- tail party: A speaker-independent audio-visual model for speech separation,”arXiv preprint arXiv:1804.03619, 2018

    Pith/arXiv arXiv 2018

  9. [9]

    Visual speech enhance- ment,

    A. Gabbay, A. Shamir, and S. Peleg, “Visual speech enhance- ment,”arXiv preprint arXiv:1711.08789, 2017

    Pith/arXiv arXiv 2017

  10. [10]

    Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,

    M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,”Information Fusion, vol. 63, pp. 273–285, 2020

    2020

  11. [11]

    An overview of deep-learning-based audio-visual speech enhancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech enhancement and separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1368–1396, 2021

    2021

  12. [12]

    Audio-visual speech enhancement using multi- modal deep convolutional neural networks,

    J.-C. Hou, S.-S. Wang, Y .-H. Lai, Y . Tsao, H.-W. Chang, and H.-M. Wang, “Audio-visual speech enhancement using multi- modal deep convolutional neural networks,”IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 2, pp. 117–128, 2018

    2018

  13. [13]

    Audio-visual speech enhancement using conditional varia- tional auto-encoders,

    M. Sadeghi, S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Ho- raud, “Audio-visual speech enhancement using conditional varia- tional auto-encoders,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1788–1800, 2020

    2020

  14. [14]

    Av-crossnet: an audiovisual complex spectral mapping network for speech separation by leveraging narrow-and cross-band mod- eling,

    V . A. Kalkhorani, C. Yu, A. Kumar, K. Tan, B. Xu, and D. Wang, “Av-crossnet: an audiovisual complex spectral mapping network for speech separation by leveraging narrow-and cross-band mod- eling,”IEEE Journal of Selected Topics in Signal Processing, 2025

    2025

  15. [15]

    Viseme-gated multilayer cross- attentional feature fusion for cognitively-inspired multimodal speech enhancement,

    N. Saleem, A. Hussain, K. Dashtipour, E. Sheikh, A. Sheikh, T. Arslan, and A. Hussain, “Viseme-gated multilayer cross- attentional feature fusion for cognitively-inspired multimodal speech enhancement,”IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 469–481, 2025

    2025

  16. [16]

    SDR– half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR– half-baked or well done?” inICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 626–630

    2019

  17. [17]

    MetricGAN: Gen- erative adversarial networks based black-box metric scores opti- mization for speech enhancement,

    S.-W. Fu, C.-F. Liao, Y . Tsao, and S.-D. Lin, “MetricGAN: Gen- erative adversarial networks based black-box metric scores opti- mization for speech enhancement,” inInternational Conference on Machine Learning. PmLR, 2019, pp. 2031–2041

    2019

  18. [18]

    MetricGAN+: An improved version of metric- gan for speech enhancement,

    S.-W. Fu, C. Yu, T.-A. Hsieh, P. Plantinga, M. Ravanelli, X. Lu, and Y . Tsao, “MetricGAN+: An improved version of metric- gan for speech enhancement,”arXiv preprint arXiv:2104.03538, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), vol. 2. IEEE, 2001, pp. 749–752

    2001

  20. [20]

    CMGAN: Conformer- based metric gan for speech enhancement,

    R. Cao, S. Abdulatif, and B. Yang, “CMGAN: Conformer- based metric gan for speech enhancement,”arXiv preprint arXiv:2203.15149, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in2010 IEEE international conference on acous- tics, speech and signal processing. IEEE, 2010, pp. 4214–4217

    2010

  22. [22]

    Lan- guage models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Lan- guage models are few-shot learners,”Advances in neural informa- tion processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  23. [23]

    A study on zero-shot non-intrusive speech assessment using large language models,

    R. E. Zezario, S. M. Siniscalchi, H.-M. Wang, and Y . Tsao, “A study on zero-shot non-intrusive speech assessment using large language models,” inICASSP 2025-2025 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  24. [24]

    Audio large language models can be descriptive speech quality evaluators,

    C. Chen, Y . Hu, S. Wang, H. Wang, Z. Chen, C. Zhang, C.- H. H. Yang, and E. S. Chng, “Audio large language models can be descriptive speech quality evaluators,”arXiv preprint arXiv:2501.17202, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Qualispeech: A speech quality assessment dataset with natural language reasoning and descrip- tions,

    S. Wang, W. Yu, X. Chen, X. Tian, J. Zhang, L. Lu, Y . Tsao, J. Ya- magishi, Y . Wang, and C. Zhang, “Qualispeech: A speech quality assessment dataset with natural language reasoning and descrip- tions,” inProceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (V olume 1: Long Papers), 2025, pp. 23 588–23 609

    2025

  26. [26]

    Dnn-based source enhancement self-optimized by reinforcement learning using sound quality measurements,

    Y . Koizumi, K. Niwa, Y . Hioka, K. Kobayashi, and Y . Haneda, “Dnn-based source enhancement self-optimized by reinforcement learning using sound quality measurements,” in2017 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 81–85

    2017

  27. [27]

    Reinforcement learning based speech enhancement for robust speech recognition,

    Y .-L. Shen, C.-Y . Huang, S.-S. Wang, Y . Tsao, H.-M. Wang, and T.-S. Chi, “Reinforcement learning based speech enhancement for robust speech recognition,” inICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6750–6754

    2019

  28. [28]

    NISQA: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,

    G. Mittag, B. Naderi, A. Chehadi, and S. M ¨oller, “NISQA: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,”arXiv preprint arXiv:2104.09494, 2021

    Pith/arXiv arXiv 2021

  29. [29]

    Using rlhf to align speech enhancement approaches to mean-opinion quality scores,

    A. Kumar, A. Perrault, and D. S. Williamson, “Using rlhf to align speech enhancement approaches to mean-opinion quality scores,” inICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  30. [30]

    Aligning generative speech enhancement with human preferences via direct preference optimization,

    H. Li, N. Hou, Y . Hu, J. Yao, S. M. Siniscalchi, and E. S. Chng, “Aligning generative speech enhancement with human preferences via direct preference optimization,”arXiv preprint arXiv:2507.09929, 2025

    arXiv 2025

  31. [31]

    Avse challenge baseline model,

    CogMhear, “Avse challenge baseline model,” https://github.com/ cogmhear/avse challenge, 2024, accessed: 2026-02-19

    2024

  32. [32]

    Conv-tasnet: Surpassing ideal time– frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time– frequency magnitude masking for speech separation,”IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 8, pp. 1256–1266, 2019

    2019

  33. [33]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    Train- ing language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Train- ing language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  35. [35]

    DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6493–6497

    2021

  36. [36]

    DNSMOS p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    ——, “DNSMOS p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” inICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2022, pp. 886–890

    2022

  37. [37]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

    Pith/arXiv arXiv 2018

  38. [38]

    Clarity enhancement challenge 1 (CEC1),

    Clarity Challenge, “Clarity enhancement challenge 1 (CEC1),” https://github.com/claritychallenge/clarity/tree/main/recipes/ cec1, 2021, accessed: 2024

    2021

  39. [39]

    Deep noise suppression (DNS) challenge,

    Microsoft, “Deep noise suppression (DNS) challenge,” https:// github.com/microsoft/DNS-Challenge, 2020, accessed: 2024

    2020

  40. [40]

    Med- leyDB: A multitrack dataset for annotation-intensive MIR re- search,

    H. Bitteur, J. Salamon, E. J. Humphrey, and J. P. Bello, “Med- leyDB: A multitrack dataset for annotation-intensive MIR re- search,” https://medleydb.weebly.com/, 2014, accessed: 2024

    2014

  41. [41]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,”arXiv preprint arXiv:2310.13289, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    BERT base multilingual un- cased sentiment,

    Y . Peirsman, “BERT base multilingual un- cased sentiment,” https://huggingface.co/nlptown/ bert-base-multilingual-uncased-sentiment, 2020, accessed: 2024

    2020

  43. [43]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguis- tics: human language technologies, volume 1 (long and short pa- pers), 2019, pp. 4171–4186

    2019

  44. [44]

    Self- supervised speech quality estimation and enhancement using only clean speech,

    S.-W. Fu, K.-H. Hung, Y . Tsao, and Y .-C. F. Wang, “Self- supervised speech quality estimation and enhancement using only clean speech,”arXiv preprint arXiv:2402.16321, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    VERSA: A versatile evaluation toolkit for speech, audio, and music,

    J. Shi, H. jin Shim, J. Tian, S. Arora, H. Wu, D. Petermann, J. Q. Yip, Y . Zhang, Y . Tang, W. Zhang, D. S. Alharthi, Y . Huang, K. Saito, J. Han, Y . Zhao, C. Donahue, and S. Watanabe, “VERSA: A versatile evaluation toolkit for speech, audio, and music,” in2025 Annual Conference of the North American Chapter of the Association for Computational Linguist...

    2025

  46. [46]

    Espnet-codec: Comprehensive training and evalua- tion of neural codecs for audio, music, and speech,

    J. Shi, J. Tian, Y . Wu, J.-W. Jung, J. Q. Yip, Y . Masuyama, W. Chen, Y . Wu, Y . Tang, M. Baali, D. Alharthi, D. Zhang, R. Deng, T. Srivastava, H. Wu, A. Liu, B. Raj, Q. Jin, R. Song, and S. Watanabe, “Espnet-codec: Comprehensive training and evalua- tion of neural codecs for audio, music, and speech,” in2024 IEEE Spoken Language Technology Workshop (SL...

    2024

  47. [47]

    Speechbertscore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,

    T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “Speechbertscore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,”arXiv preprint arXiv:2401.16812, 2024

    Pith/arXiv arXiv 2024