Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction

Haizhou Li; Helen Meng; Shuai Wang; Wenxuan Wu; Xixin Wu

REVIEW 2 major objections 4 minor 40 references

Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a text-model loss to audio-visual target speech extraction raises SI-SDR from 12.03 to 13.60 dB with no inference cost.

desk verdict The paper has a promising idea and a solid backbone, but its central comparison is confounded by a missing stage-2 control, so the headline gain should not be taken at face value yet. read the letter →

arxiv 2506.09792 v2 pith:C52RJPI4 submitted 2025-06-11 cs.SD cs.LGcs.MMeess.AS

Wenxuan Wu , Shuai Wang , Xixin Wu , Helen Meng , Haizhou Li This is my paper

classification cs.SDcs.LGcs.MMeess.AS

keywords pre-trainedlanguagemodelaudio-visualtargetspeakerextractionlinguisticconstraintscocktailpartyproblemself-supervisedspeechrepresentationsintelligibilitySI-SDRmulti-modallearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that audio-visual target speaker extraction improves when the training objective is augmented with a linguistic-constraint loss derived from a frozen pre-trained speech-language model (PSLM) or pre-trained language model (PLM), in addition to the standard SI-SDR reconstruction loss. This matters because it offers a way to inject external linguistic knowledge into the extraction model without adding any computation or parameters at inference time. On the core test set, the best configuration, using RoBERTa-B-L12, raises SI-SDR from 12.03 to 13.60 dB and improves PESQ, STOI, and SpeechBERTScore over the baseline. The same supervision generalizes to five monolingual test sets, to visual-cue-impaired inputs, and to the LRS3 cross-domain set, supporting the claim that linguistic knowledge supplies information that visual and acoustic cues do not fully provide.

What carries the argument

The load-bearing mechanism is the modified training objective in Eq. (2): a weighted sum of the standard SI-SDR reconstruction loss and a linguistic-constraint loss $L_{\mathrm{LC}}$ computed against a frozen external knowledge model. $L_{\mathrm{LC}}$ takes one of three forms: mean-squared error between continuous HuBERT or WavLM features of target and extracted speech; cross-entropy between discrete HuBERT token distributions; or mean-squared error between a RoBERTa transcript embedding and an adapter-projected, time-averaged PSLM embedding of the extracted speech. Training is done in two stages, first with SI-SDR only and then with both losses, and the pretrained models are frozen and used only during training. As a result, the deployed extraction network has no extra parameters or runtime cost; two fully connected layers form the only adapter needed for the PLM branch.

What would settle it

Retrain the model exactly as described but replace the RoBERTa transcript embeddings used in $L_{\mathrm{LC}}$ with permuted or random embeddings from the same frozen model. If the SI-SDR gains on the core test set persist close to the reported 1.57 dB improvement, the gain is a regularization artifact; if the gains collapse, the linguistic-content mechanism is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that optimizing AV-TSE with a two-term objective — SI-SDR for signal reconstruction plus a linguistic-constraint loss $L_{\mathrm{LC}}$ between representations of the extracted speech and representations of the target speech or its transcript — consistently improves extraction quality and intelligibility. The paper formulates this as $\theta^{**} = \arg\min_{\theta} \, \alpha L_{\mathrm{SI\text{-}SDR}}(f_\theta(x,v), y) + \beta L_{\mathrm{LC}}(\hat{z}, z)$, with $\alpha=1$ and $\beta=10$. For PSLM sources, $\hat{z}$ and $z$ are frame-level feature sequences from HuBERT or WavLM, compared with mean-squared error, or discrete token distributions compared with cross-entropy. For a PLM source, $z$ is the RoBERTa embedding of the target transcript and $\hat{z}$ is a sentence-averaged PSLM embedding of the extracted speech, projected by two adapter layers into the same space; the two are aligned with mean-squared error. The paper reports that this PLM version outperforms all PSLM versions and also beats comparison systems that add inference-time modules, such as a mask-and-recover strategy and an explicit HuBERT-integrated chain.

Load-bearing premise

The load-bearing premise is that a sentence-level average of the extracted speech's pre-trained speech-language model (PSLM) embedding, after a two-layer projection, still carries enough linguistic content to align meaningfully with the RoBERTa transcript embedding; if averaging washes out word order and content, the text-model constraint would be a weak regularizer rather than a true linguistic-knowledge signal.

Editorial extensions

If this is right

Using the RoBERTa-B-L12 linguistic constraint raises SI-SDR from 12.03 to 13.60 dB on the core test set, with consistent gains on SI-SDRi, SDR, PESQ, STOI, and SpeechBERTScore.
The same constraint improves monolingual test sets for English, Portuguese, Spanish, Italian, and French, including roughly 1.5 dB SI-SDR gains on the two lowest-resource languages tested, Portuguese and Spanish.
Under partial occlusion, low resolution, and fully missing visual input, the linguistic-constrained model outperforms the vanilla backbone and a visual-inpainting baseline, even though it was never trained on impaired visual cues.
On the cross-domain LRS3 test set, the RoBERTa-constrained model reaches 15.42 dB SI-SDRi, above the baseline's 13.83 dB and the comparison chain model's 15.3 dB.
All reported gains come from training-time supervision alone; at inference the model is identical to the unconstrained AV-TSE backbone, with no added parameters or computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The training-only design suggests the linguistic-constraint loss could be grafted onto other AV-TSE backbones, not just the AV-Sepformer backbone tested here, because it does not alter the model graph at inference.
Since the PLM branch uses a sentence-averaged representation, the constraint carries sentence-level semantic information; testing frame-level or word-level alignment would be a natural extension and could produce larger gains on content-heavy utterances.
The monolingual results point toward a multilingual application: a single frozen multilingual text model could provide transcript embeddings for many languages, potentially improving low-resource languages where paired audio-visual training data is scarce.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline result is probably real, but the paper doesn't actually show it. The idea is sensible: add a training-only linguistic-consistency loss from a PSLM or PLM on top of SI-SDR, so the AV-TSE model gets extra linguistic supervision without any inference cost. The systematic comparison of HuBERT/WavLM (continuous and token) against RoBERTa, with adapters for the text branch, is a genuinely useful piece of work. So are the multilingual, visual-impaired, and cross-domain evaluations. Credit where due: the method is clearly described, the backbone is standard, and the gains are consistent across tables. This is not a toy setup. But there is a load-bearing confound in the experiment design. The baseline is the stage-1 checkpoint trained only with SI-SDR. Every proposed model starts from that checkpoint and then gets additional stage-2 training with L_LC added. There is no stage-2 control trained for the same extra epochs with beta=0 (SI-SDR only), and no non-linguistic auxiliary-loss control matched for compute. So the 1.5 dB SI-SDR improvement attributed to 'linguistic knowledge' could just be continued fine-tuning or generic regularization. That issue is easy to fix and the paper should have fixed it before submitting. Also worth noting, the SpeechBERTScore semantic metric shares representation family with the HuBERT/WavLM training targets, so the semantic-coherence claim is partly circular for the PSLM variants; the SI-SDR gains are not circular because they are grounded in signal reconstruction, but the interpretation of what drives those gains is exactly what the missing control would clarify. The absence of error bars or significance tests is a lesser issue at this scale but still worth asking for. The PLM averaging concern (mean-pooling the PSLM output before aligning with RoBERTa) seems minor to me; the two-layer adapter plus the reported results suggest the signal survives. Overall: the paper deserves serious referee time, but the verdict should hinge on adding the stage-2 SI-SDR-only control and ideally a non-linguistic auxiliary loss. That is a standard experimental requirement, not a demand for a different paper. I would not cite the headline numbers until that control is in place, but I would bring this to a reading group to discuss the training-only supervision idea.

Referee Report

2 major / 4 minor

Summary. This paper proposes adding a linguistic-constraint auxiliary loss to audio-visual target speech extraction (AV-TSE) training. The base system is AV-Sepformer trained with SI-SDR; in a second stage, the objective becomes alpha*L_SI-SDR + beta*L_LC (Eq. 2), where the constraint is either an MSE or cross-entropy match to PSLM features/tokens (HuBERT, WavLM) of the clean and extracted speech, or an MSE match between a RoBERTa transcript embedding and a time-averaged PSLM embedding of the extracted speech after two adapter layers. Experiments cover a core two-speaker mixture test set, five monolingual test sets, visual-cue-impaired conditions, and an LRS3 cross-domain evaluation. The reported results show consistent improvements over the stage-1 baseline, with RoBERTa-B-L12 giving the largest gains, at no additional inference-time parameter or compute cost.

Significance. If the results hold, the contribution is practically useful: it offers a simple way to inject knowledge from frozen PLMs/PSLMs purely at training time while preserving inference efficiency. The evaluation is broad, spanning multilingual, visual-impairment, and cross-domain settings, and the gain pattern is coherent (RoBERTa > WavLM-L > smaller PSLMs; discrete and continuous PSLM targets behave similarly). The paper also includes CTC and MAR comparisons that help contextualize the method. The main weakness is that the headline comparison lacks a stage-2 SI-SDR-only control, so the improvements cannot yet be attributed to the linguistic constraint. With that control added, the paper would be a solid empirical contribution.

major comments (2)

[Sec. 3.3 / Table 1] The central comparison is confounded by the training schedule. The baseline is the stage-1 AV-Sepformer checkpoint trained only with L_SI-SDR, while every proposed model is initialized from that checkpoint and then trained for additional epochs with L_LC added (Section 3.3, alpha=1, beta=10). A stage-2 control with beta=0 (same number of epochs, same learning-rate schedule) is missing, as is a non-linguistic auxiliary-loss control matched for compute. Without this control, the gains in Table 1 (e.g., SI-SDR from 12.03 to 13.60 for RoBERTa-B-L12) and the corresponding gains in Tables 2-4 and Figure 2 cannot be attributed to the linguistic knowledge; they may reflect continued fine-tuning or generic regularization. Please add the beta=0 stage-2 baseline and, ideally, an auxiliary-loss control with the same adapter/training budget.
[Sec. 3.2 / Table 1] The semantic-coherence claim is partly circular for the PSLM variants. SpeechBERTScore is computed with HuBERT Base features, while the PSLM training losses in Section 2.2 minimize distance to HuBERT-L and WavLM features or HuBERT discrete units. Because these representation families are closely related, larger SpeechBERTScore values for the PSLM variants (Table 1, 0.8247 to 0.8510) are partly a consequence of the training objective and do not independently establish improved semantic coherence. The PLM variant is less affected, but the semantic claim should be supported by a metric outside the same self-supervised family, such as an ASR-based measure or a text-based similarity computed from recognized transcripts, or should be down-weighted.

minor comments (4)

[All tables] No error bars, confidence intervals, or significance tests are reported; since each configuration appears to be a single run, please state whether the reported numbers are averages over multiple runs or single trials, and ideally provide variance information.
[Sec. 2.2] For the PLM branch, the extracted-speech representation is obtained by averaging the PSLM latent sequence. Time-averaging discards word order and much of the sentence-level structure, so it is worth an ablation or an analysis (e.g., comparing against an un-pooled sequence or random transcript pairs) to show what linguistic information the averaged embedding actually preserves.
[Sec. 4.4 / Table 4] The AVSepChain comparison on LRS3 uses two-second speech segments as reported in the original paper, whereas the other rows use different segment handling; this cross-paper comparison should be explicitly labeled as not protocol-matched.
[Sec. 3.3] Only one value of the weighting factor beta is explored (beta=10); a small sensitivity analysis over beta would strengthen the claim that the method is not sensitive to a single hyperparameter choice.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the linguistic-constraint loss is an external supervision signal, and the reported gains are empirical comparisons, not consequences of the loss definition.

full rationale

The central derivation is an empirical training-objective modification: Eq. (2) adds an auxiliary loss L_LC(hat_z, z) to SI-SDR, where z is an external representation obtained from ground-truth speech or transcripts, not from the model's own output. This is not self-definitional. The reported SI-SDR, SI-SDRi, PESQ, and STOI gains are computed on held-out mixtures with standard signal-level metrics independent of the auxiliary loss. The semantic-coherence metric (SpeechBERTScore with HuBERT Base, Sec. 3.2) shares the self-supervised speech-representation family with the PSLM constraints (HuBERT-L-L24, WavLM-L-L24), but it is not the same model or the same layer, and for the RoBERTa constraint the training target is the transcript embedding rather than HuBERT features; therefore the semantic gains are not forced by construction. The one substantive concern is experimental rather than circular: Sec. 3.3 states that the stage-1 baseline is trained only with L_SI-SDR, then 'in the second stage, training continues with the same settings, incorporating both L_SI-SDR and L_LC', so the proposed models receive additional gradient updates and there is no stage-2 beta=0 control isolating the linguistic term. This is a potential attribution confound, but it does not make the result equivalent to its input by definition. Self-citations to [3], [4], and [5] appear as related work and baselines, not as the load-bearing justification for the method. Verdict: no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The ledger is modest: one hand-set loss weight (beta=10), a few domain assumptions about the alignment and informativeness of frozen pretrained representations, and no invented entities. The central claim depends on auxiliary feature matching being a useful training signal.

free parameters (1)

beta, the weighting factor for the linguistic loss L_LC = 10
Set by hand in Eq. (2) and Section 3.3; no sensitivity analysis is reported, so the robustness of the gains to this value is unknown.

assumptions (4)

domain assumption Time-averaged PSLM features preserve sentence-level linguistic content.
Section 2.2 uses the mean of the PSLM latent sequence as the extracted-speech representation for the PLM branch; if averaging removes content information, the constraint cannot carry linguistic knowledge.
domain assumption Frozen pretrained model features provide stable and meaningful targets for noisy or imperfect extracted speech.
Section 2.2 treats the PSLM/PLM representations of target speech and transcripts as reference signals for extracted speech, assuming the MSE and CE gradients are beneficial.
domain assumption Two-stage training, first with SI-SDR only and then with the combined loss, reaches a good optimum.
Section 3.3 states this schedule; no analysis shows the first stage is necessary or that the combined loss alone would fail.
domain assumption RoBERTa sentence embeddings and projected PSLM mean embeddings are comparable in a shared space for MSE.
Section 2.2 introduces adapters to bridge the latent-space shift, relying on the assumption that such a projection can align the two spaces.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction." pith.science (2026). https://pith.science/paper/C52RJPI4

@misc{pith2026250609792,
  author       = {Pith},
  title        = {Pith review of: Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C52RJPI4}},
  note         = {Machine review of arXiv:2506.09792}
}

read the original abstract

Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains.

Figures

Figures reproduced from arXiv: 2506.09792 by the authors.

**Figure 1.** AV-TSE model with linguistic constraints. From left to right: continuous features and discrete tokens of PSLM, continuous features of PLM. We explore two ways to incorporate PSLMs: continuous features and discrete tokens after quantization. For continuous features, we utilize feature representations extracted from the 24th layer of HuBERT-Large (HuBERT-L-L24) [9], the 24th layer of WavLM-Large (WavLM-L-L24) [19], a… view at source ↗

**Figure 2.** SI-SDR comparison under different visual cueimpaired ratios. Specifically, we compute the average performance on three visual impaired scenarios with impairment ratios ranging from 0% to 100%, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 38 canonical work pages

[1]

Most existing studies focus on improving the audio-visual fusion mechanisms [1, 2, 3, 4, 5] or addressing visual cue-impaired scenarios [6, 7]

Introduction Audio-visual target speaker extraction (A V-TSE) simulates the human ability to extract the target speaker’s utterance in a cock- tail party scenario using visual cues. Most existing studies focus on improving the audio-visual fusion mechanisms [1, 2, 3, 4, 5] or addressing visual cue-impaired scenarios [6, 7]. Despite the progress, the role ...

work page
[2]

Methods 2.1. Problem Formulation Given a mixture speech signalxand the target speaker’s visual cuev, the goal of an A V-TSE systemf θ is to extract a clean speech signalˆythat approximates the ground truth target speech yviaˆy=f θ(x, v). In most conventional A V-TSE systems, the model parameterθcan be obtained by optimizing the SI-SDR lossL SI-SDR as θ∗ =...

work page Pith review arXiv 2025
[3]

Dataset In this study, several experimental settings are considered: •Training Set:A two-speaker mixture training set is simu- lated following previous work [1, 6, 2, 3]

Experimental Setups 3.1. Dataset In this study, several experimental settings are considered: •Training Set:A two-speaker mixture training set is simu- lated following previous work [1, 6, 2, 3]. The training set contains800speakers with20,000, utterances. Speakers ap- pearing in the test sets are excluded from the training data and SNR of interfering utt...

work page
[4]

Results and Analysis In this section, linguistic knowledge from different pretrained models is evaluated and analyzed on various test sets. For a more comprehensive comparison, we also include two systems that do not explicitly incorporate pretrained models as external knowledge source but have been validated in previous studies as beneficial for enhancin...

work page
[5]

Full occ

Note that A V-Sepformer with RoBERTa-B-L12 or WavLM- L-L24 were not exposed to any visual cue-impaired scenario, but still outperformed ImagineNET Sys.2 across all impair ra- tios, where the latter have seen visual full missing conditions during training. This finding suggests the proposed linguistic Table 3:SI-SDR performance under three visual impaired ...

work page
[6]

Conclusion In this study, we propose to derive linguistic constraints from pre-trained language models and speech-language mod- els to optimize A V-TSE. Evaluated on multilingual, visual cue- impaired, and cross-domain test sets, the proposed paradigm consistently outperforms the vanilla A V-TSE backbone, while the linguistic constraints from the language...

work page
[7]

62401377, Shenzhen Sci- ence and Technology Program (Shenzhen Key Laboratory, Grant No

Acknowledgements This work was supported by National Natural Science Foun- dation of China, Grant No. 62401377, Shenzhen Sci- ence and Technology Program (Shenzhen Key Laboratory, Grant No. ZDSYS20230626091302006), Shenzhen Science and Technology Research Fund (Fundamental Research Key Project, Grant No. JCYJ20220818103001002), Program for Guangdong Intro...

work page
[8]

Muse: Multi-modal target speaker extraction with visual cues,

Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,”ICASSP 2021

work page 2021

Show all 40 references

[9]

Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction,

J. Lin, X. Cai, H. Dinkel, J. Chen, Z. Yan, Y . Wang, J. Zhang, Z. Wu, Y . Wang, and H. Meng, “Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction,” inICASSP 2023

2023
[10]

Avhumar: Audio- visual target speech extraction with pre-trained av-hubert and mask-and-recover strategy,

W. Wu, X. Chen, X. Wu, H. Li, and H. Meng, “Avhumar: Audio- visual target speech extraction with pre-trained av-hubert and mask-and-recover strategy,”CVPR 2024 Sight and Sound Work- shop

2024
[11]

Target speech extraction with pre-trained av-hubert and mask-and-recover strat- egy,

W. Wu, X. Chen, X. Wu, H. Li, and H. Meng, “Target speech extraction with pre-trained av-hubert and mask-and-recover strat- egy,” inIJCNN 2024

2024
[12]

c 2av-tse: Context and confidence-aware audio visual target speaker extraction,

W. Wu, X. Chen, S. Wang, J. Wang, L. Meng, X. Wu, H. Meng, and H. Li, “c 2av-tse: Context and confidence-aware audio visual target speaker extraction,”IEEE Journal of Selected Topics in Sig- nal Processing 2025

2025
[13]

Imaginenet: Target speaker extraction with intermittent visual cue through embedding inpainting,

Z. Pan, W. Wang, M. Borsdorf, and H. Li, “Imaginenet: Target speaker extraction with intermittent visual cue through embedding inpainting,” inICASSP 2023

2023
[14]

Restoring speaking lips from occlusion for audio-visual speech recognition,

J. Wang, Z. Pan, M. Zhang, R. T. Tan, and H. Li, “Restoring speaking lips from occlusion for audio-visual speech recognition,” inAAAI 2024

2024
[15]

Semantic en- coding during language comprehension at single-cell resolution,

M. Jamali, B. Grannan, J. Cai, A. Khanna, W. Mu ˜noz, I. Caprara, A. Paulk, S. Cash, E. Fedorenko, and Z. Williams, “Semantic en- coding during language comprehension at single-cell resolution,” Nature, vol. 631, pp. 1–7, 07 2024

2024
[16]

Hubert: Self-supervised speech representa- tion learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representa- tion learning by masked prediction of hidden units,”IEEE/ACM TASLP 2021

2021
[17]

Large language model can transcribe speech in multi-talker scenarios with versatile instructions,

L. Meng, S. Hu, J. Kang, Z. Li, Y . Wang, W. Wu, X. Wu, X. Liu, and H. Meng, “Large language model can transcribe speech in multi-talker scenarios with versatile instructions,” in ICASSP 2025

2025
[18]

Target speech extraction with pre-trained self-supervised learning models,

J. Peng, M. Delcroix, T. Ochiai, O. Plchot, S. Araki, and J. Cer- nocky, “Target speech extraction with pre-trained self-supervised learning models,”ICASSP 2024, 2024

2024
[19]

Probing self-supervised learning models with target speech extraction,

J. Peng, M. Delcroix, T. Ochiai, O. Plchot, T. Ashihara, S. Araki, and J. ˇCernock´y, “Probing self-supervised learning models with target speech extraction,” inICASSPW 2024

2024
[20]

A large-scale evaluation of speech foundation models,

S.-w. Yang, H.-J. Chang, Z. Huang, A. T. Liu, C.-I. Lai, H. Wu, J. Shi, X. Chang, H.-S. Tsai, W.-C. Huanget al., “A large-scale evaluation of speech foundation models,”IEEE/ACM TASLP 2024

2024
[21]

Transferring knowledge from large foundation models to small downstream models,

S. Qiu, B. Han, D. C. Maddix, S. Zhang, Y . Wang, and A. G. Wilson, “Transferring knowledge from large foundation models to small downstream models,” inICML 2024

2024
[22]

Knowledge transfer from pre-trained language models to cif-based speech recognizers via hierarchical distillation,

M. Han, F. Chen, J. Shi, S. Xu, and B. Xu, “Knowledge transfer from pre-trained language models to cif-based speech recognizers via hierarchical distillation,” inInterspeech 2023

2023
[23]

Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,”ICLR 2024

2024
[24]

LSCodec: Low-bitrate and speaker-decoupled discrete speech codec,

Y . Guo, Z. Li, C. Du, H. Wang, X. Chen, and K. Yu, “LSCodec: Low-bitrate and speaker-decoupled discrete speech codec,”arXiv preprint arXiv:2410.15764, 2024

2024 arXiv
[25]

ALMTokenizer: A Low- bitrate and Semantic-rich Audio Codec Tokenizer for Audio Lan- guage Modeling,

D. Yang, S. Liu, H. Guo, J. Zhao, Y . Wang, H. Wang, Z. Ju, X. Liu, X. Chen, X. Tan, X. Wu, and H. Meng, “ALMTokenizer: A Low- bitrate and Semantic-rich Audio Codec Tokenizer for Audio Lan- guage Modeling,” inICML 2025

2025
[26]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, M. Zeng, and F. Wei, “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE JSTSP 2021

2021
[27]

Roberta: A robustly optimized bert pretraining approach,

Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,”ArXiv, 2019

2019
[28]

Separate in the speech chain: cross-modal conditional audio-visual target speech extraction,

Z. Mu and X. Yang, “Separate in the speech chain: cross-modal conditional audio-visual target speech extraction,” inIJCAI 2024

2024
[29]

V oxceleb2: Deep speaker recognition,

J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” inINTERSPEECH, 2018

2018
[30]

Watch or listen: Ro- bust audio-visual speech recognition with visual corruption mod- eling and reliability scoring,

J. Hong, M. Kim, J. Y . Choi, and Y . M. Ro, “Watch or listen: Ro- bust audio-visual speech recognition with visual corruption mod- eling and reliability scoring,”2023 CVPR

2023
[31]

Lrs3-ted: a large- scale dataset for visual speech recognition,

T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large- scale dataset for visual speech recognition,”ArXiv, 2018

2018
[32]

Sdr – half-baked or well done?

J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr – half-baked or well done?”ICASSP 2019

2019
[33]

Single-sided Real-time PESQ Score Estimation,

S. Basterrech, G. Rubino, and M. Varela, “Single-sided Real-time PESQ Score Estimation,”arXiv e-prints, p. arXiv:1212.6350, Dec. 2012

2012 arXiv
[34]

An al- gorithm for intelligibility prediction of time–frequency weighted noisy speech,

C. H. Taal, R. C. Hendriks, R. Heusdens, and J. R. Jensen, “An al- gorithm for intelligibility prediction of time–frequency weighted noisy speech,”IEEE TASLP 2011

2011
[35]

SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,

T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,” inInter- speech 2024

2024
[36]

How should we extract discrete audio tokens from self-supervised models?

P. Mousavi, J. Duret, S. Zaiem, L. Della Libera, A. Ploujnikov, C. Subakan, and M. Ravanelli, “How should we extract discrete audio tokens from self-supervised models?” inInterspeech 2024, 2024

2024
[37]

Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,

A. Graves, S. Fern´andez, F. J. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” inICML 2006

2006
[38]

wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inNIPS 2020

2020
[39]

Learning audio-visual speech representation by masked multimodal cluster prediction,

B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” inICLR 2024

2024
[40]

Intuitive multilingual audio- visual speech recognition with a single-trained model,

J. Hong, S. J. Park, and Y . M. Ro, “Intuitive multilingual audio- visual speech recognition with a single-trained model,” inEMNLP 2023

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Most existing studies focus on improving the audio-visual fusion mechanisms [1, 2, 3, 4, 5] or addressing visual cue-impaired scenarios [6, 7]

Introduction Audio-visual target speaker extraction (A V-TSE) simulates the human ability to extract the target speaker’s utterance in a cock- tail party scenario using visual cues. Most existing studies focus on improving the audio-visual fusion mechanisms [1, 2, 3, 4, 5] or addressing visual cue-impaired scenarios [6, 7]. Despite the progress, the role ...

work page

[2] [2]

Methods 2.1. Problem Formulation Given a mixture speech signalxand the target speaker’s visual cuev, the goal of an A V-TSE systemf θ is to extract a clean speech signalˆythat approximates the ground truth target speech yviaˆy=f θ(x, v). In most conventional A V-TSE systems, the model parameterθcan be obtained by optimizing the SI-SDR lossL SI-SDR as θ∗ =...

work page Pith review arXiv 2025

[3] [3]

Dataset In this study, several experimental settings are considered: •Training Set:A two-speaker mixture training set is simu- lated following previous work [1, 6, 2, 3]

Experimental Setups 3.1. Dataset In this study, several experimental settings are considered: •Training Set:A two-speaker mixture training set is simu- lated following previous work [1, 6, 2, 3]. The training set contains800speakers with20,000, utterances. Speakers ap- pearing in the test sets are excluded from the training data and SNR of interfering utt...

work page

[4] [4]

Results and Analysis In this section, linguistic knowledge from different pretrained models is evaluated and analyzed on various test sets. For a more comprehensive comparison, we also include two systems that do not explicitly incorporate pretrained models as external knowledge source but have been validated in previous studies as beneficial for enhancin...

work page

[5] [5]

Full occ

Note that A V-Sepformer with RoBERTa-B-L12 or WavLM- L-L24 were not exposed to any visual cue-impaired scenario, but still outperformed ImagineNET Sys.2 across all impair ra- tios, where the latter have seen visual full missing conditions during training. This finding suggests the proposed linguistic Table 3:SI-SDR performance under three visual impaired ...

work page

[6] [6]

Conclusion In this study, we propose to derive linguistic constraints from pre-trained language models and speech-language mod- els to optimize A V-TSE. Evaluated on multilingual, visual cue- impaired, and cross-domain test sets, the proposed paradigm consistently outperforms the vanilla A V-TSE backbone, while the linguistic constraints from the language...

work page

[7] [7]

62401377, Shenzhen Sci- ence and Technology Program (Shenzhen Key Laboratory, Grant No

Acknowledgements This work was supported by National Natural Science Foun- dation of China, Grant No. 62401377, Shenzhen Sci- ence and Technology Program (Shenzhen Key Laboratory, Grant No. ZDSYS20230626091302006), Shenzhen Science and Technology Research Fund (Fundamental Research Key Project, Grant No. JCYJ20220818103001002), Program for Guangdong Intro...

work page

[8] [8]

Muse: Multi-modal target speaker extraction with visual cues,

Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,”ICASSP 2021

work page 2021