{"id":"2c146076-14ed-4a45-912f-c1334e619ae8","arxiv_id":"2507.21448","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Concatenating speech-recognition and active-speaker visual embeddings improves audio-visual speech enhancement in low-SNR multi-speaker settings, and the CPU real-time system is released open source.","lead":"This paper introduces RAVEN, a real-time speech enhancement system that uses a speaker's face, seen on video, to isolate their voice and suppress interfering speakers and noise. It is the first openly released real-time audio-visual speech enhancement system, and it tests which pre-trained visual features work best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The embedding-ranking claim rests on PESQ/ESTOI differences of 0.001–0.008 with no error bars; without repeated seeds or significance tests, 'best' is not established.","rationale":"The reader's identified weakest assumption, the DeepFilterNet-generated ground truth, primarily threatens the validity of absolute scores. But the paper's central claim is a relative ranking of visual embeddings, and the observed inter-configuration differences are so small (PESQ deltas of 0.001–0.006, ESTOI deltas of 0.004–0.008) that they may be within run-to-run training noise. This is a more immediate load-bearing concern because it directly undermines the headline conclusion regardless of the target reference. It is also straightforwardly addressable with the open-sourced code by running multiple seeds and significance tests. The ground-truth issue remains a secondary concern that could interact with the ranking if DeepFilterNet artifacts favor certain embeddings, but the absence of variance reporting is enough to make the current empirical claims unsupported. The overall verdict stays CONDITIONAL: the paper's contributions (open-source real-time system and a useful embedding comparison framework) are valuable, but the specific best-embedding conclusions need statistical backing before they can be accepted as stated.","tokens_in":8653,"tokens_out":3598,"duration_ms":41953,"concrete_test":"Using the released code, retrain the four key configurations (AVHuBERT, VSRiW, TalkNet, AVH+T) with at least 3 random seeds each, evaluate on the same fixed test set, and report mean±std and paired Wilcoxon signed-rank tests for PESQ, SISDR, and ESTOI in the Tables 1–2 conditions. If the 95% confidence intervals of the top two configurations overlap on any metric used to justify the ranking, then the headline should be revised to 'no significant difference' or the ranking should be restricted to metrics with non-overlapping intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is comparative: concatenating AVSR and ASD embeddings is best in low-SNR multi-speaker conditions, and AVSR alone is best in noise-only conditions. The evidence in Tables 1–2 consists of very small differences. In Table 2, at −10 dB with three interfering speakers, AVH+T vs AVHuBERT gives PESQ 1.096 vs 1.097 (AVHuBERT is actually higher), SISDR −4.762 vs −5.018, ESTOI 0.336 vs 0.328; at −10 dB with one speaker, PESQ 1.231 vs 1.228 (Δ=0.003) and ESTOI 0.550 vs 0.543. In Table 1 noise-only, VSRiW vs AVHuBERT gives PESQ 2.063 vs 2.057 (Δ=0.006) and ESTOI 0.779 vs 0.775. No standard deviations, confidence intervals, paired significance tests, or test sample counts are reported. With only 5 training epochs, run-to-run stochastic variation can easily exceed these margins, especially for PESQ and ESTOI. The ranking claim is therefore not statistically supported as reported. Additionally, Table 2 only reports −10 dB and −5 dB, so the scope of 'low-SNR' over other SNR values is unclear. This concern is directly testable with the released code, making it the most load-bearing issue for the central empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RAVEN, a real-time audio-visual speech enhancement (AVSE) system that combines a causal audio stream with pre-trained visual embeddings from audio-visual speech recognition (AVSR) and active speaker detection (ASD) models. Four visual front-ends are evaluated (AV-HuBERT, VSRiW, TalkNet, LoCoNet) along with concatenations of the best from each task. The central empirical claim is that concatenating AVSR and ASD embeddings (AV-HuBERT+TalkNet) yields the greatest improvement in low-SNR, multi-speaker conditions, while AVSR embeddings alone (VSRiW) perform best in noise-only scenarios. The system reportedly operates on a CPU with a processing time of 28 ms per frame, and the code is released. The paper also reports generalization results on ViSPeR and MEAD datasets.","tokens_in":8998,"tokens_out":5083,"duration_ms":51965,"significance":"If the comparative results are robust, the paper would make a useful contribution by identifying which pre-trained visual features are most suitable for different acoustic conditions and by providing the first open-source real-time AVSE implementation. The reproducible code, video demonstration, and systematic comparison across visual encoders are explicit strengths. However, the central ranking claim rests on very small PESQ/ESTOI/SISDR differences without statistical validation, and the evaluation uses DeepFilterNet-denoised audio as ground truth, which threatens the validity of absolute and relative quality measurements. The significance is therefore conditional on addressing these methodological concerns.","major_comments":[{"comment":"The central claim that concatenating AVSR and ASD embeddings is 'best' in low-SNR, multi-speaker settings is not statistically supported. In Table 2, at -10 dB with three interfering speakers, AVH+T has PESQ 1.096 versus AVHuBERT 1.097 (AVHuBERT is actually higher), and at -10 dB with one speaker the PESQ difference between AVH+T and AVH+V is 0.000. ESTOI differences range from 0.001 to 0.008. No error bars, confidence intervals, or significance tests are reported, and all models are trained for only 5 epochs, so run-to-run stochastic variation can easily exceed these margins. The paper should either provide repeated-seed results with paired significance tests or explicitly state that the observed differences are within noise.","section":"Section 4.1, Tables 1 and 2"},{"comment":"The ground-truth clean speech is generated by denoising VoxCeleb2 utterances with DeepFilterNet, and all PESQ, SISDR, and ESTOI scores in Tables 1-3 are computed relative to this model-generated reference. This is a load-bearing methodological choice: if DeepFilterNet introduces artifacts, removes speech components, or behaves differently for certain accents or emotions, the reported improvements may reflect how well the model mimics DeepFilterNet's processing rather than true clean-speech quality. The paper should validate that DeepFilterNet outputs are acceptable clean references (e.g., by comparing against human-rated clean speech or using a separate clean-speech dataset) and discuss the potential bias this introduces.","section":"Section 3.1"},{"comment":"The paper states the input SNR range is -10 to 10 dB, but Table 2 only reports results at -10 dB and -5 dB for multi-speaker conditions. The central claim concerns 'low-SNR, multi-speaker environments,' which is only partially covered by these two points. Without results at 0, 5, or 10 dB, it is unclear whether the advantage of concatenating AVSR and ASD embeddings persists across the low-SNR range or is limited to the most adverse conditions. Please report results at intermediate SNRs or justify the omission.","section":"Section 3.1 and Table 2"},{"comment":"No audio-only baseline is reported. The paper motivates AVSE by the limitations of audio-only enhancement, but the reader cannot assess whether the visual embeddings provide any benefit over the audio stream alone. All evaluated models include both audio and visual inputs, so the contribution of the visual modality is not quantified. Adding an audio-only version of the fusion model (i.e., without the visual stream) would provide a critical reference point and strengthen the claim that the pre-trained visual representations are useful for AVSE.","section":"Section 4.1"}],"minor_comments":[{"comment":"The row label 'AVH+T' and the caption's 'AVHuBERT+TalkNet' are not defined in the table; please define all abbreviations in the caption or a table footnote.","section":"Table 2"},{"comment":"The real-time evaluation reports the processing time only for VSRiW (28 ms per frame). Please report processing times for all visual encoders to substantiate the claim that the system 'meets the real-time requirement for all pre-trained embeddings'.","section":"Section 2.3"},{"comment":"The test set size is not reported. Please specify the number of test utterances for VoxCeleb2, ViSPeR, and MEAD, as metric variability depends on sample size.","section":"Section 3.2"},{"comment":"The audio CNN is described as 'not causal' while the system is claimed to be real-time. Please clarify that the non-causality is limited to a two-frame lookahead (40 ms) that is included in the reported algorithmic latency.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty claim of being the first open-source real-time AVSE system is plausible but should be verified against recent literature (e.g., RT-LA-VocE in [18]); the lack of a comparative table with existing real-time AVSE systems is a concern. The use of DeepFilterNet-denoised audio as ground truth is the most serious methodological issue and may warrant a request for additional validation before acceptance. The editor may also encourage the authors to make the code and evaluation scripts available for reproducibility of the statistical analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two things, and one of them is solid. It releases an open-source, real-time audio-visual speech enhancement system that runs on a CPU (28 ms/frame on an Apple M3 Max), with code and a demo. That is the first such open release I know of, and it is a genuinely useful engineering contribution. The second thing is an ablation of four pre-trained visual front-ends—AV-HuBERT, VSRiW, TalkNet, LoCoNet—and the claim that concatenating AVSR and ASD embeddings gives the best low-SNR multi-speaker performance. That ablation is systematic, but the rankings are not established by the numbers as reported. The margins are tiny: PESQ differences of 0.001–0.008, ESTOI differences of 0.001–0.008, with no error bars, no significance tests, and no multi-seed runs. In Table 2, at −10 dB with three interferers, the concatenated model’s PESQ is actually lower than AV-HuBERT alone (1.096 vs 1.097). The stress-test note is right: without repeated seeds or a paired test, “best” is not supported.\n\nA second soft spot is the ground truth. The authors use DeepFilterNet-denoised VoxCeleb2 as the clean reference. That means every PESQ/SISDR/ESTOI score is relative to another model’s output, not to true clean speech. It is not circular, since DeepFilterNet is independent, but it does make the absolute scores and any characterization of “clean speech” approximate. The authors acknowledge this by saying the denoised audio is “reasonably clean upon inspection,” which is honest but still a limitation, especially if DeepFilterNet removes or alters speech content in some utterances.\n\nThe paper is otherwise careful. The generalization tests on ViSPeR and MEAD are a nice addition, even though only one encoder is used. The citation pattern is fine, and the claims about prior art lacking open implementations look accurate.\n\nOverall: this is a solid workshop- to ICASSP-level paper whose engineering release is the main value. The empirical claims need a revision—multiple seeds, confidence intervals, or paired tests—before they are convincing. I would send it to peer review; the code makes the re-analysis easy, and the authors should be given the chance to fix the statistics. If the ranking claim survives with uncertainty quantification, we learn something; if it does not, that is also informative. I would bring it to a reading group and would cite it if I worked in AVSE, mainly as a baseline system.","headline":"A useful open-source real-time AVSE release, but the embedding-ranking claims need error bars before they are convincing.","tokens_in":9492,"tokens_out":3550,"would_cite":true,"duration_ms":35176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAVEN, a real-time audio-visual speech enhancement system, shows that the optimal pre-trained visual front-end depends on the acoustic condition: AVSR plus ASD embeddings win in low-SNR multi-speaker mixtures, while AVSR embeddings alone…","keywords":["audio-visual speech enhancement","real-time speech enhancement","pre-trained visual embeddings","audio-visual speech recognition","active speaker detection","mask-based fusion","phase-sensitive loss","VoxCeleb2"],"falsifier":"Re-run the training and evaluation on a dataset with genuine clean references (for example, LRS3's original audio or studio-recorded corpora) and check whether the ranking of visual front-ends — VSRiW best in noise-only, AV-HuBERT plus TalkNet best in low-SNR multi-speaker — persists and whether the absolute gains remain when ground truth is not produced by DeepFilterNet.","tokens_in":8491,"feed_emoji":"🎙️","tokens_out":8863,"duration_ms":84668,"temperature":0.7,"pith_summary":"RAVEN is a real-time audio-visual speech enhancement system that reuses frozen visual embeddings from models trained for other speech tasks rather than learning video features from scratch. The paper's central claim is that the best visual front-end depends on the acoustic scenario: in noise-only mixtures the VSRiW audio-visual speech recognition embedding performs best, while in low-SNR mixtures with one or three interfering speakers the concatenation of AV-HuBERT (speech recognition) and TalkNet (speaker detection) embeddings gives the largest gains. The authors also claim their implementation is the first open-source AVSE system that runs in real time on a CPU, with per-frame processing under 40 milliseconds. If right, this gives practitioners a direct recipe for choosing pre-trained visual features in deployed AVSE, plus a reproducible baseline for future work.","feed_headline":"Low-SNR audio: lip-reading plus speaker detection wins","feed_subtitle":"Open-source RAVEN runs real-time on a CPU; noise-only audio is best handled by lip-reading features alone.","key_machinery":"The core mechanism is mask-based late fusion. An audio stream converts the 16 kHz mixture into a compressed complex spectrogram via STFT and a five-layer CNN; a video stream extracts embeddings from a frozen pre-trained visual encoder (AVSR or ASD) at 25 fps, upsampled to audio-frame rate. The two feature sets are concatenated and passed through a unidirectional LSTM and three fully connected layers that predict a magnitude mask, which is combined with the mixture phase to reconstruct the complex spectrogram. Training uses a phase-sensitive loss (MSE on both magnitude and compressed complex spectrogram). The real-time system buffers five video frames, giving a two-frame lookahead and 120 ms algorithmic latency, while each frame's processing stays under the 40 ms streaming interval.","core_discovery":"The central discovery is that visual embeddings trained for audio-visual speech recognition (AVSR) and active speaker detection (ASD) transfer to speech enhancement, and that their relative value is condition-dependent. In noise-only conditions, VSRiW (an AVSR model) achieves the best PESQ, SISDR, and ESTOI among all tested front-ends. In low-SNR, multi-speaker conditions, concatenating AV-HuBERT (AVSR) and TalkNet (ASD) embeddings outperforms either alone, with the largest relative gains in SISDR and ESTOI. The authors attribute the benefit to complementary information: continuous lip-movement features from AVSR plus target-speaker activity cues from ASD. They also report that the model generalizes to multilingual in-the-wild data but degrades on emotionally expressive speech, and that the system meets real-time constraints on a CPU with an algorithmic latency of 120 ms caused by the two-frame video lookahead.","pith_inferences":["A testable extension: replace AV-HuBERT with an audio-only HuBERT or a differently noise-augmented AVSR model to determine whether its multi-speaker advantage comes from the AVSR objective or from its diverse noise augmentation during training.","The condition-dependent ranking suggests a deployment strategy: a system that switches front-ends based on estimated SNR and number of active speakers could get the best of both regimes.","Because all metrics are computed against DeepFilterNet-denoised audio, re-scoring on studio-clean targets would show how much of the reported ranking is tied to the denoiser's definition of clean.","The 120 ms latency is dominated by video lookahead; a system that trades accuracy for latency by reducing the lookahead could open the design to more interactive applications such as live captioning or hearing aids."],"forward_implications":["A pragmatic rule for deployment: use VSRiW-style AVSR embeddings in noise-only conditions and AV-HuBERT plus TalkNet embeddings in low-SNR, multi-speaker conditions.","Real-time AVSE is achievable on a CPU with frozen pre-trained visual features, so the system can be reproduced and integrated without specialized hardware.","Cross-task concatenation (AVSR + ASD) helps more in multi-speaker mixtures than same-task concatenation, suggesting that activity-level cues and phonetic lip-motion cues are complementary.","The released code provides a public baseline for real-time AVSE, allowing future work to benchmark against a common implementation.","Generalization tests indicate the approach transfers across languages but not across strong emotional expressions, marking emotion robustness as the next obstacle."],"supporting_citations":[{"why":"Causal real-time AVSE baseline that the paper positions against; it does not release a public implementation.","marker":"[5]"},{"why":"Causal AVSE that estimates a binary mask in noise-only settings; one of the earlier real-time systems RAVEN compares with in kind.","marker":"[17]"},{"why":"Real-time low-SNR AVSE using Emformer and a causal vocoder; another baseline lacking public code.","marker":"[18]"},{"why":"Provides the AV-HuBERT pre-trained embeddings used as one AVSR front-end.","marker":"[22]"},{"why":"Provides the VSRiW pre-trained embeddings, the best front-end in noise-only tests.","marker":"[24]"},{"why":"Provides TalkNet ASD embeddings, used in the best-performing concatenation.","marker":"[25]"},{"why":"Provides LoCoNet ASD embeddings, the weaker ASD front-end.","marker":"[26]"},{"why":"DeepFilterNet generates the denoised ground-truth speech used in training and evaluation.","marker":"[31]"}],"fun_headline_variants":["Lip-reading features excel in noise, but add speaker ID for multi-speaker","Condition-dependent visual cues: AVSR for noise, AVSR+ASD for babble","RAVEN: real-time AV speech enhancement with transferable visual embeddings","Open-source RAVEN: AVSR beats ASD in noise, pair them for interference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training targets and all evaluation scores are measured against DeepFilterNet-denoised VoxCeleb2 audio, not against true clean studio recordings; the paper notes this produces 'reasonably clean speech for most utterances upon inspection' but does not establish that this reference is faithful to the original clean speech.","fun_headline_variants_meta":{"raw":{"variants":["Lip-reading features excel in noise, but add speaker ID for multi-speaker","Condition-dependent visual cues: AVSR for noise, AVSR+ASD for babble","RAVEN: real-time AV speech enhancement with transferable visual embeddings","Open-source RAVEN: AVSR beats ASD in noise, pair them for interference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000349,"raw_usage":{"total_tokens":1881,"prompt_tokens":891,"completion_tokens":990,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":904}},"tokens_in":507,"tokens_out":990,"duration_ms":11409,"temperature":1.0,"reasoning_tokens":904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:45:13.181063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the training and evaluation on a dataset with genuine clean references (for example, LRS3's original audio or studio-recorded corpora) and check whether the ranking of visual front-ends — VSRiW best in noise-only, AV-HuBERT plus TalkNet best in low-SNR multi-speaker — persists and whether the absolute gains remain when ground truth is not produced by DeepFilterNet.","supporting_citations":[{"cited_title":"We systemat- ically analyze how visual embeddings from audio-visual speech recognition (A VSR) and active speaker detection (ASD) im- pact A VSE performance","cited_arxiv_id":null,"evidence_quote":"Causal real-time AVSE baseline that the paper positions against; it does not release a public implementation."},{"cited_title":"A Novel Real-Time, Lightweight Chaotic-Encryption Scheme for Next- Generation Audio-Visual Hearing Aids,","cited_arxiv_id":null,"evidence_quote":"Causal AVSE that estimates a binary mask in noise-only settings; one of the earlier real-time systems RAVEN compares with in kind."},{"cited_title":"Lip- Reading Driven Deep Learning Approach for Speech Enhance- ment,","cited_arxiv_id":null,"evidence_quote":"Real-time low-SNR AVSE using Emformer and a causal vocoder; another baseline lacking public code."},{"cited_title":"CochleaNet: A robust language-independent audio-visual model for real-time speech enhancement,","cited_arxiv_id":null,"evidence_quote":"Provides the AV-HuBERT pre-trained embeddings used as one AVSR front-end."},{"cited_title":"Exploring Tradeoffs in Models for Low-Latency Speech Enhancement,","cited_arxiv_id":null,"evidence_quote":"Provides the VSRiW pre-trained embeddings, the best front-end in noise-only tests."},{"cited_title":"Phase- sensitive and recognition-boosted speech separation using deep recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides TalkNet ASD embeddings, used in the best-performing concatenation."},{"cited_title":"On The Compensation Between Magnitude and Phase in Speech Separation","cited_arxiv_id":"2108.05470","evidence_quote":"Provides LoCoNet ASD embeddings, the weaker ASD front-end."},{"cited_title":"LoCoNet: Long-Short Context Network for Active Speaker Detection,","cited_arxiv_id":null,"evidence_quote":"DeepFilterNet generates the denoised ground-truth speech used in training and evaluation."}],"review_version":1}