{"id":"e630c932-d5c7-47a6-8e69-b19ca74ca5f0","arxiv_id":"2501.01518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VoiceFormer fuses text, video, and audio in a transformer to separate a target speaker, and stays robust when audio and video are misaligned by up to 200 ms.","lead":"This paper introduces VoiceFormer, which separates one person's speech from a noisy mix using either the text of what they are saying, their lip movements, or both. It reports top results on standard lip-reading video benchmarks and says the model still works when audio and video are out of sync.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim relies on custom filtered LRS2/LRS3 test sets and unmatched baselines; the 3-4 dB margin needs verification under a matched evaluation protocol before acceptance.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the state-of-the-art comparison is made on custom filtered test sets with baselines that are not trained or evaluated under identical conditions. This is the primary reason the paper's headline claim is not yet fully supported. The paper itself provides internal evidence for the architecture's coherence: text-conditioned separation is demonstrated through ablations, wrong-condition inputs degrade performance as expected, and the offset-robustness experiment includes a trained LSTM baseline. These support the methodological claims but not the comparative claim. A matched evaluation on official splits or released custom splits would settle the issue; until then, conditional acceptance remains the appropriate verdict.","tokens_in":13499,"tokens_out":6103,"duration_ms":63256,"concrete_test":"Ask the authors to release the exact filtered test sets and mixture-generation code, then rerun the Table 4 baselines (especially VisualVoice [21] and The Conversation [1]) under those exact mixtures and metrics; alternatively, evaluate VoiceFormer on the official LRS2/LRS3 test splits used by [1]/[14]/[21]. If the reported SDR margin (14.1 vs 10.8 on LRS2) persists under the matched protocol, the SOTA claim stands; if it shrinks to under 1 dB or reverses, the claim should be downgraded to 'competitive on a custom split'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that 140 hours of LRS2 and 1 hour of LRS3 were removed and that the authors constructed their own test sets (2515 and 3229 samples), rather than using the splits on which prior methods were evaluated. Table 4 then compares VoiceFormer to VisualVoice [21] (explicitly not fine-tuned on the synthetic task, footnote †), to Lee et al. [25] (explicitly on a different test set, footnote ◦), and to baselines whose training/test data are not shown to coincide with the new filtered splits. Since SDR/STOI/PESQ depend on mixture construction, the central claim of 'state-of-the-art on well-established benchmark datasets' is not yet supported by an apples-to-apples comparison. The architectural story is internally coherent and the ablations are informative, but the headline comparative number is load-bearing and rests on an unverified protocol. No code or data artifacts are released, so the custom test sets cannot be independently checked. The appropriate status is conditional acceptance pending a matched evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VoiceFormer, a U-Net with a transformer bottleneck that performs speech separation and enhancement by conditioning on raw audio jointly with visual lip movements, textual (phoneme) input, or both. The architecture concatenates modality-specific sequences in the transformer bottleneck with positional and modality encodings, allowing asynchronous and non-aligned conditioning. Experiments on synthetic mixtures from LRS2 and LRS3 report SDR improvements over prior audio-visual baselines, including 14.1/15.5 dB for audio-visual separation (A+V) and 13.1/14.1 dB for text-only conditioning (A+T). The paper also demonstrates robustness to audio-visual synchronization offsets and to missing or inconsistent modalities, and it ablates architecture choices (LSTM vs. transformer bottleneck, spectrogram vs. waveform input).","tokens_in":13704,"tokens_out":3323,"duration_ms":32029,"significance":"The proposed framework is genuinely novel in unifying conditioning on text, video, and audio in a single transformer-bottleneck architecture, and the text-conditioned separation results are a useful new capability. The robustness experiments (Figure 5, Table 3) are carefully designed and show a clear advantage of the transformer over an LSTM baseline under audio-visual misalignment. The internal ablations (Table 2) support the architectural choices. However, the headline claim of state-of-the-art performance on 'well-established benchmark datasets' is currently not supported by the evidence because the evaluation uses custom filtered test sets and unmatched baselines. If the authors provide a matched evaluation or appropriately scope the claim, the contribution would be solid.","major_comments":[{"comment":"The central SOTA claim is defined on test sets that are not the same benchmarks used by prior methods. Section 4.2 states that 140 hours of LRS2 and 1 hour of LRS3 were removed (retaining 57/197 and 439/440 hours), and that the authors constructed their own test sets of 2515 and 3229 samples. Since SDR, STOI, and PESQ depend on the mixture construction and utterance content, the numbers in Table 4 are not directly comparable to previously published results on the original LRS2/LRS3 splits. To support the Abstract's 'state-of-the-art on well-established benchmark datasets,' the authors must either evaluate on the same test splits used by the baselines or provide a matched protocol in which all models are trained and tested on identical filtered data.","section":"§4.2 (Data) and Abstract"},{"comment":"The baseline comparisons are not matched. VisualVoice [21] is explicitly not fine-tuned on the synthetic two-speaker LRS2 training set (footnote †), and Lee et al. [25] is evaluated on a different test set (footnote ◦). The Denoiser [12] row reports metrics identical to the 'Noisy input' row, indicating it fails to separate the two speakers, which is expected for an enhancement-only model; this does not provide a meaningful comparison for speaker separation. Because the baselines are not trained or evaluated under identical conditions, the reported margins (e.g., 3.3 dB over VisualVoice on LRS2) cannot be attributed to the proposed architecture or conditioning strategy. The authors should retrain or fine-tune all baselines on the same filtered training data and evaluate on the same test set, or, if this is not feasible, remove the global SOTA claim and present the results as demonstrating the method's internal improvements.","section":"§4.4, Table 4"},{"comment":"The paper does not release code, the list of removed sample IDs, or the exact scripts for constructing the synthetic mixtures and test splits. The filtering procedure ('a combination of diarization and background noise detection methods') is not specified precisely enough for independent reproduction. Without the exact test IDs and mixture parameters (e.g., SNR distribution, normalization strategy), the headline numbers in Table 4 cannot be independently verified. At minimum, the authors should provide a detailed description of the filtering pipeline, the test sample IDs, and the exact mixing recipe, and ideally release the evaluation code and pre-extracted features.","section":"§4.2 (Evaluation sequences) and Reproducibility"}],"minor_comments":[{"comment":"The text states 'we use N = 3 layers and h = 8 heads, with a model size of 532' and then says 'embedding dimensions across all modalities is set to 768.' The relationship between 532 and 768 is unclear; please clarify what 'model size' refers to (e.g., hidden dimension, feed-forward dimension, or total parameter count).","section":"§4.3 (Implementation details)"},{"comment":"The sentence 'The attention maps in Figure 3 reveal the correspondence between the audio tokens and the other modalities. attend to the features in the corresponding modalities.' contains a grammatical fragment. It should be rewritten as a single coherent sentence.","section":"§4.4 (Cross-modal attention)"},{"comment":"The ordering of modality labels is inconsistent: Table 1 and Table 2 use 'A+V+T' while Table 3 uses 'A+T+V' in the first column. Please unify the notation throughout.","section":"Table 3 and Table 1"},{"comment":"In Section 3.1, 'Similar to [13], we extract a representation of the noisy audio...' appears to refer to the Denoiser architecture [12] rather than the music source separation paper [13]. Please verify the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a promising architecture and a novel text-conditioning direction. The internal ablations and robustness studies are well done. The primary obstacle to acceptance is the unsupported global SOTA claim, which rests on a custom test protocol and unmatched baselines. If the authors re-run or re-present the comparisons under a matched protocol (or explicitly scope the claims to their own evaluation regime), the paper could become acceptable. I would not go so far as reject, as the core method and analysis appear sound, but the current version overstates the comparative evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it introduces text-conditioned speech separation, which is new relative to the audio-visual separation literature it cites. An A+T model pulling 13.1/14.1 dB SDR on LRS2/LRS3 is a real result, and the idea of conditioning on phoneme sequences without temporal alignment is worth taking seriously. The architecture, a raw-waveform U-Net with a transformer bottleneck that concatenates audio, visual, and text tokens, is a sensible way to fuse asynchronous modalities, and the ablations on missing or inconsistent inputs are informative. The robustness to audio-visual offsets, trained with augmentations up to 200 ms, is a practical property even if it is not surprising given the training procedure.\n\nThe soft spot is the comparison to prior work, which is the load-bearing claim in the abstract. Section 4.2 describes filtering LRS2 from 197 to 57 hours and constructing custom test sets of 2515 and 3229 samples. Prior methods like VisualVoice and Lee et al. were evaluated on different splits and mixture constructions, and the footnotes in Table 4 admit VisualVoice was not fine-tuned on the synthetic task and Lee et al. used a different test set. So the 3-4 dB margin over these baselines is not an apples-to-apples comparison, and the claim of state-of-the-art on well-established benchmarks is not yet supported. No code or data are released, so the custom test sets cannot be independently checked. A second, smaller issue is that the A+V+T model gives only a marginal gain over A+V (14.2 vs 14.1 on LRS2), which undercuts the pitch for multimodal fusion, though the robustness experiments with missing video do show a benefit.\n\nThe paper is internally coherent and the writing is honest about the main limitation (requiring text at inference). This deserves a serious referee, but the referee should insist on a matched evaluation: retraining or fine-tuning baselines on the same filtered splits, or reporting results on the original LRS2/LRS3 test partitions, and releasing the test set construction or the artifacts. If the SOTA claim survives that, it is a strong paper. If not, the text-conditioned capability alone is still a useful contribution worth publishing.","headline":"Text-conditioned separation is a genuinely new capability and the architecture is sound, but the state-of-the-art claim rests on custom test sets and mismatched baselines that need verification.","tokens_in":14209,"tokens_out":1286,"would_cite":true,"duration_ms":13848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VoiceFormer claims that a single transformer-bottleneck network can separate a target speaker using text, lip movements, or both, with state-of-the-art results on LRS2 and LRS3.","keywords":["speech separation","speech enhancement","text-conditioned separation","audio-visual fusion","transformer bottleneck","raw waveform","cocktail party effect","LRS2/LRS3"],"falsifier":"Evaluate VoiceFormer and the Table 4 baselines on a common, unfiltered LRS2/LRS3 test set with identical two-speaker mixtures; if the reported SDR lead shrinks or disappears, the state-of-the-art claim depends on the authors' custom filtering rather than on the architecture.","tokens_in":13299,"feed_emoji":"🎧","tokens_out":8506,"duration_ms":72663,"temperature":0.7,"pith_summary":"The paper presents VoiceFormer, a speech separation and enhancement model that conditions on the text of the target utterance, the speaker's lip movements, or both simultaneously. The central claim is that a single transformer-bottleneck architecture can fuse these heterogeneous cues even when they are not temporally aligned, and that text alone is a workable conditioning signal for pulling one voice out of a mixture. The authors report state-of-the-art separation scores on LRS2 and LRS3, and show that adding text to video makes the model substantially more tolerant of missing or corrupted video frames and of audio-video synchronisation offsets. A sympathetic reader should care because this opens a practical route to target-speaker isolation in settings where transcripts or subtitles are available but clean video is not.","feed_headline":"Text plus lip video separates speakers in noisy audio","feed_subtitle":"A single U-Net transformer handles text, video, or both, and stays accurate with 200 ms of audio-video desync.","key_machinery":"The load-bearing mechanism is a U-Net audio encoder-decoder operating directly on raw waveforms, with a multi-layer Transformer encoder inserted as the bottleneck. Audio, video, and phoneme-sequence embeddings are each augmented with a positional encoding and a learnable modality encoding, concatenated along the time dimension, and processed with full cross-attention, so every audio token can attend to the corresponding video tokens and text tokens. The positional encodings record timestamps for audio and video and word order for text, which is what allows conditioning on asynchronous streams without explicit alignment and lets the Transformer exploit language structure. The audio-side Transformer outputs are decoded back into a cleaned waveform, and the whole network is trained end-to-end with an L1 loss against the clean target.","core_discovery":"On its own terms, the paper's discovery is that the textual content of an utterance is a usable and previously underexplored cue for speaker separation. An audio-plus-text model reaches SDR 13.1 on LRS2 and 14.1 on LRS3, close to the audio-plus-video model at 14.1 and 15.5, while the full audio-plus-video-plus-text model reaches 14.2 and 15.5 and is markedly more tolerant when video frames are masked. The same design also removes the need for synchronised or same-rate audio and video streams: the model retains high SDR with offsets up to 200 ms, and the attention maps show it learns cross-modal alignment implicitly. The authors conclude that lip movements are a stronger cue than text, but that text provides robustness with little downside.","pith_inferences":["A natural extension not tested in the paper is ASR-conditioned separation: pair a real, error-prone ASR transcript with the audio stream. The word-removal experiments suggest graceful degradation, but real ASR error patterns were not measured, so the actual drop remains an open question.","Because conditioning uses only phoneme order rather than timestamps, the same architecture could likely accept other ordered symbolic inputs, such as subtitles, lyrics, or captions, as separation cues; the paper does not claim this.","The 200 ms desync tolerance suggests the method is a candidate for hearing aids or teleconferencing systems where wireless video and audio arrive with variable delays; a user study with genuinely out-of-sync recordings would be the next test."],"forward_implications":["Text-only conditioning separates two-speaker mixtures at SDR 13.1 (LRS2) and 14.1 (LRS3), confirming that content, not just voice or face identity, can drive separation.","Adding text to video raises SDR only slightly over video alone (14.2 vs 14.1 on LRS2) but makes the model markedly more tolerant of video frame masking.","The model keeps SDR above 12 with audio-video offsets up to 200 ms, while an LSTM-bottleneck baseline degrades sharply under the same misalignment.","With inconsistent text but correct video, the A+V+T model performs on par with video-only conditioning; with inconsistent video but correct text, it partially recovers instead of failing completely.","On the denoising task, all VoiceFormer variants match the audio-only Denoiser baseline, so the multi-modal machinery does not sacrifice standard enhancement performance."],"supporting_citations":[{"why":"Defines the task of speaker-independent audio-visual separation at the cocktail party whose synthetic mixing protocol this paper follows.","marker":"[14]"},{"why":"Supplies the raw-waveform U-Net encoder-decoder architecture and the audio-only Denoiser baseline for enhancement.","marker":"[12]"},{"why":"Provides the lip-movement-conditioned 'Conversation' baseline on LRS2/LRS3 that VoiceFormer is compared against.","marker":"[1]"},{"why":"Supplies the LSTM-based audio-visual baseline used in the architecture ablation and misalignment robustness comparison.","marker":"[4]"},{"why":"Provides the VisualVoice baseline with cross-modal consistency that is the strongest prior audio-visual method in Table 4.","marker":"[21]"},{"why":"Provides the LRS2 dataset of aligned audio, video, and text used for training and evaluation.","marker":"[9]"},{"why":"Provides the LRS3-TED dataset used for training and evaluation.","marker":"[2]"},{"why":"Provides the pretrained visual transformer network that extracts the lip-movement features.","marker":"[36]"},{"why":"Provides the Phonemizer library that converts input sentences to phoneme sequences for text conditioning.","marker":"[6]"},{"why":"Provides the DNS noise clips used to construct the denoising mixtures.","marker":"[35]"}],"fun_headline_variants":["Text cues rival lip reading for separating voices","Audio plus text separates speakers amidst noise and chaos","Transformer fuses text and video to pick out speech","Reading the words helps hear the voice at a party","Multi-modal model separates speakers using text and video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art comparison assumes the authors' filtered, synthetically mixed LRS2/LRS3 test sets are directly comparable to the test conditions used by prior baselines, even though the paper removes samples with noise or multiple speakers and builds its own evaluation splits.","fun_headline_variants_meta":{"raw":{"variants":["Text cues rival lip reading for separating voices","Audio plus text separates speakers amidst noise and chaos","Transformer fuses text and video to pick out speech","Reading the words helps hear the voice at a party","Multi-modal model separates speakers using text and video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2701,"prompt_tokens":861,"completion_tokens":1840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1768}},"tokens_in":477,"tokens_out":1840,"duration_ms":13792,"temperature":1.0,"reasoning_tokens":1768,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:26:50.699542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate VoiceFormer and the Table 4 baselines on a common, unfiltered LRS2/LRS3 test set with identical two-speaker mixtures; if the reported SDR lead shrinks or disappears, the state-of-the-art claim depends on the authors' custom filtering rather than on the architecture.","supporting_citations":[{"cited_title":"Real time speech enhancement in the waveform domain","cited_arxiv_id":null,"evidence_quote":"Supplies the raw-waveform U-Net encoder-decoder architecture and the audio-only Denoiser baseline for enhancement."},{"cited_title":"The conversation: Deep audio-visual speech enhance- ment","cited_arxiv_id":null,"evidence_quote":"Provides the lip-movement-conditioned 'Conversation' baseline on LRS2/LRS3 that VoiceFormer is compared against."},{"cited_title":"Self-supervised learning of audio-visual objects from video","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM-based audio-visual baseline used in the architecture ablation and misalignment robustness comparison."},{"cited_title":"VisualV oice: Audio- Visual Speech Separation with Cross-Modal Consistency","cited_arxiv_id":null,"evidence_quote":"Provides the VisualVoice baseline with cross-modal consistency that is the strongest prior audio-visual method in Table 4."},{"cited_title":"Lip reading sentences in the wild","cited_arxiv_id":null,"evidence_quote":"Provides the LRS2 dataset of aligned audio, video, and text used for training and evaluation."},{"cited_title":"Visual keyword spotting with attention","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained visual transformer network that extracts the lip-movement features."},{"cited_title":"Phonemizer: Text to phones transcription for multiple languages in python","cited_arxiv_id":null,"evidence_quote":"Provides the Phonemizer library that converts input sentences to phoneme sequences for text conditioning."},{"cited_title":"Interspeech 2021 deep noise suppression challenge","cited_arxiv_id":null,"evidence_quote":"Provides the DNS noise clips used to construct the denoising mixtures."}],"review_version":1}