{"id":"640feca5-5a78-44cc-8691-6b99330f5a20","arxiv_id":"2509.06598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A Cross-Modal Conformer that fuses CLAP audio and OWL-ViT visual embeddings with a CNN-Conformer SELD backbone, trained on large synthetic data, ranks second in DCASE 2025 Task 3 Track B.","lead":"This paper describes a system that pinpoints and names sound events in stereo videos by fusing audio and visual semantic embeddings from pre-trained models. It ranked second in the DCASE 2025 Task 3 challenge, with most gains coming from synthetic pre-training and engineering refinements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Development-set results are selected on the same test split used for evaluation, so the claimed gains from CLAP/OWL-ViT and the full recipe are optimistically biased; the challenge rank alone does not isolate the semantic contribution.","rationale":"The reader's weakest assumption--that synthetic audio-visual corpora faithfully proxy real DCASE content--is plausible, but the more immediate threat to the paper's central claim is internal validity. The paper explicitly reports selecting the best model via F1 on the development test split, the same split on which Tables 1 and 2 are evaluated. This makes the quantitative support for CLAP/OWL-ViT contributions optimistically biased and confounds the ablation comparison. The challenge ranking is external and credible evidence that the overall pipeline works, so I would not reject the paper; however, the causal attribution to semantic embeddings should be treated as conditional on re-validation with a proper held-out split and, ideally, released code and error bars. This is consistent with the reader's CONDITIONAL verdict, so I do not move the verdict. The reader partially anticipated this issue in the rationale but did not make it the weakest assumption, hence 'partial' agreement.","tokens_in":9799,"tokens_out":7397,"duration_ms":68227,"concrete_test":"Re-run the Table 1 ablation (SE, SE+CLAP, SE+CLAP+OV) and the Table 2 systems with early stopping and checkpoint selection performed on a held-out validation split carved from the DCASE development training split, leaving the development test split untouched until the final evaluation, repeated over at least five seeds. If the SE to SE+CLAP+OV F1 gap and the Table 2 versus Table 1 gains shrink below the run-to-run standard deviation, the paper's causal claims about semantic embeddings and the full recipe are not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that after pre-training, the model was 'fine-tuned for another 80 epochs on the real development set, selecting the best model via F1 score on the test split.' Tables 1 and 2 then report F1, DOAE, and related metrics computed on this same development test split. Using the evaluation split for checkpoint selection means the reported improvements--for example, SE to SE+CLAP (+0.9 F1), SE+CLAP to SE+CLAP+OV (+0.6 F1), and the large Table 2 gains over the ablation--are not unbiased estimates and may partly reflect selection on noise. No error bars, multiple seeds, or code are provided, so the magnitude of this bias cannot be assessed. The official second rank in DCASE 2025 Task 3 Track B is external evidence that the full pipeline is competitive, but it does not support the attribution to CLAP and OWL-ViT because the full system differs from the ablation in several confounded factors: synthetic pre-training, ILD and stpACC input features, weighted on/off-screen loss, YOLOv11-Pose post-processing, and majority-vote ensembling. Thus the load-bearing condition for the causal claim is not only synthetic-data realism; it is also that the internal comparisons are statistically valid, and the reported model-selection procedure violates that condition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses stereo sound event localization and detection with distance estimation (3D SELD) in regular video content, in the setting of the DCASE 2025 Challenge Task 3 Track B. The authors propose an audio-visual architecture that fuses conventional SELD audio embeddings from a CNN-Conformer encoder with semantic embeddings from CLAP (audio) and OWL-ViT (visual), using a modified Conformer module called the Cross-Modal Conformer (CMC). The training recipe includes pre-training on large synthetic audio and audio-visual corpora (audio 5k and audio-visual 2k), ILD and stpACC input features, on/off-screen weighted loss, YOLOv11-Pose visual post-processing, and majority-vote ensembling. Ablation results in Table 1 and full-system results in Table 2 are reported on the DCASE development set, and the full system is stated to have achieved second rank in the challenge Track B. The central claim is that integrating CLAP and OWL-ViT embeddings improves stereo SELD performance and that the full training recipe produces a top-tier system.","tokens_in":10053,"tokens_out":4392,"duration_ms":41817,"significance":"If the reported results hold, the paper describes a strong, practically relevant system for audio-visual 3D SELD and demonstrates a sensible way to inject language-aligned semantic embeddings into a classical SELD pipeline. The second-place rank in the externally evaluated DCASE 2025 Challenge Task 3 Track B is credible evidence that the full recipe is competitive. The paper also contributes a detailed description of synthetic data curation, including a publicly listed data release, and a careful engineering study of features, augmentation, post-processing, and ensembling. However, the internal attribution of the gains to CLAP and OWL-ViT is weakened by the model-selection procedure, by the absence of repeated runs or error bars, and by the mismatch between the ablation setup and the full-system setup. The challenge rank validates the whole pipeline but does not isolate the contribution of the semantic embeddings, which is the paper's central architectural claim.","major_comments":[{"comment":"Section 3.1 states that after fine-tuning on the real development set, the best model is selected via F1 score on the test split, and Tables 1 and 2 report all metrics on this same development-test split. Selecting checkpoints on the evaluation split and then reporting numbers from that split makes the reported increments — for example, SE to SE+CLAP (+0.9 F1), SE+CLAP to SE+CLAP+OV (+0.6 F1), and the much larger Table 2 gains over Table 1 — optimistically biased estimates. No repeated runs or confidence intervals are provided, so the magnitude of the selection bias cannot be assessed. The external challenge ranking supports the competitiveness of the full pipeline but does not support the specific attribution of these improvements to CLAP and OWL-ViT. Please report results on a held-out validation split or provide multiple-seed results with error bars.","section":"Section 3.1, Tables 1 and 2"},{"comment":"The ablation study in Table 1 uses a different training setup from the full method in Table 2: the ablation uses only log-mel spectrograms, skips the audio 5k and audio-visual 2k pre-training stages, trains on 15,000 synthetic audio-visual clips, and does not use the weighted on/off-screen loss, visual post-processing, or ensembling. The full method adds ILD and stpACC features, large-scale pre-training, and several engineering refinements. Consequently, the large performance gap between Tables 1 and 2 cannot be attributed specifically to synthetic pre-training or to the semantic embeddings; the contributions of the components are confounded. If the central claim is that the Cross-Modal Conformer with CLAP and OWL-ViT helps, the relevant evidence is the Table 1 ablation, but that evidence is subject to the model-selection problem noted above. I recommend either adding a full-recipe ablation that isolates CLAP/OWL-ViT under the final training setup or explicitly narrowing the causal claims to the Table 1 setting.","section":"Section 3.2.1 vs. Section 3.2.2 and Tables 1 and 2"},{"comment":"The claim that using synthetic-data normalization statistics for system (2) improves generalization is supported by challenge evaluation numbers stated in the text (29.4% vs. 42.5% F1) but not included in any table or referenced to a public leaderboard page. Since the development-set numbers for systems (1) and (2) are nearly identical (45.7 vs. 46.0 F1), the evaluation-subset comparison is the only evidence for this claim. Please tabulate the official evaluation-subset scores for all submitted systems or provide a direct citation to the published challenge results, so this claim can be verified.","section":"Section 3.3.2"}],"minor_comments":[{"comment":"The non-linear spatial re-framing transformation is described only verbally; please provide a formula or a reference, since it is a design choice that reportedly affects performance.","section":"Section 2.1.3"},{"comment":"The relationship between the 2,000 one-minute FOA audio-visual clips and the \"+60,000 audio-visual clips\" is not spelled out; clarify how the 5-second segmentation, rotations, and augmentation produce this count, as was done for the audio 5k dataset.","section":"Section 2.3"},{"comment":"The text states that the visual post-processing step affects on/off-screen accuracy and F1/on but leaves other metrics unchanged; Tables 2 and 2.1 are consistent with this, but the identical Acc values across some rows (e.g., systems (3) and (3.1)) make the claimed effect hard to see. A short explanation of why Acc is unchanged while F1/on improves would help.","section":"Section 3.3.2"},{"comment":"No training code is provided, although a data link is given; given the complexity of the recipe, releasing code or a more detailed configuration would improve reproducibility.","section":"General"},{"comment":"The notation \"Alpha\" and \"Beta\" in the figure caption is clear in context, but the mathematical symbols α and β are not defined in the text; please define them in Section 2.1.2.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong challenge system description, and the external ranking is valuable evidence. The main risk is that the internal attribution of gains to the semantic embeddings is not statistically supported as currently presented. Please ask the authors to re-run the ablations with a proper validation split or multiple seeds, or to soften the attribution claims to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competent challenge-system paper with an externally credible result (2nd at DCASE 2025 Task 3 Track B), but the internal evidence that CLAP and OWL-ViT contribute is weaker than the abstract implies. The paper's real value is the recipe: synthetic pre-training, Cross-Modal Conformer fusion, ILD/stpACC features, and ensembling.\n\nWhat's new: integrating frozen CLAP and OWL-ViT embeddings into a CNN-Conformer SELD backbone through a cross-modal conformer is a sensible and not-previously-published combination. The synthetic data pipeline (5k FOA clips, 2k audio-visual clips with 360-Indoor backgrounds and NitroFusion-generated foregrounds) is detailed and reproducible in spirit, and the authors provide data on Zenodo. The baseline comparisons and ablations are clearly laid out. A second-place ranking on held-out challenge data is real external evidence that the full system works.\n\nThe soft spots are mostly around statistical support. The biggest is in Section 3.1: the best model is selected via F1 on the development test split, and then Tables 1 and 2 report numbers on that same split. That makes the reported gains optimistic, and with no error bars or multiple seeds we can't tell whether +0.9 F1 from CLAP or +0.6 from OWL-ViT is signal or noise. The ablation also uses a different setup (no pre-training, only mel spectrograms) than the final system, which mixes in many confounds—synthetic pre-training, ILD/stpACC, weighted loss, pose post-processing, ensembling—so the challenge rank doesn't isolate the semantic contribution. The synthetic-to-real gap is a domain assumption, but not an unreasonable one given the rank. No code is provided, which limits reproduction; the Zenodo data helps but not enough to rerun. These are fixable weaknesses, not fatal ones. The paper is honest about what it did; the framing just overreaches when it says the gains 'underscore the effectiveness' of semantic embeddings.\n\nWho it's for: anyone working on SELD with video, especially DCASE participants wanting a state-of-the-art baseline. It deserves peer review, but as a system paper, not a causal study. For a journal, I'd want the authors to re-run with a proper validation split or multiple seeds; for a workshop, it's fine.","headline":"Solid DCASE system paper with a real second-place result; the internal evidence for semantic-embedding gains is weak due to test-split model selection and no error bars.","tokens_in":10581,"tokens_out":3113,"would_cite":true,"duration_ms":26522,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing language-aligned audio and visual embeddings through a Cross-Modal Conformer lifts stereo sound event localization above challenge baselines, with the full recipe ranking second in the 2025 challenge.","keywords":["stereo sound event localization and detection","audio-visual fusion","Cross-Modal Conformer","CLAP audio embeddings","OWL-ViT visual embeddings","synthetic data pre-training","source distance estimation","on/off-screen classification"],"falsifier":"Re-run the full recipe with the same architecture and compute but replace the synthetic pre-training corpora with an equal number of real clips from the development set; if the F1 gap between that model and the reported 48.0% is small, then synthetic data realism is not carrying the result, and the paper's attribution of its gains to pre-training would be undercut.","tokens_in":9589,"feed_emoji":"🎧","tokens_out":12073,"duration_ms":95590,"temperature":0.7,"pith_summary":"This paper claims that stereo sound event localization and detection in ordinary videos can be improved by injecting semantic information from language-aligned models into a conventional audio localization backbone. The authors add frozen CLAP audio embeddings and frozen OWL-ViT visual embeddings to a CNN-Conformer SELD encoder, merging them through a new Cross-Modal Conformer block that uses cross-attention to let one modality query another. On the DCASE 2025 stereo SELD development set, each added semantic stream raises the F1 score, and the full system, trained on large synthetic audio and audio-visual corpora with left-right channel swapping, keypoint-based visual post-processing, and a four-model ensemble, reaches 48.0% F1 and ranks second in the challenge's Track B. If true, the result shows that pretrained, language-aligned embeddings are a practical way to give SELD systems semantic understanding without training large multimodal models from scratch.","feed_headline":"Language-aligned embeddings push stereo sound localization to 2nd place","feed_subtitle":"Cross-modal fusion plus large synthetic pre-training beats the 2025 challenge baselines by wide margins.","key_machinery":"The central mechanism is the Cross-Modal Conformer (CMC), a modified Conformer block that fuses two modality streams. In a standard Conformer, the same stream passes through feed-forward, multi-head self-attention, and convolutional modules; the CMC instead runs two feed-forward layers in parallel on two modalities, called Alpha and Beta, and replaces self-attention with multi-head cross-attention in which queries come from Alpha and keys and values come from Beta. The paper uses one CMC to fuse the CNN-Conformer SELD encoder's embeddings with frozen CLAP audio embeddings, and a second, two-layer CMC to fuse the resulting audio representation with per-patch OWL-ViT visual tokens, linearly projected from 768 to 512 dimensions. Letting the audio stream repeatedly query semantic information from each pretrained encoder while keeping those encoders frozen is what carries the multimodal fusion. A final feed-forward module maps the fused representation to multi-ACCDDOA vectors for up to three simultaneous tracks, trained with class-wise ADPIT loss.","core_discovery":"The central claim is that a SELD system for stereo audio, where only azimuth in the range of ±90°, distance, and on/off-screen status are predicted, benefits from semantic embeddings extracted by frozen language-aligned encoders. The paper's ablation shows the SELD encoder alone reaches 34.6% F1, adding CLAP raises it to 35.5%, and adding OWL-ViT raises it to 36.1%, against a 26.8% audio-visual baseline. With the full recipe, including ILD and stpACC input features, pre-training on 5,000 synthetic FOA clips and 2,000 synthetic audio-visual clips, left-right channel swapping, a weighted on/off-screen loss, human keypoint post-processing, and an ensemble of four systems, the model reaches 48.0% F1 on the development set and placed second in the DCASE 2025 Challenge Task 3 (Track B). The paper reads this as evidence that semantically rich, language-aligned embeddings and a fusion module tuned for cross-modal queries carry information that spatial-temporal audio features alone do not provide.","pith_inferences":["One step the paper does not take is replacing the 1 fps, temporally pooled OWL-ViT stream with a video-language model that keeps motion; since temporal dynamics are otherwise left to the audio encoder, a temporally aware visual encoder is a natural testable extension that could improve on/off-screen accuracy.","The finding that synthetic-domain normalization statistics generalize better than fine-tuning-domain statistics suggests a broader transfer-learning recipe: keep the preprocessing distribution fixed across pretraining and fine-tuning.","Because CLAP alone supplies most of the semantic gain, an inexpensive practical consequence is that stereo or even mono audio-only SELD could benefit from language-aligned audio embeddings, potentially reducing reliance on microphone arrays while retaining competitive detection.","The soft cross-fade and indoor background selection address a specific synthetic-to-real visual gap; a direct next experiment is to measure sensitivity to the background distribution by swapping indoor for outdoor synthetic visuals and observing the change in on/off-screen accuracy."],"forward_implications":["The two-step fusion recipe means pretrained language-aligned encoders can be added to an existing SELD backbone without fine-tuning them, so semantic SELD is achievable with only a small trainable fusion module.","Because the stereo channels carry no inter-channel time differences, the paper demonstrates that ILD for direction plus stpACC for distance, combined with semantic context, is a viable feature set for stereo 3D SELD.","Pre-training on the large synthetic audio and audio-visual corpora is the largest single driver of the reported gains, and using synthetic-domain input normalization generalizes better than using development-set statistics.","Audio-only variants with CLAP embeddings match or slightly beat the audio-visual variants on F1 and DOAE, while the visual stream is what enables on/off-screen classification.","The ensemble rule, which requires agreement from at least two systems except for Bell and Knock classes, yields further consistent gains in F1, DOAE, RDE, and on/off accuracy."],"supporting_citations":[{"why":"supplies the frozen contrastive language-audio encoder whose embeddings carry the audio semantic stream","marker":"[15]"},{"why":"supplies the frozen open-vocabulary vision encoder whose per-patch tokens carry the visual stream","marker":"[16]"},{"why":"defines the Conformer block that the paper adapts into the Cross-Modal Conformer","marker":"[20]"},{"why":"provides the soundscape simulator used to generate the 5,000 synthetic FOA clips for audio pre-training","marker":"[13]"},{"why":"provides the video synthesizer used to build the 2,000 audio-visual clips for full-model pre-training","marker":"[14]"},{"why":"introduces the stpACC autocorrelation features used for source distance estimation","marker":"[21]"},{"why":"defines the class-wise ADPIT loss used to train the multi-ACCDDOA outputs","marker":"[23]"},{"why":"supplies the CNN-Conformer SELD backbone and the original audio channel swap augmentation extended here to stereo","marker":"[17]"}],"fun_headline_variants":["Semantic embeddings lift stereo sound localizer to 2nd","Language-aligned fusion nudges stereo SELD to 48% F1","Cross-modal conformer secures 2nd in DCASE stereo SELD","Semantic language models sharpen stereo event localization","Semantic fusion ranks 2nd in DCASE stereo SELD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the synthetic audio and audio-visual clips used for pre-training are realistic enough that the large gains they produce on the development set also transfer to the challenge evaluation data.","fun_headline_variants_meta":{"raw":{"variants":["Semantic embeddings lift stereo sound localizer to 2nd","Language-aligned fusion nudges stereo SELD to 48% F1","Cross-modal conformer secures 2nd in DCASE stereo SELD","Semantic language models sharpen stereo event localization","Semantic fusion ranks 2nd in DCASE stereo SELD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4292,"prompt_tokens":1047,"completion_tokens":3245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":3156}},"tokens_in":663,"tokens_out":3245,"duration_ms":20203,"temperature":1.0,"reasoning_tokens":3156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:15:16.201792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full recipe with the same architecture and compute but replace the synthetic pre-training corpora with an equal number of real clips from the development set; if the F1 gap between that model and the reported 48.0% is small, then synthetic data realism is not carrying the result, and the paper's attribution of its gains to pre-training would be undercut.","supporting_citations":[{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,","cited_arxiv_id":null,"evidence_quote":"supplies the frozen contrastive language-audio encoder whose embeddings carry the audio semantic stream"},{"cited_title":"Simple open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"supplies the frozen open-vocabulary vision encoder whose per-patch tokens carry the visual stream"},{"cited_title":"Conformer: Convolution-augmented transformer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"defines the Conformer block that the paper adapts into the Cross-Modal Conformer"},{"cited_title":"Spatial Scaper: A library to simulate and augment soundscapes for sound event localization and detection in realistic rooms,","cited_arxiv_id":null,"evidence_quote":"provides the soundscape simulator used to generate the 5,000 synthetic FOA clips for audio pre-training"},{"cited_title":"Multi-ACCDOA: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training,","cited_arxiv_id":null,"evidence_quote":"defines the class-wise ADPIT loss used to train the multi-ACCDDOA outputs"},{"cited_title":"A four-stage data augmentation approach to ResNet- Conformer based acoustic modeling for sound event localization and detection,","cited_arxiv_id":null,"evidence_quote":"supplies the CNN-Conformer SELD backbone and the original audio channel swap augmentation extended here to stereo"}],"review_version":2}