{"id":"137bb1ea-0c1b-4140-b64d-c3932925a78c","arxiv_id":"2411.14153","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MVANet applies multi-stage audio-guided video attention to improve audio-visual 3D sound event localization and source distance estimation on STARSS23.","lead":"This study builds a neural network that combines 360-degree video with spatial audio to locate sound sources and estimate their distance. It reports improved accuracy on the 2024 DCASE benchmark for audio-visual sound event localization with distance estimation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority over Wang [31] under ENH2+PT is not supported by a controlled comparison: the baseline's results come from the prior challenge submission with a different, non-SDE pre-trained audio model, and no error bars are provided.","rationale":"I identify the uncontrolled comparison as the most load-bearing concern because the central claim is explicitly a comparison to the authors' previous DCASE 2024 system. For that claim to hold, the baseline must be evaluated under the same training conditions; otherwise the reported gains could be caused by the different pre-trained audio model or by random seed variation. The reader's weakest assumption about AVPS spatial consistency is a valid reproducibility concern, but it would affect both systems similarly and would not directly threaten the relative comparison. The proposed test of re-training Wang [31] with the same pre-trained model and multiple seeds would settle both the confound and the uncertainty issue. The novelty and architecture claims are less load-bearing because Table I provides a no-pretraining ablation showing MVANet also improves over Wang [31] without pre-training, although those differences are also single-run and would benefit from the same multi-seed test. Overall, the paper is a capable challenge system but the headline comparison needs stronger evidence.","tokens_in":8401,"tokens_out":13538,"duration_ms":121630,"concrete_test":"Re-implement the early-fusion ResNet-Conformer baseline (Wang [31]) in the MVANet codebase; initialize its audio branch with the same ENH3 pre-trained RC model used for MVANet; train under the ENH2+PT setup with the same AVPS augmentation and at least five random seeds; evaluate on the STARSS23 dev-test set. If the re-trained baseline achieves metrics within the variance of MVANet (0.567/12.37°/0.239) or better, the central claim of outperformance fails. If MVANet wins consistently across all seeds, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table II compares MVANet ENH2+PT (F20: 0.567, DOAE: 12.37°, RDE: 0.239) with 'Wang [31] ENH2+PT' (F20: 0.550, DOAE: 13.16°, RDE: 0.250). The text identifies Wang [31] as the previous DCASE 2024 challenge system, not a model re-trained in the same codebase with the same initialization. Two variables change at once: the proposed multi-stage attention versus early fusion, and the pre-trained audio model (Wang [31] used a DCASE 2023 model without source-distance pre-training, while MVANet uses an AO-RC model trained with the SDE task under ENH3). The RDE gain from 0.250 to 0.239 is much larger than the no-pretraining ablation in Table I (0.251 to 0.250), suggesting the improvement is largely due to the new pre-training rather than the attention module. Additionally, no error bars or multiple seeds are reported, so the DOAE difference of 0.79° may fall within run-to-run variation. To establish the central claim, the authors need to re-train Wang [31] under identical conditions, including the same pre-trained audio model, AVPS implementation, and random seeds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MVANet, a multi-stage video attention network for audio-visual 3D sound event localization and detection (SELD) with source distance estimation. The network uses audio features from multiple stages to attend to relevant spatial regions in video frames, and introduces a 'source Cartesian coordinate' output that represents DOA and distance by multiplying normalized DOA vectors by the estimated distance. Training uses the STARSS23 dev-set with audio-visual pixel swapping (AVPS) and simulated data augmentation, plus initialization from a pre-trained audio-only ResNet-Conformer. On the STARSS23 dev-test set, the best configuration (ENH2+PT) reports F20°/1 = 0.567, DOAE = 12.37°, RDE = 0.239, outperforming the authors' previous top-ranked DCASE 2024 system (F20°/1 = 0.550, DOAE = 13.16°, RDE = 0.250) without model ensemble.","tokens_in":8678,"tokens_out":3195,"duration_ms":30204,"significance":"If the reported results are robust, the paper makes a modest but useful contribution to the DCASE community: it demonstrates that multi-stage audio-guided video attention can improve audio-visual 3D SELD, and it proposes a simple output representation that jointly encodes DOA and distance. The use of an official challenge dataset and comparison with the authors' own prior challenge submission are appropriate. However, the central claim of superiority over the previous system is weakened by a lack of controlled comparison (two variables change simultaneously), by the absence of error bars or multiple seeds, and by an underspecified augmentation step that may create contradictory spatial supervision. The paper is technically plausible but needs additional experiments and clarifications before the claim can be accepted.","major_comments":[{"comment":"The AVPS augmentation is described as generating new video frames by flipping and rotating the original frames [11], but the paper never states whether the first-order ambisonic audio channels and the DOA/distance labels are transformed correspondingly (e.g., azimuth negation for horizontal flips, elevation and channel permutation for rotations). If the audio and labels are not co-transformed, the augmented training pairs contain contradictory spatial supervision, which would directly corrupt the SCE branch and undermine the very learning the central claim relies on. The authors must specify the exact co-transformations applied to the FOA audio, the Cartesian coordinate labels, and the distance labels, or correct the augmentation procedure.","section":"Section II-C"},{"comment":"The main comparison between MVANet (ENH2+PT) and Wang [31] (ENH2+PT) is not controlled. The Wang [31] row is taken from the prior DCASE 2024 challenge submission, which used a pre-trained audio model optimized for SED and DOA estimation without source distance pre-training, whereas MVANet uses an AO-RC model trained with the SDE task under ENH3. Thus two variables change at once: the fusion/attention architecture and the pre-trained audio encoder. The RDE improvement from 0.250 to 0.239 is larger than the corresponding no-pretraining gap (0.251 to 0.250 in Table I), suggesting that much of the gain may be attributable to the new pre-training rather than the proposed attention module. To establish the stated claim, the authors need to retrain the Wang [31] system under identical conditions (same pre-trained audio model, same AVPS implementation, same data splits, same seed protocol) or run an ablation that controls for pre-training.","section":"Table II and Section III-C"},{"comment":"All reported metrics are single-run results on the official dev-test split, with no error bars or significance testing. The improvement in DOAE from 13.16° to 12.37° is a 0.79° difference that may easily fall within run-to-run variation for models of this complexity, especially given the stochastic nature of training. The paper should report means and standard deviations over at least three random seeds, or otherwise justify that the differences are statistically meaningful.","section":"Tables I and II"},{"comment":"The ablation comparing MVANet with Wang [31] does not control for parameter count, computational cost, or training recipe beyond the stated setup. The reader cannot tell whether the observed improvements come from the proposed multi-stage attention or simply from a larger/more expressive model. A controlled ablation that matches capacity or reports parameter counts and FLOPs is needed to support the claim that the attention mechanism itself is the source of the gains.","section":"Table I and Section III-B"}],"minor_comments":[{"comment":"The phrase 'an novel' in the Introduction should be corrected to 'a novel'.","section":"Abstract and Introduction"},{"comment":"In the sentence following Eq. (5), the term 'DOA loss L2' is inaccurate; Eq. (7) is the source coordinate estimation (SCE) loss, not a DOA loss. Please rename it to 'SCE loss' or 'coordinate loss'.","section":"Section II-B"},{"comment":"The text contains the typo 'EHN2' in 'under the “EHN2” setup'; this should read 'ENH2' consistently with the table captions.","section":"Section III-B"},{"comment":"Implementation details are incomplete for reproducibility: batch size, total number of training epochs, optimizer hyperparameters other than the learning-rate upper bound, and the exact tri-stage scheduler settings are not reported.","section":"Section III-A"},{"comment":"The baseline row 'Baseline BASE' (F20°/1 = 0.113) is not defined in the text or in the table caption; please specify which system this refers to (e.g., the official challenge baseline or a local implementation).","section":"Section III-B and Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a challenge technical report expanded into a letter. The main technical idea (multi-stage audio attention for AV-SELD) is reasonable and within the scope of the journal. The main risk is not novelty but experimental rigor: the comparison against the authors' own prior system is uncontrolled, and the augmentation description is ambiguous. If the authors can supply the controlled comparison, co-transformation details, and multi-seed results, the paper would be publishable. I also note that the manuscript does not yet provide the promised code, which would help reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid challenge-system writeup that would be fine at a workshop, but the central comparison is confounded and the paper overstates the novelty of its output format. Worth a referee, not worth treating as a clean SOTA result yet.\n\nWhat's genuinely useful: applying multi-stage audio-guided video attention to 3D SELD, with careful use of simulated data, AVPS, and audio pretraining. The ablations in Table I show consistent gains over their own early-fusion baseline across setups, which is real evidence that the architecture does something. The paper is clearly written and the visualizations are honest about residual error.\n\nThe problems are in the attribution and the reproducibility details. The Table II comparison with Wang [31] under ENH2+PT changes two things at once: the fusion architecture and the pretrained audio model. The text confirms Wang [31] used a DCASE 2023 pretrained model without source-distance training, while MVANet uses an audio model pretrained with SDE under ENH3. The ablation without pretraining (Table I) shows a much smaller improvement (F20 0.441 to 0.458; RDE essentially unchanged), which suggests a good chunk of the Table II gain comes from the new pretraining, not from multi-stage attention. To support the 'outperforms our top-ranked method' claim, they need to retrain Wang [31] with the same audio pretraining and augmentations. Also, no error bars or seeds anywhere; a 0.79 degree DOAE difference and 0.017 F-score difference may well be within run-to-run variance.\n\nThe AVPS augmentation is another missing link. They say they flip and rotate video frames but never say whether the FOA channels and DOA/distance labels are transformed correspondingly. A horizontal flip without negating azimuth would feed the model contradictory spatial supervision. This is likely just an underspecified detail, but for a paper whose claim is about spatial localization, it has to be stated.\n\nOn novelty: the 'novel SED-SCE output representation' is the same Cartesian-coordinates-from-DOA-times-distance idea cited as [8]. The paper is open about this in the introduction, so calling it novel in the abstract and conclusion is overclaiming.\n\nBottom line: useful system description for the DCASE community, with a credible architecture and honest ablations, but the headline comparison is confounded and the reproducibility details are incomplete. I'd send it to peer review at a workshop or conference and ask for the baseline retrain, error bars, and the AVPS label handling. Not a desk reject, not a clean accept.","headline":"A competent DCASE system paper whose headline gain is real but partly attributable to a changed pretraining setup, not just the attention module.","tokens_in":9230,"tokens_out":2666,"would_cite":false,"duration_ms":24514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MVANet uses multi-stage audio-guided video attention to beat its top-ranked DCASE 2024 system on 3D sound localization, without ensembling.","keywords":["sound event localization and detection","source distance estimation","audio-visual fusion","multi-stage attention","Cartesian coordinate representation","data augmentation","STARSS23","DCASE 2024"],"falsifier":"Inspect the released code or the AVPS data loader for one training sample: take a horizontally flipped video frame and check whether the ambisonic W-X-Y-Z channels have been correspondingly re-ordered and the azimuth label negated. If the spatial channels and labels are not transformed to match the flipped frame, the training signal is spatially inconsistent and the reported gains cannot be attributed cleanly to the proposed architecture.","tokens_in":8187,"feed_emoji":"🎯","tokens_out":10106,"duration_ms":82399,"temperature":0.7,"pith_summary":"MVANet claims that for audio-visual 3D sound event localization and detection—identifying what is sounding, from which direction, and at what distance—letting audio features from multiple network stages re-weight video frames picks out the parts of the frame that actually produce sound. The paper couples this with an output format that multiplies the normalized direction-of-arrival vector by the source distance, so a single regression target, the true Cartesian coordinates, carries both DOA and distance information. With audio-visual pixel swapping, simulated audio-visual training data, and audio-only pre-training, the full system reaches an F-score of 0.567, a DOA error of 12.37°, and a relative distance error of 0.239 on the STARSS23 dev-test set. That beats the authors' own previous top-ranked DCASE 2024 system on all three metrics with a single model and no ensemble. If the claim holds, multi-stage cross-modal attention and a unified coordinate target are sufficient ingredients to improve 3D SELD.","feed_headline":"MVANet beats top 3D sound localization system without ensembling","feed_subtitle":"Multi-stage audio-video attention and a unified coordinate target cut DOA error to 12.37° and distance error to 0.239.","key_machinery":"The load-bearing component is the multi-stage attention module: at each of four stages, an attention function $Att(V_t, A_t)$ projects audio and video feature vectors into a shared dimension, sums them, and applies a sigmoid to generate spatial attention weights that re-weight the video feature map, focusing it on sounding objects. The other key ingredient is the source Cartesian coordinate output format: the normalized DOA unit vector is multiplied by the source distance, so the SCE branch regresses $(x, y, z)$ coordinates whose direction encodes azimuth and elevation and whose Euclidean norm encodes distance. The system is trained with a two-term loss (binary cross-entropy for SED, mean squared error for SCE) and uses two augmentations: audio-visual pixel swapping that flips and rotates video frames, and simulated audio-visual data generated by convolving sound events with measured room impulse responses.","core_discovery":"The central discovery is that applying an audio-guided attention module at four different stages of a ResNet-Conformer audio-visual network, instead of fusing audio and visual features once, more fully exploits the location cues embedded in video frames for both direction and distance estimation. The SCE branch predicts the true Cartesian coordinates of each active source, with the vector's direction giving the DOA and its length giving the distance, and trains under a single MSE loss alongside the SED branch's binary cross-entropy. On the STARSS23 dev-test set, the ENH2+PT configuration reaches an F20°/1 of 0.567, a DOAE of 12.37°, and an RDE of 0.239, surpassing the authors' own first-place DCASE 2024 system (0.550, 13.16°, 0.250) without model ensembling.","pith_inferences":["The same multi-stage audio-guided attention could be transplanted to other audio-visual spatial tasks, such as sound separation or active speaker localization, wherever video should attend to the sounding source.","Regressing true Cartesian coordinates suggests an additional evaluation metric: a Euclidean error on the $(x,y,z)$ vector, which may expose failures that angle-only and distance-only metrics obscure.","Whether the AVPS co-transform is actually implemented or not, checking it would give a concrete test of how much of the reported gain comes from augmentation versus the multi-stage attention design."],"forward_implications":["A single MVANet model, without ensemble, outperforms the previous top-ranked system, so ensembling is not required to reach the state of the art in this challenge setup.","Unifying DOA and distance into one Cartesian-coordinate regression target makes source distance estimation a direct extension of prior ACCDOA-style outputs, simplifying multi-task 3D SELD.","Audio-only pre-training on simulated spatial data transfers to the audio-visual model, reducing reliance on the scarce real audio-visual recordings.","The multi-stage attention mechanism can be inserted into existing audio-visual backbones to improve spatial localization even when the visual content is cluttered with non-sounding objects."],"supporting_citations":[{"why":"Supplies the ACCDOA output format that the new Cartesian coordinate representation extends.","marker":"[4]"},{"why":"Introduces audio-guided video attention, the mechanism MVANet generalizes across multiple stages.","marker":"[17]"},{"why":"Provides the STARSS23 audio-visual dataset and the official dev-test split used for all evaluations.","marker":"[23]"},{"why":"Describes the audio-visual simulation method that generates additional training pairs.","marker":"[24]"},{"why":"Supplies the simulation library and the audio-only pre-training data generation pipeline.","marker":"[25]"},{"why":"Provides the audio channel swapping augmentation that AVPS extends to video frames.","marker":"[28]"},{"why":"The authors' previous top-ranked DCASE 2024 system, the main baseline MVANet compares against and surpasses.","marker":"[31]"}],"fun_headline_variants":["MVANet's multi-stage video attention beats top 3D SELD baseline","MVANet cuts DOA error to 12.37° without ensembling","MVANet achieves best 3D SELD results on STARSS23 without ensemble","Multi-stage video attention lifts 3D SELD past top DCASE entry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest assumption is that the audio-visual pixel swapping augmentation transforms video frames and the spatial audio and labels together: the paper flips and rotates video frames but never states whether the first-order ambisonic channels and the DOA/distance labels are co-transformed (for example, azimuth negation on horizontal flips); if they are not, the augmented training pairs carry contradictory spatial supervision.","fun_headline_variants_meta":{"raw":{"variants":["MVANet's multi-stage video attention beats top 3D SELD baseline","MVANet cuts DOA error to 12.37° without ensembling","MVANet achieves best 3D SELD results on STARSS23 without ensemble","Multi-stage video attention lifts 3D SELD past top DCASE entry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3190,"prompt_tokens":946,"completion_tokens":2244,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2156}},"tokens_in":562,"tokens_out":2244,"duration_ms":14237,"temperature":1.0,"reasoning_tokens":2156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:27:55.039077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released code or the AVPS data loader for one training sample: take a horizontally flipped video frame and check whether the ambisonic W-X-Y-Z channels have been correspondingly re-ordered and the azimuth label negated. If the spatial channels and labels are not transformed to match the flipped frame, the training signal is spatially inconsistent and the reported gains cannot be attributed cleanly to the proposed architecture.","supporting_citations":[{"cited_title":"ACCDOA: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the ACCDOA output format that the new Cartesian coordinate representation extends."},{"cited_title":"Audio-visual event localization in unconstrained videos,","cited_arxiv_id":null,"evidence_quote":"Introduces audio-guided video attention, the mechanism MVANet generalizes across multiple stages."},{"cited_title":"STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,","cited_arxiv_id":null,"evidence_quote":"Provides the STARSS23 audio-visual dataset and the official dev-test split used for all evaluations."},{"cited_title":"Spatial scaper: A library to simulate and augment soundscapes for sound event localization and detection in realistic rooms,","cited_arxiv_id":null,"evidence_quote":"Supplies the simulation library and the audio-only pre-training data generation pipeline."},{"cited_title":"A four-stage data augmentation approach to resnet-conformer based acoustic model- ing for sound event localization and detection,","cited_arxiv_id":null,"evidence_quote":"Provides the audio channel swapping augmentation that AVPS extends to video frames."},{"cited_title":"The nerc-slip system for sound event localization and detection with source distance estimation of DCASE 2024 challenge,","cited_arxiv_id":null,"evidence_quote":"The authors' previous top-ranked DCASE 2024 system, the main baseline MVANet compares against and surpasses."}],"review_version":1}