{"id":"626da35c-342c-479d-99ca-5d738cd706dc","arxiv_id":"2412.11248","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A class-aware feature decoupling module with a background class plus co-occurrence and local-global fusion blocks improves weakly-supervised audio-visual video parsing.","lead":"This paper proposes a new network for audio-visual video parsing that separates each segment's features into per-event and background components before aggregating information across time and modalities. The method reports a new state-of-the-art on the LLP benchmark, with gains around two percentage points over the previous best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA margin may be inherited from the VALOR pseudo-label teacher rather than from the proposed decoupling architecture; the training loss directly regresses segment predictions and the co-occurrence map toward VALOR-generated labels.","rationale":"The reader's weakest assumption already pins the load-bearing risk to the VALOR-generated pseudo-labels. I agree: this is the largest threat to the central SOTA claim because it affects the whole training objective, not just one module. The concern is not that using pseudo-labels is illegitimate per se; it is that the comparison is confounded. Eq. 7 uses the pseudo-labels for segment-level BCE; Eqs. 10-11 additionally train the co-occurrence map on these labels, and the co-occurrence loss contributes the final 0.6% in the loss ablation. With no teacher-free or label-noise control, the reported 2.2% margin over VALOR cannot be cleanly attributed to the class-aware decoupling and enhancement mechanism. A secondary internal inconsistency (supplementary Sec. 6.2 states an 88% F1 for our model, contradicting the 65.4% reported everywhere else) further underscores the need for code and reproducibility, but the pseudo-label confound is the decisive issue. No verdict change is needed: the paper should remain conditional pending the control experiment.","tokens_in":16453,"tokens_out":12498,"duration_ms":111925,"concrete_test":"Retrain MM-CSE and the VALOR baseline under identical supervision: (i) full loss with VALOR pseudo-labels as in Eq. 12, and (ii) teacher-free, using only the video-level label Y in the basic loss and omitting the co-occurrence loss. Compare the average F-score margins on LLP. If the MM-CSE-over-VALOR margin under (ii) remains close to the 2.2% reported under (i), the architecture claim is supported; if it collapses or reverses, the headline result is substantially teacher distillation rather than decoupling/enhancement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported +2.2% average F-score over VALOR* (Table 1) is the central claim. However, the training objective in Eq. 7 supervises segment predictions with pseudo-labels generated by the VALOR teacher (Sec. 3.3), and the novel event co-occurrence loss Lec (Eqs. 10-11) builds its target matrix M from those same pseudo-labels and trains the learned co-occurrence map beta to match it. Thus the final result is a joint effect of the proposed architecture and a distillation signal from the very baseline it claims to surpass. The paper provides no control experiment using only the available video-level labels, no analysis of pseudo-label accuracy, and no noise-injection or label-corruption test to show the gains are robust to teacher errors. Table 4 shows the full loss set is needed for the final number (65.4 vs 64.8 without Lec), so the contribution of the co-occurrence loss is entangled with teacher label quality. Until this confound is removed, the margin cannot be attributed to the semantic-interference argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-CSE, a network for audio-visual video parsing (AVVP). It introduces a Class-Aware Feature Decoupling (CAFD) module that splits holistic audio/visual features into K event-specific features plus one background feature, and a Fine-Grained Semantic Enhancement (FGSE) module with Segment-wise Event Co-occurrence Modeling (SECM) and Local-Global Semantic Fusion (LGSF) blocks for intra- and cross-modal interactions. The model is trained with reconstruction, orthogonality, and event co-occurrence losses in addition to a basic binary cross-entropy loss. On the LLP dataset, the authors report state-of-the-art results: 65.4% average F-score with CLAP/CLIP features (vs. 63.2% for VALOR*) and 61.5% with VGGish/ResNet features (vs. 60.5% for VALOR). The paper includes ablations of each module and loss, efficiency comparisons, and qualitative visualizations.","tokens_in":16732,"tokens_out":7469,"duration_ms":57014,"significance":"The idea of decoupling semantically mixed features into class-wise event-specific and background features to reduce semantic interference is well-motivated and relevant to the AVVP task. The modular architecture is plausible, and the reported gains over the previous state of the art are non-trivial. The paper also provides an efficiency analysis and qualitative evidence of improved event co-occurrence modeling. However, the central claim is currently not fully pinned down because the training objective heavily relies on VALOR-generated pseudo-labels, including the co-occurrence loss, so the margin over VALOR cannot be cleanly attributed to the proposed architecture without a control experiment. In addition, I found concrete numerical inconsistencies between the main paper and the supplementary material, which need to be corrected. If the authors add the missing control and fix these issues, the paper would be a solid empirical contribution.","major_comments":[{"comment":"The central claim of a +2.2% improvement over VALOR* (Table 1) is confounded by the supervision signal. The training objective Lbasic in Eq. 7 uses BCE(pa, ŷ^a) + BCE(pv, ŷ^v), where ŷ^a and ŷ^v are segment-level pseudo-labels generated by the VALOR teacher, and the co-occurrence loss Lec (Eqs. 10–11) builds its target matrix M from these same pseudo-labels. Therefore the reported gains are a joint effect of the proposed architecture and the teacher's labeling quality. Table 4 shows that Lbasic alone (row #1) already reaches 64.3%, and adding Lec raises it to 65.4%; without a control that trains only with the video-level labels Y, or an analysis of pseudo-label accuracy/noise, the margin cannot be attributed to the semantic-interference reduction claimed in Sec. 1. Please add an experiment that trains with only video-level supervision (BCE(P,Y) only) and/or with corrupted or noised pseudo-labels to demonstrate robustness to teacher errors.","section":"Sec. 3.3 (Eqs. 7 and 10), Table 4"},{"comment":"The ablation results for the FGSE blocks are inconsistent between the main paper and the supplementary. In Table 3, \"blocks w/o SECM\" gives Avg 61.6 and \"blocks w/o LGSF\" gives 63.2; in Supplementary Table 9 the same two rows report Avg 63.2 and 61.6, respectively. The relative importance of the SECM and LGSF blocks is thus reversed across the two tables. Please correct the numbers and re-state the corresponding conclusion in Sec. 4.3, since the claim that both blocks contribute is not affected but the magnitude and direction of each contribution are.","section":"Table 3 vs. Supplementary Table 9"},{"comment":"The last sentence states that ETAVR achieves an F1 score of 72% on LLP, \"compared to 88% for our model.\" No 88% result appears in Table 1 or anywhere else in the paper; the best reported average is 65.4%. This discrepancy is serious because it misrepresents the comparison and can mislead readers about the state of the art. Please clarify the evaluation protocol and provide the correct numbers, or remove the comparison if it uses a different setting.","section":"Supplementary Sec. 6.2 (Discussion with More Related Works)"}],"minor_comments":[{"comment":"The notation for concatenation [;] is ambiguous: eF^m_bg is T×1×d2 and must be repeated K times before concatenation with eF^m_e, yielding a T×K×2d2 input to h. Please state this explicitly.","section":"Eq. 1"},{"comment":"Please specify that the L2 normalization in ∥·∥ is applied along the feature dimension (d2) and that the inner product is taken over that dimension to produce the T×K×1 relevance map.","section":"Eq. 4"},{"comment":"The statement \"The code will be publicly available\" is vague; please provide a link or a clear commitment, and report mean±std over at least three random seeds to support the numerical claims.","section":"Sec. 4.1 (Implementation details)"},{"comment":"The framework diagram is very dense; adding a legend or simplifying the notation would improve readability.","section":"Fig. 2"},{"comment":"The t-SNE visualization lacks axis labels and a count of embedded points; consider adding a quantitative cluster-quality metric (e.g., silhouette score) to substantiate the claim of intra-class compactness and inter-class separation.","section":"Sec. 4.4 (Fig. 5)"},{"comment":"The phrase \"the most recent work (Lai, Chen, and Yu-Chiang 2023)\" is inaccurate because the same paragraph cites later segment-wise pseudo-labeling works (Zhou et al. 2023, 2024c); please rephrase to avoid misleading the reader about chronology.","section":"Sec. 2 (Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The dependence on VALOR pseudo-labels is a known trend in this line of work; however, for a claim of \"new state-of-the-art\" over VALOR, the paper must isolate the architectural contribution from the teacher's labeling quality. The two numerical inconsistencies (main vs. supplementary ablations, and the 88% ETAVR claim) suggest the manuscript was not carefully checked and should be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note. I read MM-CSE (arXiv:2412.11248). Bottom line: it's a decent, honest piece of engineering that pushes the LLP benchmark by ~2 points, with a clear story and ablations that mostly check out. The main caveats are the missing code and error bars, a confusing supplementary number, and a pseudo-label dependency that deserves a dedicated analysis.\n\nWhat's genuinely new: the explicit background class in the decoupling and the segment-wise co-occurrence loss. The rest—class-wise decoupling, attention, local-global fusion—are familiar from MGN, LEAP, HAN, but the combination is new and it works. The ablation in Table 4 is the most informative part: with only the basic loss, the model already hits 64.3 average F-score, beating VALOR at 63.2. So the architecture itself is doing real work, not just the extra losses.\n\nThe stress-test worry about VALOR pseudo-labels is worth taking seriously, but it's not as clean a confound as it sounds. The model uses segment-level pseudo-labels from VALOR for the basic loss (Eq. 7) and for the co-occurrence target (Eq. 10). That means part of the reported gain could be inherited from the teacher. But the Lbasic ablation already surpasses VALOR, so the margin isn't purely a teacher artifact. Still, the paper should report pseudo-label accuracy, how much the final numbers move when you train on video-level labels only, and ideally a label-noise robustness test. Without code or multiple runs, the SOTA claim rests on a single seed.\n\nOne thing that stood out: the supplementary compares to ETAVR and states ETAVR gets 72% F1 on LLP while their method gets 88%. That 88% is wildly inconsistent with the 65.4% in Table 1. I suspect a typo or a mismatch in what metric/task is being reported. It doesn't affect the main results, but it's sloppy and a reviewer should catch it.\n\nAlso, in the supplementary comparison with LEAP under the same backbone, the gain is 62.4 vs 62.1—0.3 points, which is within noise. So the \"SOTA\" claim over the closest competitor is thin, though the 2.2% over VALOR* with CLAP/CLIP features is more substantial.\n\nWho is this for? Anyone working on weakly-supervised audio-visual parsing or using pseudo-labeling for temporal localization. It's a useful baseline and a clear example of how decoupling helps. I'd like to see the code and a robustness analysis before relying on the exact numbers.\n\nRecommendation: send it to review. The work is sound enough to merit referee time; it's incremental but not trivial, and the pseudo-label question is exactly what reviewers should probe.","headline":"Solid, incremental SOTA on LLP with a clean decoupling idea and a few reporting gaps; the pseudo-label confound is real but not fatal.","tokens_in":17178,"tokens_out":2338,"would_cite":true,"duration_ms":20506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that explicitly decoupling audio/visual segment features into event-specific and background streams before intra- and cross-modal interaction avoids semantic interference and yields new state-of-the-art audio-visual video…","keywords":["audio-visual video parsing","semantic interference","class-aware feature decoupling","event co-occurrence modeling","local-global semantic fusion","weakly supervised learning","pseudo-labels","background modeling"],"falsifier":"Train the same MM-CSE architecture using pseudo-labels with a known fraction of random label corruption (e.g., flipping 10%, 20%, and 40% of the segment-level event labels) and measure the resulting F-score; a method truly robust to semantic interference should degrade gracefully, while a method that is merely repeating the teacher's errors would show a large drop. Alternatively, on a random subset of LLP videos, ask human annotators to label a few segments and compare the pseudo-labels to human labels to quantify the noise floor and test whether the model's errors correlate with pseudo-label errors.","tokens_in":16318,"feed_emoji":"🎬","tokens_out":7429,"duration_ms":59144,"temperature":0.7,"pith_summary":"The paper takes on audio-visual video parsing—recognizing all events in each audio and visual segment of a video—and argues that the usual practice of feeding whole-segment features into attention layers causes semantic interference: a segment with only speech, for instance, can pick up violin semantics by attending to a segment containing a violin. To avoid this, the authors propose a network that first decouples each segment's mixed representation into one feature per event class plus a background feature, using separate linear projections. It then performs intra- and cross-modal enhancement at the class level, modeling which events tend to co-occur within a timestamp and matching each local segment to a global video representation of the same class. On the LLP benchmark, this MM-CSE network reaches an average F-score of 65.4% with CLAP/CLIP features (2.2 points above the previous state of the art, VALOR) and 61.5% with VGGish/ResNet features (1.0 point higher). The authors attribute the gains to class-aware decoupling and the explicit background stream.","feed_headline":"Class decoupling lifts audio-visual parsing to 65.4 F-score","feed_subtitle":"Separating each segment's events from its background before cross-modal fusion beats VALOR by 2.2 points.","key_machinery":"The load-bearing mechanism is the Class-Aware Feature Decoupling (CAFD) module paired with the Fine-Grained Semantic Enhancement (FGSE) module. CAFD projects each segment's mixed audio/visual feature through K+1 independent linear layers, producing K event-specific features and one background feature, and blends the background into each event stream via a learned weight; a reconstruction loss and an orthogonality loss keep the decoupling faithful and separable. FGSE then operates on these class-wise features: the Segment-wise Event Co-occurrence Modeling (SECM) block computes a softmax similarity matrix among class features at the same timestamp (and between the two modalities) to model which events tend to appear together, supervised by a co-occurrence loss that compares this map to the co-occurrence pattern in the pseudo-labels; the Local-Global Semantic Fusion (LGSF) block averages the class-wise features across time to form a global video representation per class, computes a cosine similarity between each local segment and that global representation, and fuses the matched global semantics into the local feature. L stacked FGSE layers refine the features, and the resulting class-level audio and visual representations are fed to an event parser. Everything is trained under the weak video-level label plus segment-level pseudo-labels.","core_discovery":"The paper's central assertion is that the bottleneck in audio-visual video parsing is semantic interference during intra- and cross-modal temporal interaction: when whole segments are mixed together, unrelated event semantics bleed across segments. The proposed solution decouples each segment's audio/visual feature into K event-specific features and one background feature, then runs class-level attention so that an event class aggregates information only from the same or co-occurring classes, both within a modality and across modalities. With the addition of a co-occurrence loss that supervises the learned class-relation map and a local-global fusion that pulls in informative global video semantics, the network sets a new state of the art: 65.4% average F-score on LLP with CLAP/CLIP features and 61.5% with VGGish/ResNet features, exceeding the VALOR baseline by 2.2 and 1.0 points respectively.","pith_inferences":["Untested in the paper: training MM-CSE with pseudo-labels from a different teacher or with oracle segment labels on a subset would reveal how much of the 2.2-point gain comes from the decoupling architecture versus inheriting the teacher's labeling tendencies.","Untested extension: replacing the fixed K linear layers with text-conditioned projections would allow parsing unseen event classes without retraining.","Testable variant: making the background blending weight per-class rather than a single scalar per segment could give finer control on segments that contain both an event and informative context."],"forward_implications":["If the decoupling claim holds, any audio-visual method that attends over holistic segment features is leaving performance on the table, and class-wise decoupling should benefit other video-understanding tasks where multiple events overlap.","The background class contributes a 1.2-point average gain over decoupling with only event classes (65.4% vs 64.2%), indicating that explicitly modeling non-event content is a large part of the improvement.","The co-occurrence loss adds 0.6 points on average (64.8% to 65.4%), showing that supervising the inter-class relation map with segment-level pseudo-labels improves multi-event recognition.","The same architecture gains 2.2 points with CLAP/CLIP features and 1.0 point with VGGish/ResNet features, so the benefit is not tied to a single feature extractor."],"supporting_citations":[{"why":"Supplies the segment-level pseudo-labels used as ground truth for supervision and for the co-occurrence loss, and provides the VALOR baseline the method surpasses.","marker":"(Lai, Chen, and Yu-Chiang 2023)"},{"why":"Introduces the LLP dataset, the task formulation, the MMIL pooling, and the HAN baseline that the paper builds upon and compares against.","marker":"(Tian, Li, and Xu 2020)"},{"why":"MGN, a prior method that uses multiple learnable class tokens; the paper contrasts its class-wise decoupling with MGN's grouping of event semantics.","marker":"(Mo and Tian 2022)"},{"why":"CLIP, the visual feature extractor used in the main experiments that report the 65.4% average F-score.","marker":"(Radford et al. 2021)"},{"why":"CLAP, the audio feature extractor paired with CLIP for the state-of-the-art results.","marker":"(Wu et al. 2023)"},{"why":"VGGish, the audio backbone used for the older-feature comparison that yields the 61.5% average F-score.","marker":"(Hershey et al. 2017)"},{"why":"ResNet-152, the visual backbone used with VGGish for the comparison results.","marker":"(He et al. 2016)"}],"fun_headline_variants":["Class-aware decoupling stops semantic interference in video parsing","Decoupling events from background lifts audio-visual parsing to 65.4","New audio-visual parsing SOTA: 65.4 F-score via class decoupling","Avoiding cross-modal semantic bleed boosts parsing to 65.4","Class-wise feature decoupling yields 65.4 F-score in video parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's segment-level supervision comes from pseudo-labels generated by a separate teacher model (VALOR), and those pseudo-labels also serve as ground truth for the co-occurrence loss; if those pseudo-labels are noisy or biased, the gains may partly reflect the teacher's errors rather than the model's own strength.","fun_headline_variants_meta":{"raw":{"variants":["Class-aware decoupling stops semantic interference in video parsing","Decoupling events from background lifts audio-visual parsing to 65.4","New audio-visual parsing SOTA: 65.4 F-score via class decoupling","Avoiding cross-modal semantic bleed boosts parsing to 65.4","Class-wise feature decoupling yields 65.4 F-score in video parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2886,"prompt_tokens":1019,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1770}},"tokens_in":635,"tokens_out":1867,"duration_ms":11885,"temperature":1.0,"reasoning_tokens":1770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:07:45.011849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same MM-CSE architecture using pseudo-labels with a known fraction of random label corruption (e.g., flipping 10%, 20%, and 40% of the segment-level event labels) and measure the resulting F-score; a method truly robust to semantic interference should degrade gracefully, while a method that is merely repeating the teacher's errors would show a large drop. Alternatively, on a random subset of LLP videos, ask human annotators to label a few segments and compare the pseudo-labels to human labels to quantify the noise floor and test whether the model's errors correlate with pseudo-label errors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the segment-level pseudo-labels used as ground truth for supervision and for the co-occurrence loss, and provides the VALOR baseline the method surpasses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the LLP dataset, the task formulation, the MMIL pooling, and the HAN baseline that the paper builds upon and compares against."},{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al","cited_arxiv_id":null,"evidence_quote":"CLIP, the visual feature extractor used in the main experiments that report the 65.4% average F-score."},{"cited_title":"P.; Gemmeke, J","cited_arxiv_id":null,"evidence_quote":"VGGish, the audio backbone used for the older-feature comparison that yields the 61.5% average F-score."}],"review_version":1}