{"id":"47d61bae-7ff4-49f6-9687-54a2c780e49c","arxiv_id":"2505.02096","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TeMTG fuses CLIP/CLAP text embeddings with audio and visual features and applies K-hop graph attention to achieve state of the art segment-level event parsing on the LLP dataset.","lead":"This paper presents TeMTG, a weakly supervised model that combines text descriptions from pretrained CLAP and CLIP with a temporal graph to parse audio and visual events in videos. The authors report state of the art results on the LLP benchmark, though their own ablation shows the temporal graph component can reduce performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation (Table 2) shows removing the multi-hop temporal graph component improves or ties every metric relative to the full model, so the named contribution is not supported and the SOTA attribution to MTG is confounded.","rationale":"The reader's conditional verdict is appropriate, but their stated weakest assumption (pseudo-label text alignment) is not the most load-bearing concern. The paper's own Table 2 directly falsifies the contribution of the multi-hop temporal graph: the full model is worse than or equal to text enhancement alone on every metric. This is stronger than a pseudo-label noise risk because it is evidenced by the authors' own controlled experiment, not a hypothetical failure mode. The SOTA claim about the full model is numerically true against prior work, but the narrative that MTG is a beneficial component is contradicted. The reader did mention the ablation inconsistency in their rationale, but did not foreground it as the primary weakness, hence partial agreement. The verdict remains conditional: the paper can be accepted if the authors either demonstrate that MTG helps (with proper seeding and statistical testing) or honestly reposition the contribution around text enhancement, with MTG removed or clearly labeled as non-beneficial. This is a significant issue, but not necessarily fatal, because the empirical numbers and the clear availability of a better configuration (TE-only) make a path to revision straightforward. No ad hominem is intended; the concern is purely about the consistency of the experimental evidence with the claims.","tokens_in":8019,"tokens_out":5321,"duration_ms":69240,"concrete_test":"Run the full Table 2 ablation (CoLeaF†, w/o TE, w/o MTG, full TeMTG) with at least 5 random seeds using the authors' exact training recipe, reporting mean±std for all metrics. Perform paired tests (e.g., paired bootstrap or Wilcoxon signed-rank) between w/o MTG and full TeMTG on Event@AV and segment-level A. If w/o MTG remains ≥ TeMTG on Event@AV and most metrics with non-overlapping confidence intervals, then MTG cannot be credited with any improvement. Also perform a hyperparameter sweep over hop count K (e.g., 1,2,4,6) and GAT layers to see whether any configuration of the full model beats w/o MTG; if none does, the claimed combination is not justified and the paper should be repositioned with text enhancement as the primary contribution and MTG as an optional module or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the multi-hop temporal graph (MTG), yet Table 2 provides a controlled comparison that contradicts this contribution. Removing MTG (i.e., text enhancement alone) strictly improves or ties full TeMTG on every reported metric: segment-level A 76.5 vs 74.4, V 72.9 vs 72.9, AV 62.4 vs 62.0, Type@AV 70.6 vs 69.8, Event@AV 75.7 vs 74.1; event-level A 66.6 vs 61.9, V 69.0 vs 69.0, AV 53.4 vs 53.2, Type@AV 63.1 vs 61.4, Event@AV 66.0 vs 62.2. Thus MTG never improves any metric and costs up to 4.8 points (event-level A). Section 3.2 attributes the +5.3 segment-level Event@AV gain over NREP to 'our multi-hop temporal graph better captures cross-segment temporal dependencies,' but the TE-only ablation already exceeds NREP by a larger margin (75.7 vs 68.8, +6.9), showing the gain comes from text enhancement, not MTG. Section 3.3 acknowledges that 'some TeMTG indicators are slightly lower than using only the TE mechanism,' but a 4.8-point degradation is not slight. Because the abstract and introduction claim that the combination of text enhancement and multi-hop temporal graph modeling is the source of SOTA performance, this is an internal inconsistency in the central argument. Additionally, the best configuration found in the paper's own ablation (w/o MTG) is not the model proposed, so the paper does not even present the strongest model it found.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TeMTG, a weakly supervised audio-visual video parsing (AVVP) framework that combines text-enhanced feature fusion with a multi-hop temporal graph (MTG). The text-enhancement component uses CLAP/CLIP to generate segment-level pseudo labels, converts them into natural-language descriptions, and fuses the resulting text embeddings with audio and visual features via modality-specific MLPs. The multi-hop temporal graph connects each segment to neighbors within K hops and applies multi-head graph attention to model temporal dependencies. Experiments on the LLP dataset report state-of-the-art results on several segment-level and event-level metrics, with the largest gains on audio F1 and Event@AV. An ablation study compares the full model against variants without text enhancement and without the MTG.","tokens_in":8388,"tokens_out":3767,"duration_ms":43551,"significance":"If the reported results hold, the text-enhancement mechanism is practically significant: on the standard LLP benchmark it yields large improvements over recent methods, such as a 5.3-point gain in segment-level Event@AV over NREP (74.1 vs. 68.8), and the evaluation is conducted against external baselines without circularity. However, the paper's central contribution is the multi-hop temporal graph, and the paper's own ablation contradicts the claim that this component is beneficial. The full TeMTG model is outperformed or tied by the text-enhancement-only variant on every segment-level metric and on most event-level metrics, including a 4.8-point drop in event-level audio F1 and a 3.8-point drop in event-level Event@AV. The attribute of the SOTA gains to MTG in Section 3.2 is therefore not supported by the controlled comparison, and the paper does not present the best model it found. The core idea of text enhancement is promising and well motivated, but the paper as written does not establish the value of the proposed multi-hop temporal graph, so the contribution claim needs substantial revision.","major_comments":[{"comment":"The ablation shows that removing the multi-hop temporal graph (the w/o MTG variant) strictly improves or ties the full TeMTG on every segment-level metric and on all event-level metrics except visual F1, with the largest degradation being 4.8 points on event-level audio F1 (61.9 vs. 66.6) and 3.8 points on event-level Event@AV (62.2 vs. 66.0). This contradicts the abstract's claim that the combination of text enhancement and multi-hop temporal graph modeling is the source of SOTA performance, and it undermines the attribution in Section 3.2 of the +5.3 Event@AV gain over NREP to the multi-hop temporal graph, since the w/o MTG variant exceeds NREP by an even larger margin (75.7 vs. 68.8). The authors must either revise the contribution claim, present a configuration or analysis in which MTG helps, or reposition the contribution around text enhancement alone.","section":"Section 3.3, Table 2"},{"comment":"The sentence \"Some TeMTG indicators are slightly lower than using only the TE mechanism\" understates the observed degradation: compared to w/o MTG, the full model drops by 4.8 points on event-level audio F1, by 3.8 points on event-level Event@AV, and by 1.6 points on segment-level Event@AV, and it never improves on any reported metric. Because this sentence is the only discussion of the central negative result, the paper does not currently reconcile the proposed architecture with its own controlled comparison, which is an internal inconsistency that must be resolved before the SOTA claim can be accepted.","section":"Section 3.3"},{"comment":"The definition of the adjacency matrix is internally inconsistent: the text says \"1, if 0 ≤ j−i ≤ K; 0, otherwise,\" but the equation assigns 0 in both cases, making the graph edgeless and the subsequent GAT propagation vacuous. This appears to be a typographical error, but as written it prevents the multi-hop temporal graph from being implemented as described, so the equation must be corrected to assign 1 to the within-K entries.","section":"Section 2.3, Equation (6)"},{"comment":"The SOTA claim is phrased too strongly: TeMTG is below NREP on segment-level AV parsing (62.0 vs. 64.4), below NREP on event-level audio (61.9 vs. 62.8), and below both NREP and LSLD+ on event-level AV (53.2 vs. 57.6). The paper should state precisely which indicators are improved, quantify the trade-offs, and avoid the unqualified phrase \"achieves SOTA performance\" in the introduction and conclusion.","section":"Section 3.2, Table 1"}],"minor_comments":[{"comment":"Equation (4) for visual fusion uses z^a_t in the MLP expression, but it should use z^v_t to be consistent with the definition of the concatenated visual feature.","section":"Section 2.2, Equation (4)"},{"comment":"The symbol \"bs\" in R^{bs×T×d} is not defined; it should be \"b\" for consistency with the audio case and the later notation.","section":"Section 2.2, Equation (3)"},{"comment":"The venue name \"NuerIPS\" is misspelled; it should be \"NeurIPS.\"","section":"References, [12]"},{"comment":"The caption says \"second results highlighted in text,\" but the table does not visibly indicate second-best entries; either underline them or adjust the caption.","section":"Table 1 caption"},{"comment":"The paper fixes the number of hops K=4 without any sensitivity analysis; given that the ablation shows the MTG component is harmful or neutral, a sweep over K would help clarify whether the negative effect is particular to this setting or fundamental.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is internal: the paper's own ablation (Table 2) shows the named contribution, the multi-hop temporal graph, never improves the full model and degrades several metrics substantially. The authors acknowledge only \"slightly lower\" performance, which is an understatement. This is not a question of external consensus or circularity; it is a mismatch between the proposed architecture and the evidence. The results for text enhancement alone are strong and could support a revised paper, but as submitted the contribution claim is not supported. I recommend major revision rather than rejection because the issue is fixable by reframing, additional analysis, or re-running experiments, and the underlying benchmark results are valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline is that the paper's advertised contribution doesn't survive its own ablation. The multi-hop temporal graph (MTG) is supposed to be a central piece, but Table 2 shows that removing it strictly improves or ties every single metric at both segment and event level. On event-level audio, the gap is 4.8 points (66.6 vs 61.9). Section 3.2 credits MTG for the +5.3 Event@AV gain over NREP, yet the text-enhancement-only variant beats NREP by 6.9. So the gain comes from text enhancement, not the graph.\n\nWhat is actually good here: the text-enhancement recipe works. Converting CLAP/CLIP pseudo-labels into natural-language descriptions, re-embedding them with the same encoders, and fusing via a small MLP gives large improvements over CoLeaF and lifts several LLP metrics beyond previous SOTA. That is a legitimate, if incremental, extension of the pseudo-label line in VALOR++ and LSLD+, and the authors report their ablation fully — the table is right there, so the contradiction is visible rather than hidden. I give credit for that.\n\nThe soft spots are the obvious ones. The named contribution is not supported by the evidence the paper itself provides. The best configuration in Table 2 is 'w/o MTG', which is not the model proposed. There are no error bars, no code, and only partial training details (K=4, but no pseudo-label threshold, MLP hidden dimension, seeds, or GAT heads). For a 5-page conference paper these omissions are common, but they matter more here because the positive result already looks fragile without them.\n\nNovelty is incremental — each block is known — but that alone wouldn't bother me. The issue is that the story the authors tell about their own numbers is off. They acknowledge in Section 3.3 that 'some indicators are slightly lower' with the full model, but 4.8 points is not slight, and it directly undercuts the abstract's claim that combining text enhancement and MTG is what achieves SOTA.\n\nWho gets value from this paper: anyone working on weakly supervised AVVP who wants a fast, effective text-enhancement trick. The MTG part should be repositioned as optional or dropped.\n\nIf I were the editor, I would send it to review rather than desk-reject, because the empirical finding about text enhancement is real and the ablation is informative. But I'd expect a major revision that either demotes MTG or finds a setting where it actually helps.","headline":"Table 2 contradicts the paper's central claim: removing the multi-hop temporal graph makes the model better, so the reported SOTA gains come from text enhancement alone.","tokens_in":8957,"tokens_out":4246,"would_cite":false,"duration_ms":50172,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TeMTG combines text embeddings generated from CLAP and CLIP pseudo labels with a multi-hop temporal graph to reach state-of-the-art segment-level results on the LLP audio-visual video parsing benchmark.","keywords":["audio-visual video parsing","weakly supervised learning","text-enhanced feature fusion","multi-hop temporal graph","graph attention network","CLAP","CLIP","LLP dataset"],"falsifier":"Re-run the ablated configurations on the LLP test split with several seeds: the paper's own Table 2 shows the text-enhancement-only model (w/o MTG) already exceeds the full TeMTG on segment-level audio F1 (76.5 vs 74.4) and Event@AV (75.7 vs 74.1). If that ordering reproduces, the claim that the temporal graph improves joint segment parsing on top of text enhancement fails for those indicators, leaving only the comparison against other published methods as evidence for the overall state-of-the-art claim; if the full model wins on replication, the component claim stands.","tokens_in":7811,"feed_emoji":"🎬","tokens_out":15660,"duration_ms":145573,"temperature":0.7,"pith_summary":"This paper tries to establish that weakly supervised audio-visual video parsing improves when explicit semantic guidance from text is fused into the features and when temporal relations between segments are modeled explicitly. The proposed framework, TeMTG, turns CLAP and CLIP segment-level pseudo labels into template natural-language sentences, re-embeds those sentences with the same pretrained text encoders, and merges the resulting text embeddings into the audio and visual features. It then builds a bidirectional $K$-hop temporal graph over the segments of each modality and propagates information with multi-head graph attention, so each segment sees both nearby and more distant neighbors. On the LLP benchmark the full system reports state-of-the-art segment-level results — 74.4 audio F1, 72.9 visual F1, and 74.1 Event@AV — exceeding the best prior published numbers by 4.2, 1.6, and 5.3 points. A sympathetic reading is that text semantics and explicit local temporal structure are complementary remedies for the ambiguous event boundaries left by video-level supervision.","feed_headline":"74.4/72.9: text-augmented graph model tops LLP video parsing","feed_subtitle":"Weak labels blur boundaries; template text plus a temporal graph beats prior bests by up to 5.3 points.","key_machinery":"Two mechanisms carry the argument. The first is template-caption re-embedding: segment-level pseudo labels $p^a_t, p^v_t$ from CLAP and CLIP are converted into fixed natural-language sentences, concatenated with conjunctions when several events co-occur, re-embedded through the text branches of the same CLAP/CLIP encoders, and fused with the audio/visual features by concatenation followed by a two-layer MLP with LayerNorm and a final linear projection back to the original dimension. The second is the Multi-Hop Temporal Graph: for each modality, segments are nodes and bidirectional edges connect every node to all nodes within $K$ steps ($K=4$ in the experiments), with a self-loop; the adjacency matrix is $A_{ij}=1$ if $0 \\le j-i \\le K$, and message passing uses multi-head residual graph attention with LeakyReLU attention weights, followed by global mean pooling and an MLP that re-injects the whole-video representation into every node. The text fusion does the work of making semantically similar segments distinguishable; the graph does the work of propagating context across time so that both short and long events stay temporally coherent. The graph layer is inserted after each of the two feature-aggregation branches of the CoLeaF baseline.","core_discovery":"The central claim is that combining text-enhanced feature fusion with multi-hop temporal graph modeling yields the best published results on the LLP dataset for weakly supervised audio-visual video parsing. On the paper's own terms, the text enhancement component supplies what weak video-level labels cannot: per-segment semantic guidance. Segment-level pseudo labels produced by CLAP and CLIP are rewritten as captions such as 'This is the sound of x audio event' or 'There is no sound in the segment', re-embedded by the same pretrained text encoders, and fused with audio and visual features through a two-layer MLP. The multi-hop temporal graph then links each segment to its neighbors up to $K$ steps away in both directions and applies residual multi-head graph attention, letting the model reason about short-term continuity and long-range event dependencies. The reported segment-level gains over the previous best system are +4.2 audio F1, +1.6 visual F1, and +5.3 Event@AV; the paper also reports that joint audio-visual event parsing (62.0 segment-level AV F1) remains below one competitor's 64.4, which it attributes to residual noise in the CLAP/CLIP pseudo labels at event boundaries.","pith_inferences":["The paper's own ablation suggests the temporal graph may cost accuracy once text enhancement is present (the w/o MTG configuration beats the full model on most segment-level metrics); my inference is that the state-of-the-art claim is carried by the text component plus the comparison to published systems, and a natural redesign would gate or weight the graph's propagation so its smoothing does not","A cheap experiment the paper does not report would sweep the hop count $K$ from 1 upward: if performance saturates at small $K$, the long-range modeling justification is not doing the work, while a peak near $K=4$ would confirm that temporal range genuinely matters and show how far the smoothing extends.","The template-caption trick should transfer to other weakly supervised temporal tasks with asynchronous modalities, such as sound event detection with video or audio-visual localization; replacing the fixed templates with free-form captions from a language model, or adding a confidence gate before re-embedding, would test whether pseudo-label noise rather than fusion architecture is the bottleneck.","The graphs are built separately per modality, so a further extension the paper leaves implicit is an inter-modal temporal edge set that models asynchrony directly (events heard but not seen, or vice versa); the current design handles that only implicitly through the later attention-based aggregation."],"forward_implications":["On the LLP benchmark, the combined model reports 74.4 segment-level audio F1, 72.9 visual F1, and 74.1 segment-level Event@AV, the best published numbers on those indicators and gains of 4.2, 1.6, and 5.3 points over the previous best system.","The ablation shows text enhancement alone is the largest single contributor, lifting audio parsing by more than ten points over the same-feature baseline, which implies that re-embedded caption semantics can substitute, in part, for the segment-level supervision the task lacks.","The $K$-hop temporal graph, when used without text enhancement, still improves single-modality parsing over the baseline (audio 64.2 to 64.8, visual 67.4 to 68.9 at segment level), indicating that explicit local temporal dependencies are learnable from weak video-level labels.","Because the full model's joint Event@AV is the best reported, the framework implies that text semantics plus temporal propagation yields comparatively accurate audio-visual event boundaries, which matters for applications such as surveillance and content-based indexing.","The drop from text-enhancement-only to the full model (for example 76.5 to 74.4 audio F1) shows the two modules interact; the paper attributes this to a smoothing effect of temporal aggregation that reduces feature distinctiveness and names it as a limitation to explore."],"supporting_citations":[{"why":"Defines the AVVP task, the LLP dataset, the MMIL pooling used for video-level aggregation, and the evaluation protocol.","marker":"[1]"},{"why":"Sets the F1-score and mIoU evaluation convention the paper follows and is the strongest contrastive-learning comparison.","marker":"[6]"},{"why":"CoLeaF is the adopted baseline framework; the two-branch aggregation is where the temporal graphs are inserted.","marker":"[7]"},{"why":"Supplies the idea of using CLAP/CLIP segment-level pseudo labels, which TeMTG rewrites as captions and re-embeds.","marker":"[8]"},{"why":"CLAP provides the audio features and the audio text encoder used to re-embed audio captions.","marker":"[9]"},{"why":"CLIP provides the visual features and the visual text encoder used to re-embed visual captions.","marker":"[10]"},{"why":"NREP is the previous best published system; the reported gains of +4.2, +1.6, and +5.3 points are measured against it.","marker":"[16]"},{"why":"Supplies the multi-head graph attention mechanism used for message passing on the $K$-hop temporal graph.","marker":"[19]"}],"fun_headline_variants":["TeMTG text-plus-graph model tops LLP audio-visual parsing","Up to 5.3 gain: text-enhanced temporal graph beats prior bests","Weak labels sharpened by text and multi-hop graph yield SOTA","Template text and temporal graph modeling lift LLP video parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that re-embedding CLAP/CLIP pseudo labels transformed into fixed template sentences yields text vectors that genuinely describe what happens in each segment; if the pseudo labels are noisy, the text guidance can distort the very features it is meant to disambiguate, and the paper's own reported weakness on audio-visual event boundaries suggests such noise is already present.","fun_headline_variants_meta":{"raw":{"variants":["TeMTG text-plus-graph model tops LLP audio-visual parsing","Up to 5.3 gain: text-enhanced temporal graph beats prior bests","Weak labels sharpened by text and multi-hop graph yield SOTA","Template text and temporal graph modeling lift LLP video parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3110,"prompt_tokens":993,"completion_tokens":2117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":609,"tokens_out":2117,"duration_ms":18440,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:01:15.055337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ablated configurations on the LLP test split with several seeds: the paper's own Table 2 shows the text-enhancement-only model (w/o MTG) already exceeds the full TeMTG on segment-level audio F1 (76.5 vs 74.4) and Event@AV (75.7 vs 74.1). If that ordering reproduces, the claim that the temporal graph improves joint segment parsing on top of text enhancement fails for those indicators, leaving only the comparison against other published methods as evidence for the overall state-of-the-art claim; if the full model wins on replication, the component claim stands.","supporting_citations":[{"cited_title":"Unified multisensory perception: Weakly-supervised audio-visual video parsing","cited_arxiv_id":null,"evidence_quote":"Defines the AVVP task, the LLP dataset, the MMIL pooling used for video-level aggregation, and the evaluation protocol."},{"cited_title":"Collecting cross-modal presence-absence evidence for weakly-supervised audio-visual event percep- tion","cited_arxiv_id":null,"evidence_quote":"Sets the F1-score and mIoU evaluation convention the paper follows and is the strongest contrastive-learning comparison."},{"cited_title":"ColeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio- Visual Video Parsing","cited_arxiv_id":null,"evidence_quote":"CoLeaF is the adopted baseline framework; the two-branch aggregation is where the temporal graphs are inserted."},{"cited_title":"Modality-independent teachers meet weakly-supervised audio-visual event parser","cited_arxiv_id":null,"evidence_quote":"Supplies the idea of using CLAP/CLIP segment-level pseudo labels, which TeMTG rewrites as captions and re-embeds."},{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation","cited_arxiv_id":null,"evidence_quote":"CLAP provides the audio features and the audio text encoder used to re-embed audio captions."},{"cited_title":"Resisting Noise in Pseudo Labels: Audible Video Event Parsing With Evidential Learning","cited_arxiv_id":null,"evidence_quote":"NREP is the previous best published system; the reported gains of +4.2, +1.6, and +5.3 points are measured against it."},{"cited_title":"Graph Attention Networks","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-head graph attention mechanism used for message passing on the $K$-hop temporal graph."}],"review_version":1}