Pith. sign in

REVIEW 6 major objections 4 minor 4 references

CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization

T0 review · 6 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that timestamps where audio and visual classifiers agree can serve as anchors, and propagating those anchors along the timeline yields 30.0% mAP on UnAV-100 and 21.3% on ActivityNet1.3, the best reported results.

desk verdict A legit new weakly-supervised DAVEL task with a plausible anchor-propagation method, but the ActivityNet experiment is currently under-specified and the UnAV-100 numbers lack error bars; worth refereeing after those are fixed. read the letter →

arxiv 2508.04566 v1 pith:VI7RF5VD submitted 2025-08-06 cs.CV cs.AIcs.MM

Jinxing Zhou , Ziheng Zhou , Yanghao Zhou , Yuxin Mao , Zhangling Duan , Dan Guo This is my paper · ORCID
classification cs.CVcs.AIcs.MM
keywords weakly-supervisedlearningaudio-visualeventlocalizationdensecross-modalsalientanchorstemporalsemanticpropagationuntrimmedvideounderstandingmultipleinstanceJensen-Shannondivergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces W-DAVEL, a setting in which a model must localize every audio-visual event in a long, untrimmed video - predicting both which event categories occur and when - using only video-level labels, with no temporal boundaries given. The central claim is that such localization is achievable by exploiting cross-modal salient anchors: a small set of timestamps where independently trained audio and visual classifiers produce the most consistent event predictions. The CLASP method computes a mutual agreement score from the Jensen-Shannon divergence between the two modality predictions, selects the top-scoring timestamps both globally and inside local windows, fuses the chosen audio-visual features, and propagates those anchor semantics to every timestamp through cross-attention. On the two benchmarks the paper establishes, CLASP reports 30.0% average mAP on UnAV-100 and 21.3% on ActivityNet1.3, surpassing the previous best method by 3.1 and 1.7 points. If the claim holds, reliable timestamps can substitute for expensive frame-level annotation in dense event localization.

What carries the argument

The load-bearing object is the mutual agreement score $s = 1 - d_{\mathrm{JSD}}$, where $d_{\mathrm{JSD}}$ is the Jensen-Shannon divergence between per-timestamp sigmoid predictions $P_a$ and $P_v$ from two modality-specific classifiers (Eqs. 1-3). One scalar per timestamp ranks reliability: global anchor identification takes the top-$K$ over the whole video; local anchor identification takes the top-$k$ inside each of $M$ temporal windows. The chosen audio and visual features are summed within each modality and fused into $Z_{av} \in \mathbb{R}^{K \times d}$ (Eq. 4). Anchor-based Temporal Propagation then applies multi-head cross-attention, $\hat{F}_A = F_A + \mathrm{MHA}(F_A, Z_{av}, Z_{av

What would settle it

Shuffle the pairing in a trained CLASP model: compute the agreement score between each audio timestamp's prediction and the visual prediction of a different timestamp (or a different video), leaving the rest of the pipeline untouched. If average mAP on UnAV-100 stays near 30.0%, genuine cross-modal agreement is not the active ingredient; if it collapses, agreement is load-bearing. A cheaper complementary check: on a labeled subset, count how many of the chosen top-K anchors fall inside ground-truth event segments - the method's premise requires nearly all of them to.

Watch

Extended reading notes

Core claim

The paper's claim, stated on its own terms: under video-level supervision alone, some timestamps can already be predicted with confidence - the cross-modal salient anchors - and these are exactly the timestamps where independently trained audio and visual event classifiers agree most. CLASP turns that intuition into a pipeline: two modality-specific classifiers emit per-timestamp probabilities; an agreement score, 1 minus the Jensen-Shannon divergence between the two outputs, ranks every timestamp; the top-K globally and the top-k within each of M local windows identify anchor features, which are summed per modality and fused into a compact multimodal anchor representation; and a cross-atten

Load-bearing premise

The load-bearing premise is that a timestamp where the audio and visual classifiers agree is genuinely an event moment; if both classifiers are confidently wrong in the same way, or background segments routinely score high agreement, the anchors and the propagation amplify that shared error instead of correcting it.

Editorial extensions

If this is right

  • Video-level labels alone reach 30.0% average mAP on UnAV-100 and 21.3% on ActivityNet1.3, establishing the first W-DAVEL benchmarks with margins of 3.1 and 1.7 points over the previous best method.
  • Global and local anchor identification are complementary: ablations show that removing either one lowers performance (29.5% or 29.8% vs. 30.0% average mAP), so both scales of selection contribute.
  • Anchors fused from both modalities beat single-modality anchors (30.0% vs. 29.6% or 29.4%), confirming that cross-modal agreement, not one modality's confidence, drives the gain.
  • The anchor counts are stable hyperparameters: K=10 global anchors, k=4 per window, and M=14 windows give the best results, with graceful degradation when varied.
  • The gap between this weak-supervision result (30.0%) and the roughly 50% average mAP of fully supervised DAVEL methods quantifies the headroom that remains in the task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper leaves implicit: replace the agreement score with a properly normalized probability distance (JSD on softmax-normalized predictions, or cosine similarity between class-conditional embeddings) to check whether the informal use of JSD on sigmoid outputs costs accuracy; the ablations suggest the score function itself is a tunable axis.
  • The anchor-propagation idea should transfer to other weakly supervised dense tasks - temporal action proposals or weakly supervised audio-visual segmentation - wherever a few trustworthy timestamps can seed semantic spreading. This is an editorial inference, not a paper claim.
  • Because anchors are chosen per video at inference time by the model's own agreement, CLASP could be bootstrapped on unlabeled videos, using high-agreement timestamps as pseudo-labels for a second training round; the paper does not explore this.
  • The method should struggle exactly where agreement is spuriously high - long homogeneous scenes such as an empty road with steady traffic noise, where background timestamps may outrank short genuine events; stress-testing on such videos would separate true anchor semantics from mere classifier consensus.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper introduces a new weakly-supervised variant of Dense Audio-Visual Event Localization (W-DAVEL), in which only video-level event labels are available. The proposed method, CLASP, consists of three modules: Mutual Event Agreement Evaluation (MEAE) computes a cross-modal agreement score from independently predicted audio and visual event probabilities; Cross-modal Salient Anchor Identification (CSAI) selects global and local temporal anchors using that score; and Anchor-based Temporal Propagation (ATP) enhances per-timestamp features through cross-attention with the anchor features. The authors evaluate CLASP on UnAV-100 and ActivityNet1.3, comparing against prior AVEP and DAVEL methods under a weakly-supervised protocol, and report state-of-the-art results, including a 3.1% Avg mAP gain over CCNet on UnAV-100 and a 1.7% Avg mAP gain on ActivityNet1.3.

Significance. If the experimental claims are reliable, the paper makes a useful contribution by defining a practical weakly-supervised formulation of DAVEL and proposing a coherent bootstrapping mechanism that does not rely on external foundation models or manual pseudo-labels. The idea of using cross-modal agreement to identify reliable timestamps and then propagating semantic information from those anchors is intuitive and well aligned with the weak-supervision setting. However, the paper's central claim is an empirical SOTA claim, and several load-bearing experimental details are missing or inconsistent. The reported accuracy gains are meaningful only if the benchmark construction, baseline adaptation, metric computation, and hyperparameter selection protocol are fully documented.

major comments (6)
  1. [Experiments, 'Datasets' paragraph] The construction of the ActivityNet1.3 W-DAVEL benchmark is not specified. The paper does not state (i) how the 203 activity classes are mapped to the C event classes used by P_a/P_v/p_av, (ii) how segment-level audio-visual event ground truth is derived from action temporal annotations, (iii) what video-level weak labels are used, or (iv) how the fully-supervised baselines DAVE and CCNet are adapted to weak supervision. Without this, Table 2 and the claimed 1.7% improvement over CCNet are not reproducible.
  2. [Evaluation Metric and Tables 1-2] The reported 'Avg.' is not the arithmetic mean of the five tIoU mAP values as defined in the text. For example, in Table 1 the CLASP row (29.6, 26.3, 23.0, 19.7, 15.8) has mean 22.9, not the reported 30.0; the same discrepancy appears in every row of Tables 1-2 and all ablation tables. Since the paper's headline '3.1% mAP' gain over CCNet uses this number, the metric must be defined precisely or all Avg values corrected.
  3. [Implementation Details and Tables 3-5] Hyperparameters K, k, and M are selected based on experiments on UnAV-100, but the paper does not state whether these ablations are performed on the validation or test split. If the test set is used for model selection, the reported test performance is optimistically biased. Please clarify the split and, if necessary, re-run with validation-based selection.
  4. [Table 1, 'Evaluation on the UnAV-100 dataset'] The paper says 'All methods are trained and evaluated under weakly supervised settings,' but DAVE and CCNet were designed for fully supervised DAVEL and the AVEP baselines for trimmed videos. The text never specifies how each baseline is adapted to W-DAVEL (loss change, MIL pooling, foreground suppression, etc.). Without a uniform protocol, the comparisons are not meaningful.
  5. [Methodology, MEAE, Eq. (2)] Jensen-Shannon divergence is defined for probability distributions that sum to one, but P_a and P_v are sigmoid multi-label scores whose rows do not normalize to 1. The agreement score s = 1 - d_JSD therefore lacks the theoretical interpretation claimed and may be driven by calibration differences. Please either normalize each row before computing JSD or justify the use of unnormalized scores.
  6. [Experiments, 'Implementation Details' and 'Evaluation Metric'] The inference procedure after thresholding p_av at 0.5 is not described; the paper does not specify how per-segment event probabilities are grouped into event instances for tIoU-based mAP (e.g., contiguous segment merging, non-maximum suppression, or actionness smoothing). This step is necessary to reproduce Tables 1-2.
minor comments (4)
  1. [Throughout] The abstract and several sections contain the typo 'DA VEL' instead of 'DAVEL'.
  2. [Tables 3-9] The 'Avg.' column values are unexplained relative to the mean of the tIoU columns; beyond the major issue above, the captions should define exactly how 'Avg.' is computed.
  3. [Figure 2] The boxes labeled 'Linear' and 'Transformer' in Figure 2 are not all defined in the caption; a notation list or dimension annotations would improve clarity.
  4. [References] There are two 'Cheng et al.' entries with different years; the in-text citation for JoMoLD should be checked to ensure it refers to the correct year and publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CLASP's claims are empirical benchmark comparisons; the anchor-propagation mechanism is a weakly-supervised bootstrapping heuristic, not a reduction to its inputs.

full rationale

The paper's derivation chain is: video-level labels -> modality classifiers Pa/Pv -> agreement score s -> top-K/top-k anchors -> fused anchor features Zav -> cross-attention enhancement -> final pav -> MIL-pooled video-level cross-entropy loss. Every quantity is a function of the same video-level labels; no fitted parameter is renamed as a prediction, no label is defined by the output, and no load-bearing claim is justified only by a self-citation. The agreement score (1 - JSD on sigmoid outputs) is a modeling heuristic, not a circular derivation, and the anchors are selected from the model's own predictions as a bootstrapping strategy common in weak supervision. The comparisons to CCNet and other methods are external empirical results, independent of the proposed equations. The ActivityNet1.3 evaluation under-specifies how W-DAVEL labels are constructed, which is a reproducibility/transparency concern, but it does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The core contributions rest on hand-tuned hyperparameters chosen on the test set, on a technically informal use of JSD, and on the domain assumption that pretrained features and MIL pooling are adequate. The 'cross-modal salient anchor' is an internal construct with no independent evidence.

free parameters (4)
  • K (global anchor count) = 10
    Number of global anchors, tuned on UnAV-100 test set via ablation (Table 3).
  • k (local anchor count per window) = 4
    Number of local anchors per window, tuned on UnAV-100 test set via ablation (Table 4).
  • M (number of temporal windows) = 14 (UnAV-100), 16 (ActivityNet1.3)
    Set separately per dataset, tuned via ablation (Table 5 and Implementation Details).
  • Inference threshold = 0.5
    Fixed threshold to convert predicted probabilities to event detections during inference.
assumptions (4)
  • ad hoc to paper Jensen-Shannon divergence between sigmoid multi-label class probabilities is a valid cross-modal agreement measure.
    Eqs. 2-3 apply JSD to Pa and Pv from sigmoid classifiers; these are not normalized distributions, so the standard definition is not satisfied.
  • domain assumption Pretrained VGGish and I3D features encode sufficient audio and visual event semantics for the weakly supervised task.
    Unimodal Feature Preparation section relies on these pretrained representations without re-verifying their adequacy for DAVEL.
  • domain assumption Video-level labels correspond to events that co-occur in both audio and visual modalities.
    Problem statement defines the target event as present only when both modalities agree; the supervision y comes from video-level event labels assumed to satisfy this.
  • domain assumption MIL pooling of timestamp-level predictions to video-level labels is sufficient for learning event semantics.
    Classification and Model Training section uses MIL pooling and video-level cross-entropy; this assumes the aggregation recovers timestamp-level supervision.
invented entities (1)
  • cross-modal salient anchors
    purpose: Temporal timestamps selected by top agreement scores, used as seeds to propagate event semantics via cross-attention.
    Defined purely from the model's own agreement measure; no external or falsifiable handle beyond downstream benchmark performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization." pith.science (2026). https://pith.science/paper/VI7RF5VD

@misc{pith2026250804566,
  author       = {Pith},
  title        = {Pith review of: CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI7RF5VD}},
  note         = {Machine review of arXiv:2508.04566}
}
read the original abstract

The Dense Audio-Visual Event Localization (DAVEL) task aims to temporally localize events in untrimmed videos that occur simultaneously in both the audio and visual modalities. This paper explores DAVEL under a new and more challenging weakly-supervised setting (W-DAVEL task), where only video-level event labels are provided and the temporal boundaries of each event are unknown. We address W-DAVEL by exploiting \textit{cross-modal salient anchors}, which are defined as reliable timestamps that are well predicted under weak supervision and exhibit highly consistent event semantics across audio and visual modalities. Specifically, we propose a \textit{Mutual Event Agreement Evaluation} module, which generates an agreement score by measuring the discrepancy between the predicted audio and visual event classes. Then, the agreement score is utilized in a \textit{Cross-modal Salient Anchor Identification} module, which identifies the audio and visual anchor features through global-video and local temporal window identification mechanisms. The anchor features after multimodal integration are fed into an \textit{Anchor-based Temporal Propagation} module to enhance event semantic encoding in the original temporal audio and visual features, facilitating better temporal localization under weak supervision. We establish benchmarks for W-DAVEL on both the UnAV-100 and ActivityNet1.3 datasets. Extensive experiments demonstrate that our method achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2508.04566 by the authors.

Figure 1
Figure 1. Illustration of the Weakly-supervised Dense Audio-Visual Event Localization (W-DAVEL) task. The W-DAVEL task aims to temporally localize events that co￾occur in both audio and visual modalities, using only weak supervision from video-level event labels. more. However, the majority of existing works focus on short￾term audio-visual scene understanding. Recently, Geng et al. (Geng et al. 2023) proposed the Dense Audio… view at source ↗
Figure 2
Figure 2. It consists of five main modules. (1) In the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 2
Figure 2. Illustration of our cross-modal salient anchor-based semantic propagation (CLASP) framework for W-DAVEL. Global Anchor Identification (GAI). The GAI aims to iden￾tify the salient anchors by performing a global comparison over the entire video timeline. Specifically, timestamps with the top-K values in s are selected. Let I g denote the index of the selected timestamps, where each index I g i belongs to {1, ..., T} f… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Visualization examples of the weakly-supervised dense audio-visual event localization. We compare our method with CCNet (Zhou et al. 2025c). ‘GT’ is the ground truth. The red arrows mark the identified global ten salient anchor timestamps. Setups 0.5 0.6 0.7 0.8 0.9 Av…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages

  1. [2015]

    In CVPR, 961–970

    ActivityNet: A large-scale video benchmark for human activity understanding. In CVPR, 961–970. Hershey, S.; Chaudhuri, S.; Ellis, D. P.; Gemmeke, J. F.; Jansen, A.; Moore, R. C.; Plakal, M.; Platt, D.; Saurous, R. A.; Seybold, B.; et al. 2017. CNN architectures for large- scale audio classification. In ICASSP, 131–135. Hong, F.-T.; Feng, J.-C.; Xu, D.; Sh...

    arXiv 2017

  2. [2020]

    Audio-Visual Instance Segmentation

    Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In ACM MM, 3884–3892. Fan, Y .; Wu, Y .; Du, B.; and Lin, Y . 2023. Revisit weakly- supervised audio-visual video parsing from the language perspective. NeurIPS, 36: 40610–40622. Fuglede, B.; and Topsoe, F. 2004. Jensen-Shannon divergence and Hilbert s...

    work page Pith review arXiv 2023

  3. [2022]

    In ECCV, 431–448

    Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing. In ECCV, 431–448. Cheng, Y .; Wang, R.; Pan, Z.; Feng, R.; and Zhang, Y

    work page

  4. [2025]

    In AAAI, 5075–5083

    Patch-level sounding object tracking for audio-visual question answering. In AAAI, 5075–5083. Liu, X.; Xia, N.; Zhou, J.; Li, Z.; and Guo, D. 2025. Towards energy-efficient audio-visual classification via multimodal interactive spiking neural network. ACM Transactions on Multimedia Computing, Communications and Applications, 21(5): 1–24. Mahmud, T.; and M...

    arXiv 2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.