{"id":"f65e4c9e-1ccf-4109-99d5-6283cb127a4e","arxiv_id":"2412.11715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DAAN combines a differential-attention module (QDMA) and a sample-level gradient modulation block (CSGM) to improve audio-visual zero-shot classification, reporting the best UCF101 GZSL harmonic mean so far.","lead":"This paper proposes DAAN, a network for audio-visual zero-shot learning that tries to balance how much the model relies on sound versus video. On two of three standard benchmarks it reports top accuracy, though on one benchmark an older model still scores higher.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 6 forces convergence rate to zero, so CSGM degenerates to a constant scaling and cannot perform the claimed sample-level gradient modulation.","rationale":"I read the paper in good faith. The central claim is that DAAN achieves SOTA audio-visual (G)ZSL through two modules, QDMA and CSGM, with CSGM balancing sample-level content discrepancies via a contrastive convergence rate and an optimization rate. For that claim to hold, the convergence rate V_c in Eq. 6 must be non-zero and sample-dependent. It is not: the δ function zeroes any term with e=n, and both V_c products contain two such terms, making V_c identically zero. The modulation then collapses to a fixed scaling γ, which is exactly what Eq. 8 would do without any discrepancy awareness. This is a load-bearing mathematical flaw in the main contribution, not a matter of external consensus. The reader's weakest_assumption focused on Eq. 7's missing per-modality gradients; I find Eq. 6 more fundamental, because even if G_a and G_v were defined separately, they are multiplied by V_c ≡ 0. An empirical check—logging η_i in the actual code—would decide whether this is a typo or a real implementation issue. If it is a typo, the manuscript needs a corrected Eq. 6 and a re-run ablation; if not, the reported CSGM gains are unexplained. I therefore keep the reader's conditional verdict unchanged: the paper is not accept-ready, but a corrected derivation and released code could make it testable.","tokens_in":10448,"tokens_out":11645,"duration_ms":98006,"concrete_test":"Evaluate Eq. 6 analytically with the paper's δ definition: V_i_{a,c} and V_i_{v,c} are identically zero because two factors per product have e=n. To confirm against the implementation, obtain the training code and log η_i for a random batch; if η_i is constant γ for all samples, CSGM is inert, and the UCF ablation gain should be reproducible with a fixed-γ baseline. If the code instead produces non-zero V_c, the printed formula must be corrected, and the UCF ablation re-run with the corrected formula to verify the 18.33% HM improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 6 defines V_i_{a,c} = δ_i_{a+,w+,a−} · δ_i_{w+,a−,a−} · δ_i_{a+,w−,w−}, and analogously for v, with δ_i_{m,e,n} = max(d(θ_m,θ_e) − d(θ_m,θ_n), 0). In the factors δ_i_{w+,a−,a−} and δ_i_{a+,w−,w−} (and their v counterparts) the arguments e and n coincide, so each of those factors is max(0,0)=0. Hence V_i_{a,c} ≡ V_i_{v,c} ≡ 0 for every sample. Substituting into Eq. 5 yields η_i_{a,p} = η_i_{v,p} = max(0,γ) = γ, reducing Eq. 8 to Θ − γG + noise. The CSGM block therefore contains no sample-level or modality-dependent information; the optimization-rate factor V_o in Eq. 7 is multiplied by zero and cannot restore it. The 18.33% HM gain attributed to CSGM in Table II cannot be explained by the stated mechanism. This is an internal inconsistency, not a disagreement with external consensus, and it directly undermines the paper's central contribution.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DAAN, an audio-visual zero-shot learning architecture with two novel components: QDMA, a sparse-attention block that reduces redundant information in the higher-quality modality, and CSGM, a sample-level gradient modulation block intended to balance content discrepancies by scaling parameter updates according to a contribution rate derived from optimization and convergence rates. The method is evaluated on VGGSound, UCF101, and ActivityNet for both ZSL and GZSL, and an ablation study on UCF attributes gains to the two blocks. The manuscript claims state-of-the-art performance, though the reported numbers are mixed relative to MDFT and other baselines.","tokens_in":10687,"tokens_out":5187,"duration_ms":48435,"significance":"If the proposed mechanism worked as described, the paper would address a real and recognized problem: modality imbalance in audio-visual zero-shot learning, and it would do so at the sample level, which is a genuinely finer granularity than prior global gradient-modulation schemes. The motivation is clear, the benchmark suite is standard, and the ablation design is reasonable in intent. However, the central CSGM derivation as printed degenerates: Eq. (6) forces both convergence rates to be identically zero, reducing the contribution rate to a constant and eliminating the claimed sample- and modality-specific modulation. This is an internal inconsistency, not a matter of disagreement with external consensus, and it directly undermines the paper's main contribution. The paper also has no released code, and several equations cannot be implemented as written. The significance of the empirical results is therefore compromised because the main mechanism claimed to produce them is not actually defined in a functioning form.","major_comments":[{"comment":"Eq. (6) defines V_i_{a,c} = δ_i_{a+,w+,a−} · δ_i_{w+,a−,a−} · δ_i_{a+,w−,w−}. In the second and third factors, the arguments e and n coincide: δ_i_{w+,a−,a−} = max(d(θ_w+, θ_a−) − d(θ_w+, θ_a−), 0) = 0, and δ_i_{a+,w−,w−} = max(d(θ_a+, θ_w−) − d(θ_a+, θ_w−), 0) = 0. Therefore V_i_{a,c} ≡ 0 for every sample, and analogously V_i_{v,c} ≡ 0. Substituting into Eq. (5) gives η_i_{a,p} = η_i_{v,p} = max(0, γ) = γ, and Eq. (8) reduces to Θ − γG + ϵ, a constant scaling with no sample-level or modality-dependent content. This contradicts Table II, where the row 'Base* + QDMA + CSGM (V_c)' reports HM 31.31 versus 26.46 for 'Base* + QDMA'; if V_c were identically zero, the two rows would be indistinguishable up to noise. As printed, the CSGM mechanism cannot explain the ablation gain, and the central contribution of the paper is not supported.","section":"§II-B, Eq. (6)"},{"comment":"The two displayed formulas for V_i_{a,o,p} and V_i_{v,o,p} are identical: both equal ||G_p||² / ||Θ_p||². No separate per-modality gradients G_a,p, G_v,p or parameters Θ_a,p, Θ_v,p are defined before this equation, although Eq. (8) then uses Θ_i_{a,p}, G_i_{a,p} and Θ_i_{v,p}, G_i_{v,p}. Without a per-modality parameterization of G and Θ, the claimed audio-versus-visual gradient modulation cannot be derived from the stated mathematics, and the entire content-discrepancy balancing mechanism depends on an unstated quantity that never appears in the paper.","section":"§II-B, Eq. (7)"},{"comment":"Eq. (1) as printed is dimensionally inconsistent: S(Q_i_1 K_i_1^T) is a softmax attention matrix, while S(Q_i_2 K_i_2^T)V_i is an attended feature vector. Subtracting a matrix from a vector is not a valid operation under the stated shapes, and the accompanying text says the two softmax scores are subtracted but attaches V_i to only the second term. The intended sparse-attention operation needs to be written with consistent tensor shapes, and all later equations that consume o1_i depend on this correction.","section":"§II-A, Eq. (1)"},{"comment":"The text after Eq. (5) states that V_i_c and V_i_o,p are 'normalized under the regulation of weight parameter µ', but no normalization formula or definition of µ appears in the method section. The hyperparameter section lists distinct µ values per dataset, yet the equations for V_i_c and V_i_o contain no µ. This missing definition makes the contribution-rate formula non-reproducible and prevents the reader from verifying whether the claimed modulation is actually controlled by the stated hyperparameter.","section":"§II-B, Eq. (5) and §III-A"}],"minor_comments":[{"comment":"The abstract and Section III-B claim state-of-the-art performance and that DAAN 'significantly exceeds almost all baselines', but Table I shows DAAN is below MDFT on VGGSound HM (8.32 vs. 8.72) and ActivityNet ZSL (11.15 vs. 12.55), and its UCF seen accuracy (52.38) is below TCaF (58.60). The performance claims should be qualified to specific metrics and datasets.","section":"§III-B, Table I and abstract"},{"comment":"The text says the (G)ZSL scores peak at n=5 with maxima of 27.59 and 26.54, and that γ peaks at 0.3, but Table II reports DAAN with HM 32.42 and ZSL 31.09 using n=2 and γ=0.45 for UCF. It is unclear what Fig. 3 is plotting or under which configuration, and the discrepancy should be resolved.","section":"§III-C, Fig. 3"},{"comment":"Eq. (11) uses the term d(ϕ_i_a, ϕ_i_w), but ϕ_i is only defined in Eq. (4) as an element of {ϕ_a,i, ϕ_v,i}; no ϕ_w,i is ever defined, so the regularization loss is incomplete as written.","section":"§II-C, Eq. (11)"},{"comment":"There are numerous language and typographical errors, e.g., 'Experiments demonstrates', 'an novel approach', 'the modal’s parameters', '2-norm demonstrates', and inconsistent use of 'modal' for 'model'. These should be corrected in a revision.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The central contribution of the paper is not merely hard to verify; as written, the key equation (Eq. 6) degenerates to a constant, and the ablation table contradicts that degeneration. This is an internal inconsistency rather than a matter of interpretation. Even if the authors corrected the indices in Eq. (6), the missing per-modality gradient definitions in Eq. (7) would still require substantial additional derivation and new experiments to support the claimed mechanism. I recommend rejection, though the authors may resubmit a corrected version if the mechanism is properly formalized and the ablation claims are re-established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: the central CSGM mechanism does not exist as written. In Eq. 6, V^a_c and V^v_c are products of three δ factors; two of the three factors for each modality are max(d(θ,·)−d(θ,·),0)=max(0,0)=0 because the argument pair is identical in those factors. So every convergence rate is zero for every sample. Substituting into Eq. 5 gives η = max(0,γ) = γ, a constant lower bound, and Eq. 8 becomes plain SGD plus Gaussian noise. The 18.33% HM improvement attributed to CSGM in Table II cannot be explained by this mechanism. The stress-test note is correct, and the reader's Eq. 7 point is also confirmed: the two formulas there are identical, with no separate G_a and G_v defined. This is an internal contradiction, not a missing detail.\n\nThere is something worthwhile in the paper. The problem—modality imbalance in audio-visual zero-shot learning—is real, and the high-level design is plausible: QDMA is a reasonable transfer of differential attention, the TCN temporal bypass makes sense, and the ablation table shows that the full model does improve over the base. The efficiency comparison with MDFT is also fair. If the implementation actually works, it is doing so through some mechanism other than the one described.\n\nThe soft spots are also clear: Eq. 1 is dimensionally inconsistent (a softmax matrix subtracted from a vector), the beta hyperparameter is never specified, there are no error bars, and the abstract's SOTA claim is stronger than Table I supports. The paper would need a careful rewrite of Section II-B before anyone can evaluate it.\n\nWho is this for? People working on multimodal gradient modulation should read it as an example of how a plausible idea can fail in the equations. I would not cite it in its current form. A serious editor could send it back to the authors with a clear request: fix the math, define per-modality gradients, release code, and rerun the ablations. If the numbers stand after that, it becomes an empirical paper about a working architecture; if not, the current text should not be trusted.\n\nMy recommendation: do not accept as is; send to major revision if the authors can respond to these specific complaints, and require code release.","headline":"CSGM's convergence rate is identically zero by Eq. 6, so the paper's main mechanism is a constant multiplier; the empirical claims are unsupported.","tokens_in":11214,"tokens_out":3609,"would_cite":false,"duration_ms":33469,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that audio-visual zero-shot learning fails because of two distinct imbalances—quality discrepancy across modalities and content discrepancy across samples—and that its DAAN architecture, built from sparse attention and…","keywords":["audio-visual zero-shot learning","generalized zero-shot learning","modality imbalance","gradient modulation","sparse attention","temporal convolutional network","video classification","harmonic mean"],"falsifier":"Instrument DAAN during UCF101 training and log the per-sample coefficients $\\eta_{a,p}^i$ and $\\eta_{v,p}^i$; if they are identical for every sample because the optimization-rate formula supplies the same ratio, then CSGM is not actually balancing per-modality contributions, and retraining with CSGM removed should reproduce the reported gains if the true cause is elsewhere.","tokens_in":10217,"feed_emoji":"🎬","tokens_out":12775,"duration_ms":107960,"temperature":0.7,"pith_summary":"The paper sets out to fix a failure mode in audio-visual zero-shot learning: when one modality, usually vision, carries more task-relevant information, the model over-relies on it and under-learns the other, which hurts recognition of classes never seen in training. DAAN attacks this with two blocks: QDMA subtracts one softmax attention map from another on the same modality to strip redundant information from the strong modality, and CSGM rescales per-sample gradients using a contribution rate built from an optimization rate and a contrastive convergence rate. On the VGGSound, UCF101, and ActivityNet benchmarks, the authors report the best harmonic mean (the standard generalized-ZSL metric that balances seen and unseen accuracy) among the compared methods on UCF101 and ActivityNet, and a ZSL accuracy lead over the strongest baseline on VGGSound despite a slightly lower harmonic mean. A sympathetic reading is that modality imbalance is best treated at sample level, not only with global fusion weights.","feed_headline":"Per-sample gradient control lifts zero-shot video recognition","feed_subtitle":"The network balances audio and visual learning, hitting top harmonic means on UCF101 and ActivityNet.","key_machinery":"The load-bearing object is the per-sample, per-modality contribution rate $\\eta_{a,p}^i = \\max(V_{a,c}^i \\times V_{a,o,p}^i, \\gamma)$ (and its visual counterpart), which multiplies the gradient in the CSGM update. It combines a convergence rate $V_c$ computed from Euclidean distances between positive and negative audio, visual, and text embeddings with the optimization rate $V_o = \\|G_p\\|^2_2/\\|\\Theta_p\\|^2_2$, the ratio of squared gradient norm to squared parameter norm. The other component is QDMA's sparse attention, defined as $S(Q_1K_1^T) - \\beta\\,S(Q_2K_2^T)$ on the same modality, which is meant to cancel redundant information carried by the dominant modality. The CSGM block, which the ablation identifies as the main source of harmonic-mean improvement, is where the paper's balancing claim lives.","core_discovery":"The central claim is that the modality imbalance that suppresses unseen-class recognition in audio-visual zero-shot learning has two separate causes, and both can be removed inside one architecture. Quality discrepancy means the visual stream often carries more target-related information than the audio stream; DAAN's QDMA block reduces this by computing two softmax attention scores from splits of the same feature map and subtracting one from the other, leaving a sparser attention that keeps key information and discards redundancy. Content discrepancy means individual samples vary in how much audio or visual evidence they contribute; DAAN's CSGM block rescales the gradient of each modality module separately for each training sample, using a contribution rate $\\eta_{i,p}$ formed from the optimization rate $\\|G_p\\|^2_2/\\|\\Theta_p\\|^2_2$ and a convergence rate measured from contrastive distances among positive and negative embeddings. On the three benchmarks, this combination is reported to give the best generalized zero-shot harmonic mean on UCF101 (32.42) and ActivityNet (13.87) among the methods compared, and the ablation credits most of the harmonic-mean gain to CSGM.","pith_inferences":["Editorial inference: if the implementation follows the printed equations literally, the audio and visual optimization rates are identical, so the reported gains would have to come from global gradient scaling or QDMA rather than from modality-specific balancing.","Editorial inference: the convergence-rate term suggests a direct test—replace CSGM's contrastive estimate with ground-truth class labels to set per-sample weights; if gains disappear, the contrastive signal is the active ingredient.","Editorial inference: the same per-sample gradient gating could transfer to other audio-visual tasks, such as retrieval or captioning, where one modality often dominates and global fusion weights do not fix per-example variation."],"forward_implications":["On UCF-GZSL, DAAN reports the highest harmonic mean among the compared methods (32.42), and a ZSL accuracy of 31.09 that beats all baselines except MDFT.","On ActivityNet-GZSL, DAAN's harmonic mean of 13.87 is the best among the compared methods, so the benefit is not confined to short action clips.","Ablations on UCF show each component adds value: QDMA raises ZSL from 19.03 to 28.78, and the full CSGM raises harmonic mean from 26.46 to 32.42.","DAAN reaches comparable or better accuracy than MDFT with roughly 2.5 times lower runtime, because it avoids a spiking neural network.","The method currently applies only to complete audio-visual samples; the authors state that videos with missing modalities are outside its scope."],"supporting_citations":[{"why":"Provides the MDFT spiking-transformer baseline that DAAN compares against and the runtime-efficiency contrast.","marker":"[2]"},{"why":"Defines the cross-modal attention baseline and the audio-visual generalized zero-shot learning protocol DAAN builds on.","marker":"[4]"},{"why":"Supplies the temporal cross-modal attention baseline and dataset evaluation setup used for comparison.","marker":"[5]"},{"why":"Motivates the greedy-training diagnosis and supplies the squared gradient-to-parameter norm ratio used as the optimization rate.","marker":"[11]"},{"why":"Provides the prior on-the-fly gradient modulation strategy that CSGM refines from global accuracy-based to per-sample modulation.","marker":"[12]"},{"why":"Introduces sample-level modality valuation, the notion of content discrepancy that CSGM operationalizes.","marker":"[14]"},{"why":"Supplies the temporal convolutional network used in QDMA to extract temporal embeddings.","marker":"[16]"},{"why":"Inspires the weighted subtraction of softmax attention scores used to build QDMA's sparse attention.","marker":"[17]"}],"fun_headline_variants":["Per-sample gradient control plus sparse attention lifts zero-shot video","Modality imbalance solved with quality and content discrepancy attention","Contrastive gradient scaling and sparse attention boost zero-shot recognition","Audio-visual zero-shot: tackling quality and content gaps simultaneously","Two discrepancy fixes join forces for stronger audio-visual ZSL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The balancing trick depends on treating audio and visual learning signals separately, but the printed formulas for the optimization rate appear to define the same value for both modalities, so the claimed mechanism rests on an unstated implementation detail.","fun_headline_variants_meta":{"raw":{"variants":["Per-sample gradient control plus sparse attention lifts zero-shot video","Modality imbalance solved with quality and content discrepancy attention","Contrastive gradient scaling and sparse attention boost zero-shot recognition","Audio-visual zero-shot: tackling quality and content gaps simultaneously","Two discrepancy fixes join forces for stronger audio-visual ZSL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3330,"prompt_tokens":976,"completion_tokens":2354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2271}},"tokens_in":592,"tokens_out":2354,"duration_ms":15302,"temperature":1.0,"reasoning_tokens":2271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:02.384593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument DAAN during UCF101 training and log the per-sample coefficients $\\eta_{a,p}^i$ and $\\eta_{v,p}^i$; if they are identical for every sample because the optimization-rate formula supplies the same ratio, then CSGM is not actually balancing per-modality contributions, and retraining with CSGM removed should reproduce the reported gains if the true cause is elsewhere.","supporting_citations":[{"cited_title":"Motion-decoupled spiking transformer for audio- visual zero-shot learning,","cited_arxiv_id":null,"evidence_quote":"Provides the MDFT spiking-transformer baseline that DAAN compares against and the runtime-efficiency contrast."},{"cited_title":"Audiovisual generalised zero-shot learning with cross-modal attention and language,","cited_arxiv_id":null,"evidence_quote":"Defines the cross-modal attention baseline and the audio-visual generalized zero-shot learning protocol DAAN builds on."},{"cited_title":"Temporal and cross-modal attention for audio-visual zero-shot learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal cross-modal attention baseline and dataset evaluation setup used for comparison."},{"cited_title":"Characterizing and overcoming the greedy nature of learning in multi- modal deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Motivates the greedy-training diagnosis and supplies the squared gradient-to-parameter norm ratio used as the optimization rate."},{"cited_title":"Balanced multimodal learning via on-the-fly gradient modulation,","cited_arxiv_id":null,"evidence_quote":"Provides the prior on-the-fly gradient modulation strategy that CSGM refines from global accuracy-based to per-sample modulation."},{"cited_title":"Enhancing multimodal cooperation via sample-level modality valuation,","cited_arxiv_id":null,"evidence_quote":"Introduces sample-level modality valuation, the notion of content discrepancy that CSGM operationalizes."},{"cited_title":"Differential transformer,","cited_arxiv_id":null,"evidence_quote":"Inspires the weighted subtraction of softmax attention scores used to build QDMA's sparse attention."}],"review_version":1}