{"id":"102a3afd-44ed-4167-b0bb-75cb1dc0dfc5","arxiv_id":"2412.10817","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Injected learned Gaussian noise, tuned with a variational loss, improves CLIP's few-shot classification accuracy slightly over CoOp and CLIP-Adapter.","lead":"This paper proposes PiNI, a method that fine-tunes the frozen vision-language model CLIP by injecting learned noise into both image and text representations. It reports small but consistent accuracy gains over prompt-tuning and adapter baselines on 11 few-shot classification datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never shows that the stochastic noise component is what helps; with no regularization on Σ, Eq. (13) can be minimized by a deterministic μ-only perturbation, so the claimed π-noise mechanism is not established.","rationale":"I partially disagree with the reader's identification of the weakest assumption. The Eq. (9) step is indeed invalid as written: ε⊥y | ρ does not imply p(ε|x,y,ρ)=p(ε|x,ρ). But the actual noise generator takes x as input and samples independently of y, so p(ε|x,y,P)=p(ε|x,P) follows from ε⊥y | x,ρ, not from ε⊥y | ρ. The stated assumption can be corrected, and the variational derivation can be repaired. The deeper, load-bearing issue is that the method never demonstrates that the stochastic part of the noise contributes to the reported gains. The objective in Eq. (13) has no mechanism to force nonzero variance, and the paper provides no evidence that the learned distribution satisfies I(T,E)>0. The empirical results across 11 datasets may be correct, but the central scientific claim that fine-tuning is achieved through beneficial noise is not established without the σ ablation. This justifies a conditional acceptance: the authors should provide the ablation and correct the independence statement.","tokens_in":14899,"tokens_out":15452,"duration_ms":151416,"concrete_test":"Run a controlled ablation on the same 11 datasets at 16 shots with three variants: (i) full learned Gaussian (μ,Σ); (ii) μ-only, with Σ fixed to 0; and (iii) full Gaussian re-run with fixed seeds, reporting mean accuracy and standard deviation over at least 3 seeds, plus the average learned ||Σ||. If (ii) matches (i) within error bars, the stochastic component is not load-bearing and the π-noise claim fails; if (ii) is significantly worse, the concern is resolved. As a secondary check, re-derive Eq. (9) from ε⊥y | x,ρ to confirm the variational bound is repairable.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that PiNI works by learning positive-incentive noise. The training loss in Eq. (13) is an unregularized expectation of -log q(y | x, G_θ(ϵ,x,P), P), with ε = Σθ(x,P)·ϵ + μθ(x,P) drawn from an uncorrelated Gaussian. Nothing in the objective penalizes small Σ: the optimizer can drive the variance toward zero and still minimize the loss, turning the injected 'noise' into a deterministic, input-dependent offset μθ(x,P). The paper reports no learned variance values and no comparison with a μ-only (Σ≡0) variant, and it never checks the defining condition I(T,E)>0 from Eq. (1). If the gains persist with Σ≡0, the method is functionally a learned residual feature adapter rather than a noise-based fine-tuning scheme, and the claimed connection to π-noise is unsupported. This concern is independent of the Eq. (9) issue raised by the reader; even if that derivation is repaired by conditioning ε⊥y | x,ρ (which the actual sampling procedure satisfies), the missing σ control remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Positive-incentive Noise Injector (PiNI), a method for fine-tuning frozen CLIP by injecting learned noise into the visual and text encoders for few-shot classification. The authors reformulate CLIP inference with a prompt variable, define task entropy, derive a variational upper bound on conditional entropy, and learn Gaussian noise parameters (mu, Sigma) using MLP, CNN, cross-attention, or learnable embeddings. Experiments on 11 datasets report consistent gains over zero-shot CLIP, CoOp, CLIP-Adapter, and Linear Probe, with larger improvements at lower shot counts; additional experiments cover domain generalization and different visual backbones. The paper claims this is the first noise-based fine-tuning scheme for CLIP and that the learned noise is positive-incentive noise in the sense of Eq. (1).","tokens_in":15166,"tokens_out":6649,"duration_ms":60688,"significance":"If the reported gains are robust, PiNI would be a genuinely simple and parameter-efficient alternative to prompt- and adapter-based PEFT, and the code release and broad benchmark are useful assets. The paper also provides a clear exploration of injection locations and generator architectures. However, the theoretical derivation in Section 3.3 contains a load-bearing conditional-independence gap, and the experiments do not currently establish that stochastic noise, rather than a deterministic learnable offset, is responsible for the improvements. Because these points can be addressed with a corrected derivation and targeted ablations, the significance is conditional on that additional evidence.","major_comments":[{"comment":"Equation (9) states that from ε ⊥ y | ρ one can obtain p(ε|x,y,ρ)=p(ε|x,ρ). This inference is invalid: conditional independence of ε and y given ρ does not imply conditional independence given x, and in the graphical model of Figure 2, x may be a collider or a descendant that creates dependence. The equality is needed to pass from Eq. (8) to Eq. (10), so as written the variational upper bound and the claim that the learned noise minimizes H(T|E) do not follow. The correct fix is to define the sampling mechanism so that ε is generated from x and P independently of y, i.e., ε ⊥ y | (x,ρ), and to state that assumption explicitly.","section":"§3.3, Eq. (9)"},{"comment":"The loss in Eq. (13) is an unregularized expectation of −log q(yi | xi, Gθ(ϵij, xi, P), P) with Gθ(ϵ, x, P) = Σθ(x,P)ϵ + μθ(x,P). Nothing in this objective prevents Σθ from collapsing to zero, in which case the 'noise' reduces to a deterministic input-dependent offset μθ and the method becomes a residual feature adapter. The paper does not report learned variance values, does not compare with a μ-only (Σ≡0) variant, and does not check the defining condition I(T,E)>0 in Eq. (1). The central mechanistic claim that beneficial stochastic noise drives the gains is therefore unsupported; a variance-controlled ablation and a report of Σ statistics are needed.","section":"§3.3–3.4, Eq. (13)"},{"comment":"The default combination of injection location and generator architecture is selected from Table 1, which reports 16-shot accuracy on the same datasets used in the main comparisons, but the paper does not state whether this selection was made on a validation split or on the test set. If the latter, the subsequent comparisons to CoOp and CLIP-Adapter are optimistically biased by test-set selection. Please clarify the selection procedure and, if necessary, re-evaluate the chosen configuration on held-out data.","section":"§4.2, Table 1"},{"comment":"The prompts used for PiNI differ from those used for the baselines: Section A.2 states that the template is modified to place all category names at the end and that random words are inserted at the beginning, whereas CLIP, CoOp, and CLIP-Adapter use the hand-crafted template. Since the reported gains over CoOp are small on several datasets, the template change is a plausible confound. A control experiment that evaluates CoOp (or zero-shot CLIP) with the same modified template, or ablates PiNI with the original template, should be reported.","section":"§A.2, §4.3"}],"minor_comments":[{"comment":"The sentence referring to Zhang, Zhu, and Li (2024) and GNNs appears unrelated to the surrounding discussion of noise generators and should be removed or integrated properly.","section":"§3.4"},{"comment":"The phrase 'the noise has the same shape as the input at the injection location' should be made precise for the text side, where the noise shape depends on the number of prompt tokens and the embedding dimension.","section":"§3.4"},{"comment":"The nearest-word evidence does not by itself demonstrate richer semantics; if Σ collapses, the sampled prompts are nearly identical, so the table should be accompanied by variance statistics for the learned prompt distribution.","section":"§B.3, Table 8"},{"comment":"The phrase 'conditional independent' should be 'conditionally independent' (e.g., Section 3.4), and the notation in Figure 2 should be described in a way that makes the assumed generative direction explicit.","section":"Throughout"},{"comment":"The paper should state whether performance numbers are averaged over multiple random seeds and should report standard deviations, since the few-shot regime is known to be noisy and several reported margins are small.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The authors should be given the chance to address the variance-collapse and test-selection concerns. If their ablations show that σ>0 is essential and that the configuration selection was done on validation, the empirical contribution is solid. The theoretical framing could be simplified or corrected by stating ε ⊥ y | (x,ρ) as the sampling assumption rather than deriving it from a weaker independence claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the empirical gains look real—PiNI consistently beats CoOp and CLIP-Adapter across 11 datasets, especially in the 1–2 shot regime. But the paper's central claim, that these gains come from learned 'π-noise', is not established. There's a concrete technical flaw in the derivation and a more fundamental issue: nothing in the training objective prevents the noise variance from collapsing to zero.\n\nWhat's actually new: the specific combination of injecting learned stochastic perturbations into both visual and text streams for CLIP fine-tuning hasn't been done before, as far as I know. The experiments are thorough: 11 datasets, shot counts from 1 to 16, domain generalization, multiple backbones, and a reasonable exploration of noise generator architectures and injection points. Gains over CoOp are modest (1–3 points on average) but consistent, and the low-shot improvement is genuinely useful. Code is public.\n\nThe soft spots are real. First, Eq. (9) is wrong: from ε⊥y|ρ you cannot conclude p(ε|x,y,ρ)=p(ε|x,ρ). Conditioning on x matters. Without that step, the variational bound in Eq. (13) isn't an upper bound on H(T|E) as claimed, so the π-noise interpretation doesn't follow. Second, and more damaging, the loss in Eq. (13) has no term that keeps Σθ from shrinking. The reparameterization can trivially set σ to zero and the \"noise\" becomes a deterministic offset μ(x,P). The paper never reports learned variances, never runs a Σ≡0 ablation, and never checks the defining condition I(T,E)>0. For all we know, the method works because it learns a residual feature transformation, not because noise helps. That would make the title and the theoretical framing misleading.\n\nMinor: the best generator configuration in Table 1 is selected on the same datasets used for final reporting, and there are no error bars, so the 1–3 point improvements may be within run-to-run noise.\n\nWho this is for: people doing parameter-efficient CLIP fine-tuning. They'll find the empirical recipe and the design space useful, but they should ignore the variational derivation.\n\nMy recommendation: send to peer review, but with strong pressure for major revision. The empirical contribution deserves scrutiny, and the authors need to add a σ→0 ablation, report variance parameters, and either fix the derivation or drop the π-noise framing. As written, the central claim is not supported.","headline":"Empirical gains look real, but the theoretical claim that PiNI works via beneficial 'π-noise' is unsupported: the key independence step is invalid and the training loss allows the noise variance to collapse to zero.","tokens_in":15666,"tokens_out":3988,"would_cite":false,"duration_ms":36623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a frozen CLIP can be adapted to downstream classification by learning to inject 'positive-incentive noise' into its visual and text encoders, without adding prompt or adapter modules, and that this outperforms prompt…","keywords":["vision-language models","CLIP","few-shot classification","parameter-efficient fine-tuning","positive-incentive noise","noise injection","variational inference","prompt distribution"],"falsifier":"Estimate the true task entropy H(T|E) on a held-out set by sampling noise from the learned generator and measuring the conditional entropy of labels given noisy embeddings, and compare it with the variational loss L from Eq. (13); if L is not an upper bound, the variational justification fails. A simpler check: train PiNI on a dataset where the label is strongly correlated with low-level image statistics, such as color-biased synthetic data, and test whether the noise generator's output changes when the label is revealed; if it does, the conditional-independence assumption ε ⊥ y | ρ is violated.","tokens_in":14687,"feed_emoji":"🎲","tokens_out":5759,"duration_ms":49139,"temperature":0.7,"pith_summary":"The paper argues that a frozen vision-language model can be adapted to a downstream classification task not by adding learnable prompts or adapter modules, but by learning to inject customized noise into the visual and text encoders. This noise is meant to be 'positive-incentive noise', which should reduce the uncertainty of the prediction task rather than corrupting it. The authors reformulate CLIP's inference as a probabilistic process involving an image, a prompt, and a noise variable, then derive a variational training objective that produces noise from a Gaussian distribution whose mean and variance are predicted by small learned networks. On 11 few-shot classification datasets, the proposed method, PiNI, beats zero-shot CLIP, CoOp, and CLIP-Adapter, with the largest margins at very small training-set sizes. If correct, it shows that the learnable parameters of parameter-efficient fine-tuning can live in the noise distribution rather than in the model architecture.","feed_headline":"Learned noise, not new modules, fine-tunes CLIP","feed_subtitle":"PiNI injects customized noise into a frozen CLIP and beats CoOp, CLIP-Adapter, and zero-shot CLIP on 11 few-shot datasets.","key_machinery":"The central object is the learned noise distribution p(ε|x, P), modeled as a diagonal Gaussian whose parameters (μ, Σ) come from a small network fθ(x, P), with samples produced via the reparameterization trick ε = Σ·ϵ + μ. The argument is carried by the task-entropy inequality I(T, E) = H(T) − H(T|E) > 0, which states that the noise reduces classification uncertainty. To make this computable, the paper reformulates CLIP inference with an explicit prompt variable, defines H(T|E) as an expectation over images, labels, prompts, and noise, and replaces the intractable posterior p(y|x, ε, P) with a tractable variational approximation q, yielding the training loss in Eq. (13). The noise is factored into visual and text components, injected respectively into the visual features, using a cross-attention generator, and the prompt embeddings, using learnable embeddings.","core_discovery":"The central claim is that customizing the noise distribution itself is a viable mechanism for aligning vision and language in a frozen CLIP. The paper shows that if noise ε satisfies I(T, E) > 0, meaning it reduces the entropy of the task, then injecting such π-noise into both the visual encoder's input or features and the text encoder's prompt embeddings can improve classification. They reframe CLIP inference as p(y|x, P) with an explicit prompt set P, define task entropy H(T) = H(y|x, P), and derive a variational upper bound whose minimization produces noise that simplifies the task. The resulting PiNI trains only the small networks that output the noise's mean and variance, leaving all CLIP weights frozen. Empirically, PiNI outperforms the baselines on 11 datasets and shows robustness to distribution shift, especially in low-shot regimes.","pith_inferences":["The paper does not test this, but the empirical gains might partly come from stochastic embedding augmentation rather than from true π-noise; a direct check would compare PiNI against injecting random Gaussian noise with the same learned mean and variance.","If the conditional-independence assumption ε ⊥ y | ρ fails in practice, the variational bound in Eq. (6) may still hold approximately, but the theoretical justification would reduce to a heuristic; this could be tested by estimating the true conditional entropy of labels given noisy embeddings on a held-out set.","The same 'learn a noise distribution, keep the base model frozen' recipe could transfer to other dual-encoder architectures and tasks such as retrieval, VQA, or open-vocabulary detection by defining an appropriate task entropy for each.","Injecting noise at inference time, as the paper does, effectively turns a single deterministic prompt into a distribution over prompts; this could serve as a lightweight alternative to test-time ensembling over multiple hand-crafted prompts."],"forward_implications":["PiNI fine-tunes a frozen CLIP by learning only the parameters of small noise generators, so the adaptation cost does not scale with the backbone size.","The method's gains are largest at 1, 2, and 4 shots per class, suggesting noise injection is especially sample-efficient when training data are scarce.","Because the noise generator is learned on the source distribution, the same noise-injection procedure transfers to out-of-distribution ImageNet variants, improving average robustness over all three baselines.","The learned prompt distribution is interpretable: sampling from it yields synonyms and grammatical variants such as 'photo', 'photos', 'pic', and 'picture', which shows the method increases semantic diversity rather than only fitting the training classes."],"supporting_citations":[{"why":"Supplies CLIP, the frozen vision-language model that PiNI fine-tunes, and the zero-shot baseline.","marker":"(Radford et al. 2021)"},{"why":"Defines Positive-incentive Noise and the mutual-information condition I(T, E) > 0 that motivates the method.","marker":"(Li 2022)"},{"why":"Establishes the variational π-noise framework that the paper adapts to CLIP.","marker":"(Zhang, Huang, and Li 2023)"},{"why":"Supplies the variational inference upper bound used to make the training loss tractable.","marker":"(Blei, Kucukelbir, and McAuliffe 2017)"},{"why":"Supplies the reparameterization trick used to sample noise while preserving gradient flow.","marker":"(Kingma and Welling 2014)"},{"why":"Provides CoOp, the main prompt-tuning baseline, and the dataset protocol the experiments follow.","marker":"(Zhou et al. 2022b)"},{"why":"Provides CLIP-Adapter, the adapter baseline that PiNI is compared against.","marker":"(Gao et al. 2024b)"},{"why":"Connects contrastive data augmentation to positive-incentive noise, helping the paper distinguish its method from augmentation.","marker":"(Zhang, Xu, Huang, and Li 2024)"}],"fun_headline_variants":["Custom noise, not extra modules, adapts CLIP","PiNI tune CLIP by learning task-beneficial noise","Noise that reduces task entropy aligns CLIP modalities","Fine-tune CLIP with positive-incentive noise only","Injecting learned noise fine-tunes frozen CLIP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire theoretical derivation assumes the learned noise contains no information about the true class beyond what the prompt already provides; if the noise distribution actually depends on the label, the proof that the training loss bounds the task entropy breaks, even though the method might still work in practice.","fun_headline_variants_meta":{"raw":{"variants":["Custom noise, not extra modules, adapts CLIP","PiNI tune CLIP by learning task-beneficial noise","Noise that reduces task entropy aligns CLIP modalities","Fine-tune CLIP with positive-incentive noise only","Injecting learned noise fine-tunes frozen CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1517,"prompt_tokens":950,"completion_tokens":567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":486}},"tokens_in":566,"tokens_out":567,"duration_ms":4982,"temperature":1.0,"reasoning_tokens":486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:36:05.074332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate the true task entropy H(T|E) on a held-out set by sampling noise from the learned generator and measuring the conditional entropy of labels given noisy embeddings, and compare it with the variational loss L from Eq. (13); if L is not an upper bound, the variational justification fails. A simpler check: train PiNI on a dataset where the label is strongly correlated with low-level image statistics, such as color-biased synthetic data, and test whether the noise generator's output changes when the label is revealed; if it does, the conditional-independence assumption ε ⊥ y | ρ is violated.","supporting_citations":[{"cited_title":"P.; and Welling, M","cited_arxiv_id":null,"evidence_quote":"Supplies the reparameterization trick used to sample noise while preserving gradient flow."}],"review_version":1}