{"id":"b078d557-422d-483a-b1eb-534c7914fcbc","arxiv_id":"2501.19164","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An inference-time visual perturbation, optimized with zeroth-order gradients and CLIP text similarity, modestly reduces object hallucination across 8 LVMs on POPE, BEAF, and CHAIR.","lead":"This paper proposes adding optimized adversarial noise to images before asking a large vision-language model a question, aiming to reduce object hallucinations without retraining the model. The method reports small but consistent gains across 8 models on standard hallucination benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Null-text and distorted-image anchor responses may already encode the same hallucination bias VAP claims to remove, so the Ls2/Ls3 minimization can push perturbed responses toward hallucinated anchors rather than away from bias.","rationale":"The reader identified the null-text response assumption as the weakest assumption, and I agree that this is the most load-bearing point. However, I would sharpen it further: the issue is not only that fθ(x+δ,∅) might be biased/generic, but also that the entire three-term objective (especially Ls2 and Ls3, which the reader mentioned only in passing) relies on x̄ and the null-text response being valid 'visual uncertainty' and 'unbiased' anchors. The paper never validates these anchors and never shows that the loss decomposition—rather than a simple yes/no threshold shift—is what drives the gains. I also note the internal inconsistency in the gradient formula (Eq. 11 vs Algorithm 1 step 11: the zeroth-order estimator should multiply each difference by γn, but Eq. 42 omits the γn term, changing the estimated direction), which further undermines reproducibility. These are addressable, so the verdict should remain CONDITIONAL rather than REJECT: the method is not disproven by the text, but the evidence does not yet establish the claimed mechanism or the strength of the central claim. The proposed concrete test—comparing full VAP against a conservative-anchor control and measuring null-text response quality—would settle whether the concern lands.","tokens_in":23783,"tokens_out":2324,"duration_ms":19734,"concrete_test":"Run a controlled ablation on LLaVA-v1.5 and Intern-VL2 under POPE-Adversarial, decomposing the objective: (a) Ls1 only, (b) Ls2 only, (c) Ls3 only, (d) all three, and (e) a 'bias-shift control' that replaces the null-text anchor fθ(x,∅) with a fixed generic caption 'There is an image.' and the distorted anchor fθ(x̄,∅) with a fixed 'No object is present.' response. If variant (e) matches or exceeds the full VAP gains on F1 and accuracy, the method's benefit comes from pushing responses toward conservative anchors rather than from grounding in visual content. Additionally, measure the null-text response quality on 100 POPE images: if fθ(x,∅) frequently omits or hallucinates the queried object, the Ls1 anchor is not a faithful visual description, and Eq. 3 is optimizing toward the wrong target.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that VAP reduces object hallucinations by optimizing three similarity terms (Eqs. 3, 5, 6) that are supposed to (Ls1) ground the prompted response in visual content and (Ls2/Ls3) reduce parametric knowledge bias by contrasting against a distorted image x̄. This reasoning is load-bearing on the assumption that the anchor responses fθ(x,∅), fθ(x̄,∅), and fθ(x̄,c) are (i) faithful to the image and (ii) free of the same linguistic-prior bias that causes hallucinations. The paper provides no evidence for this. In fact, Table 10 shows that applying VAP does not make models more conservative across the board; it shifts 'yes' ratios dramatically (e.g., LLaVA-v1.5 from 85.3% to 14.7%), which is consistent with the optimized noise suppressing the prompted response's agreement with the null-text response, but it does not establish that the null-text response is visually grounded rather than merely generic or biased. For an image of a stop sign, fθ(x,∅) might say 'a street scene' or even hallucinate an object; maximizing S(fθ(x+δ,c), fθ(x+δ,∅)) would then pull the prompted answer toward that possibly hallucinated description. Likewise, minimizing S(fθ(x+δ,c), fθ(x̄,∅)) and S(fθ(x+δ,∅), fθ(x̄,∅)) assumes x̄ is sufficiently corrupted to suppress parametric bias while still being a useful contrast; Eq. 4 with µ_T and per-model T (Table 5) is never validated on any benchmark, and no ablation shows that Ls2/Ls3 alone reduce POPE/BEAF hallucination rather than merely changing the yes/no threshold. Because the improvements in Tables 1-3 are small (0.34-2.81 points) and no significance tests or confidence intervals are reported, a plausible alternative explanation is that VAP's main effect is to shift the decision boundary (e.g., make the model say 'no' more often), which POPE's F1 can reward if precision rises more than recall falls. Appendix C.1 indeed shows recall often drops slightly while precision rises.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Visual Adversarial Perturbation (VAP), a data-centric method that adds optimized noise to input images to reduce object hallucinations in large vision-language models (LVMs) without modifying the base model. The perturbation is optimized with three similarity-based losses computed in CLIP text-embedding space: maximizing agreement between the prompted response and the null-text response of the perturbed image (Ls1), and minimizing agreement with responses to a distorted version of the image under prompted and unprompted conditions (Ls2 and Ls3). The optimization uses zeroth-order gradient estimation. Experiments on eight LVMs report consistent improvements on POPE, BEAF, and CHAIR benchmarks, with per-model hyperparameters and a proxy-model variant discussed in the appendix.","tokens_in":80,"tokens_out":7505,"duration_ms":142558,"significance":"The idea of repurposing adversarial perturbations as a data-centric cure for hallucination is novel and timely, and the breadth of the evaluation across eight LVMs and three benchmarks is a clear strength. If the effect is real and reproducible, VAP would be a practical, model-agnostic intervention that is orthogonal to decoding-based methods and requires no model modification. However, the current evidence is weakened by a sign inconsistency in the core objective, a missing term in the reported optimization algorithm, the lack of statistical significance testing, and an unvalidated assumption about the null-text anchor. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The objective in Eq. (7) sums a maximization term (Ls1, Eq. (3)) with two minimization terms (Ls2, Eq. (5); Ls3, Eq. (6)) under a single loss LS, and Eq. (9) then sets δ = ∇x LS with the perturbed image updated as x̂ = x + αδ. As written, gradient ascent on LS increases Ls2 and Ls3, which is the opposite of the stated goal of reducing similarity to the distorted anchors. Please clarify the sign convention; for example, if LS is to be maximized, it should be LS = Ls1/σ1^2 − Ls2/σ2^2 − Ls3/σ3^2, or the minimization terms should be negated. This is a load-bearing inconsistency that affects every experiment.","section":"§3.2–3.3, Eqs. (7)–(10)"},{"comment":"The zeroth-order gradient estimator in Algorithm 1 (Eq. 42) omits the random direction γn: δ is written as (1/(Nβ)) Σ_n (L^(n)_S − L^(0)_S), which is a scalar, not a vector. Equation (11) correctly includes γn. As written, the algorithm is not a valid gradient estimate and cannot be reproduced. Please correct to δ = (1/(Nβ)) Σ_n (L^(n)_S − L^(0)_S) γ_n.","section":"Appendix G, Algorithm 1, Eq. (42)"},{"comment":"The Ls1 term maximizes similarity between the prompted response and the same model's null-text response fθ(x+δ,∅). The paper assumes this null-text response is a faithful visual description, but no evidence is provided; if the null-text response itself contains hallucinations or generic priors, Ls1 could reinforce them. Please report the hallucination rates of the null-text responses on the same benchmarks, and perform a control experiment where the null-text anchor is replaced by a human-written or independently generated caption. Without this, the claimed grounding mechanism is not established.","section":"§3.2, Eq. (3)"},{"comment":"No statistical significance tests or confidence intervals are reported. Many improvements are small (e.g., +0.42 F1 for LLaVA-OV in Table 1; +0.18 TU for LLaVA-OV in Table 2), and with 500 POPE samples and 1,000 CHAIR samples these differences may be within sampling noise. Please provide standard deviations over multiple runs or seeds, and paired tests (e.g., bootstrap or McNemar) for the headline metrics.","section":"§4.1–4.2, Tables 1–3"},{"comment":"The balancing coefficients σ1, σ2, σ3 and the distortion timestep T are manually tuned per model. The selection procedure and validation set are not described. Since the reported gains are small, please show that a fixed generic configuration (e.g., all σ=1, T=500) still yields consistent improvements across the eight models, or provide a per-model sensitivity analysis on a held-out split.","section":"§4.1, Table 5"},{"comment":"The noise schedule μ_T is not defined; the paper only references 'the noise scheduling coefficient at timestep T'. Without specifying the schedule (e.g., cosine, linear), the distorted image generation is not reproducible. Moreover, the claim that the distorted image 'quantifies parametric knowledge bias' is not validated independently. Please provide the schedule and an ablation over T showing its effect on the final hallucination metrics.","section":"§4.1, Eq. (4) and Algorithm 1"}],"minor_comments":[{"comment":"Table 1 uses '+AVP' instead of '+VAP' in the LVM column; please make the acronym consistent.","section":"Table 1"},{"comment":"Figure 8(f) contains a typo: 'Is there a bickcle in the image?' should be 'bicycle'.","section":"Figure 8"},{"comment":"Section 4.2 refers to 'Figure 5 presents results from specific examples', but Figure 5 shows perturbation strength; the examples are in Figure 4. Please correct the cross-reference.","section":"§4.2"},{"comment":"In Section 4.2, the phrase 'beneficial visual nosise' should be 'beneficial visual noise'.","section":"§4.2"},{"comment":"The CHAIR evaluation restricts responses to 30 characters, which deviates from the standard CHAIR protocol; please justify this choice and report full-caption results if available.","section":"§4.1, CHAIR setup"},{"comment":"The citation 'Anonymous, 2025' is under review and appears in the related work; please update or anonymize appropriately if allowed by the venue.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites 'Anonymous, 2025' (under review) as a key motivation; this is acceptable but the authors should ensure it does not violate double-blind policies. The project page is provided but no code is linked; for a method with many implementation details, code would substantially help reproducibility. The paper's contribution is more empirical than theoretical, and the sign error in Eq. (7) is fixable but currently makes the method as written impossible to follow; the editor may wish to weigh whether the authors can address the algorithmic and statistical concerns within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper proposes VAP, a training-free method that adds adversarially optimized noise to input images to reduce object hallucinations in LVMs. The combination is new—black-box zeroth-order optimization of a CLIP-text-similarity loss that pulls the prompted response toward the same model's null-text response and away from responses on a distorted image. That is a clever idea, and the paper is honest about some trade-offs, including false-drop analysis and proxy-model generalization. The evaluation is broad: eight LVMs, POPE, BEAF, and CHAIR, plus a Gaussian-noise comparison and a parameter ablation. That is above the usual bar.\n\nThe soft spots are real, though. The gains are mostly 0.3 to 2.1 F1 points, with no confidence intervals or significance tests. Per-model choices of sigma and T give a lot of tuning knobs, and without a held-out validation story that smells like overfitting. The algorithm also has an internal inconsistency: the zeroth-order gradient in Eq. 11 includes the gamma_n term, but Algorithm 1's line 42 just sums the loss differences without it. That needs fixing before anyone can reproduce it.\n\nThe deeper concern is conceptual. The method assumes the null-text response and the distorted-image response are reliable visual anchors. If those anchors are themselves biased or generic, maximizing similarity to them can reinforce hallucination rather than reduce it. Table 10's dramatic yes-ratio drops—LLaVA-v1.5 from 85.3% to 14.7%—are consistent with the perturbation shifting the decision boundary toward \"no,\" not with better visual grounding. Appendix C.1 shows precision rising while recall often falls, which is exactly what a threshold shift looks like. The authors do not address this alternative explanation.\n\nAlso missing: a comparison to existing training-free methods like VCD, OPERA, or HALC on the same benchmarks. They only show VAP+VCD combined, not VCD (or others) alone, so the practical value over known methods is unclear.\n\nWho this is for: researchers working on hallucination mitigation or input-space interventions. It deserves a serious referee because the idea is original and the experimental sweep is broad, but the paper needs code, significance testing, a standalone VCD comparison, and a serious discussion of the anchor assumption and threshold-shift concern before I'd trust the effect. I'd engage with the paper, but I wouldn't cite it yet.","headline":"A genuinely new combination of inference-time adversarial noise and the model's own text anchors, but the evidence mostly supports a threshold shift, not real grounding.","tokens_in":24789,"tokens_out":1778,"would_cite":false,"duration_ms":19558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A bounded adversarial noise pattern added to input images reduces object hallucinations in eight vision-language models, without retraining.","keywords":["object hallucination","large vision-language models","visual adversarial perturbation","hallucination mitigation","zero-gradient optimization","black-box optimization","POPE evaluation","BEAF evaluation"],"falsifier":"Run VAP on a matched pair of image sets, one whose null-text descriptions are verified accurate and one whose null-text descriptions contain an object absent from the image; if hallucination scores improve on the second set as much as on the first, the claim that null-text responses are faithful visual anchors is falsified.","tokens_in":23539,"feed_emoji":"🖼️","tokens_out":8852,"duration_ms":79217,"temperature":0.7,"pith_summary":"This paper claims that a carefully optimized pattern of visual noise, added to an input image before it reaches a large vision-language model (LVM), can suppress object hallucinations without retraining or modifying the model. The method, called visual adversarial perturbation (VAP), treats hallucination as a mismatch between what the model says and what the image actually shows, and searches for a small perturbation that pulls the model's answer toward its own neutral visual description of the image and away from language-driven priors. Across eight LVMs, the perturbed images consistently improve scores on the POPE, BEAF, and CHAIR hallucination benchmarks, while equal-strength Gaussian noise degrades performance. If the effect holds, it gives a practical, model-agnostic input filter for reducing one of the main reliability failures of vision-language systems.","feed_headline":"Adversarial noise makes vision-language models hallucinate less","feed_subtitle":"Adding optimized noise to images improves hallucination scores across eight LVMs, with no retraining.","key_machinery":"The mechanism is a three-term adversarial objective computed in text-embedding space. The first term, $L_{s1}$, anchors the answer to visual content by maximizing similarity between the prompted response and the model's null-prompt response; the second and third terms, $L_{s2}$ and $L_{s3}$, reduce parametric knowledge bias by minimizing similarity between responses to the perturbed image and a noised contrastive negative image, under prompted and null conditions. The combined loss is optimized by a zeroth-order (zero-gradient) estimator, so the perturbation is generated in a fully black-box fashion and requires no access to or modification of the LVM's internals. The bounded perturbation itself is the entire intervention.","core_discovery":"The central claim is that adversarial perturbation, usually treated as a threat to model reliability, can be repurposed as a corrective input filter. For a frozen LVM $f_\\theta$, an image $x$, and a question $c$, VAP maximizes the semantic similarity between the model's prompted answer $f_\\theta(x+\\delta,c)$ and the model's own null-text description $f_\\theta(x+\\delta,\\emptyset)$, while minimizing similarity between responses to the perturbed image and a heavily distorted contrastive image $\\bar{x}$, in both prompted and unprompted settings. The optimized perturbation $\\delta$, bounded to $\\epsilon = 2$ and produced by a zero-gradient estimator that uses a CLIP text encoder as the similarity surrogate, is added to the image before inference. The paper reports consistent hallucination reductions across eight state-of-the-art LVMs under three evaluation settings, with the largest gains under adversarial sampling where parametric knowledge bias is most exposed.","pith_inferences":["A testable extension follows from the paper's yes-ratio shifts: because VAP sharply reduces affirmative responses, much of its POPE gain may come from de-biasing a yes-prior rather than from visual grounding, so a decomposition experiment should separate recalibration from grounding.","Because the null-text response is the anchor, the method inherits any bias in that response; on images where the model's neutral description is itself hallucinated, the optimizer would pull the prompted answer toward that hallucination, making the quality of the unprompted visual summary the real ceiling on improvement.","The input-level, black-box design suggests a deployment reading the paper does not test: a single small proxy model could generate VAP noise once per image, and that noise could be applied before sending images to any commercial LVM API, at a small fraction of the target model's inference cost."],"forward_implications":["On the POPE text-axis benchmark, VAP improves accuracy by up to +2.81 percentage points and F1 by up to +2.09 points across the eight tested LVMs, with the largest gains under adversarial sampling.","On the BEAF before-after benchmark, VAP improves the change-aware True Understanding metric by up to +2.31 points and F1TUID by up to +1.74 points, indicating better sensitivity to object presence changes.","On the CHAIR captioning benchmark, VAP lowers both CHAIRI and CHAIRS for all eight models; Intern-VL2, for example, drops CHAIRI by 0.68 and CHAIRS by 0.90.","VAP requires no retraining or decoding changes and stacks with the decoding-time method VCD for additional CHAIR reductions.","Perturbations optimized on a smaller proxy model transfer to a larger model, giving Intern-VL2-1B to 8B a +1.78 F1 gain at one-eighth the inference cost."],"supporting_citations":[{"why":"Defines the POPE probing benchmark whose accuracy and F1 scores are the paper's primary text-axis evidence.","marker":"(Li et al., 2023)"},{"why":"Supplies the BEAF before-after benchmark with change-aware metrics such as TU and F1TUID.","marker":"(Ye-Bin et al., 2024)"},{"why":"Provides the CHAIR captioning hallucination metrics used for open-ended evaluation.","marker":"(Rohrbach et al., 2018)"},{"why":"Contributes the zeroth-order black-box optimization scheme used to approximate the perturbation gradient.","marker":"(Chen et al., 2017)"},{"why":"Basis for treating LVMs as black boxes with zero-gradient adversarial perturbation.","marker":"(Zhao et al., 2023)"},{"why":"Motivates visual uncertainty as a measure of parametric knowledge bias that the paper's negative-image loss targets.","marker":"(Guan et al., 2024)"},{"why":"Defines visual contrastive decoding, the decoding method VAP is combined with and compared against.","marker":"(Leng et al., 2024)"},{"why":"Provides the random gradient-free optimization theory behind the zero-gradient estimator in Eq. 11.","marker":"(Nesterov & Spokoiny, 2017)"}],"fun_headline_variants":["Perturbing images to stop LVM hallucinations","Adversarial noise flips from attack to cure for LVMs","Optimized image noise reduces LVM hallucinations","Noise that cures object hallucinations in LVMs","Adversarial perturbation: from threat to cure for LVMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's null-text response to an image is a faithful description of what is actually in the image; if that neutral response is itself biased or hallucinated, maximizing the prompted answer's similarity to it could push the model deeper into the same error.","fun_headline_variants_meta":{"raw":{"variants":["Perturbing images to stop LVM hallucinations","Adversarial noise flips from attack to cure for LVMs","Optimized image noise reduces LVM hallucinations","Noise that cures object hallucinations in LVMs","Adversarial perturbation: from threat to cure for LVMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2961,"prompt_tokens":858,"completion_tokens":2103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2024}},"tokens_in":474,"tokens_out":2103,"duration_ms":14560,"temperature":1.0,"reasoning_tokens":2024,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:04:33.019790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VAP on a matched pair of image sets, one whose null-text descriptions are verified accurate and one whose null-text descriptions contain an object absent from the image; if hallucination scores improve on the second set as much as on the first, the claim that null-text responses are faithful visual anchors is falsified.","supporting_citations":[{"cited_title":"Beaf: Observing before-after changes to evaluate hallucination in vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the BEAF before-after benchmark with change-aware metrics such as TU and F1TUID."},{"cited_title":"A., Burns, K., Darrell, T., and Saenko, K","cited_arxiv_id":null,"evidence_quote":"Provides the CHAIR captioning hallucination metrics used for open-ended evaluation."},{"cited_title":"Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models","cited_arxiv_id":null,"evidence_quote":"Contributes the zeroth-order black-box optimization scheme used to approximate the perturbation gradient."},{"cited_title":"M., and Lin, M","cited_arxiv_id":null,"evidence_quote":"Basis for treating LVMs as black boxes with zero-gradient adversarial perturbation."},{"cited_title":"Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models","cited_arxiv_id":null,"evidence_quote":"Motivates visual uncertainty as a measure of parametric knowledge bias that the paper's negative-image loss targets."},{"cited_title":"Mitigating object hallucinations in large vision-language models through visual contrastive decoding","cited_arxiv_id":null,"evidence_quote":"Defines visual contrastive decoding, the decoding method VAP is combined with and compared against."}],"review_version":1}