{"id":"e08c2183-4ddf-4ca8-8246-ed06b41b3ced","arxiv_id":"2505.17061","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoD reduces hallucinations in large vision-language models by measuring the Jensen-Shannon divergence between outputs from full and attention-masked image tokens and switching between complementary and contrastive decoding.","lead":"A new decoding method for vision-language models checks whether the output changes when only the most-attended image tokens are kept, then either amplifies or suppresses the attended information to reduce hallucination. It reports gains across four hallucination benchmarks, but the closest prior method is only compared in an appendix, where it slightly edges out MoD.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoD's consistency gate may be circular: wrong but dominant attention can make pθ(yt|vatt) mirror the hallucinated pθ(yt|v), so low JS routes to complementary (§3.3, Eq.5-6) instead of contrastive.","rationale":"The reader's weakest assumption is on target, and my concern is a sharpened version of it: not only can language priors create spurious agreement, the masked distribution is derived from the same attention that may be wrong, so agreement can be manufactured by the failure mode MoD is supposed to detect. That makes this the load-bearing point: the adaptive switch in Eq. 6, which is the paper's claimed novelty, inherits all of its justification from this implication. If the proposed diagnostic shows vatt-only accuracy is low in the complementary branch, the mechanism section would need substantial revision even if the empirical tables remain intact. The paper does have independent support: a released codebase, shared fixed hyperparameters, and consistent gains across four benchmarks and three models, plus an ablation showing MoD beats always-complementary and always-contrastive. Those positives argue against rejection, but the missing control is exactly the condition under which the central claim could be accepted, so CONDITIONAL remains the right verdict and the paper should be required to add the proposed ablation before the mechanism claim is taken as established. The DeGF appendix comparison and single-seed reporting are real but secondary issues; they temper 'state-of-the-art' wording without bearing on whether the routing signal works.","tokens_in":22387,"tokens_out":9779,"duration_ms":104460,"concrete_test":"On POPE (all three settings) and MME (all three LVLMs), log d(v,vatt) for every decoding step. (1) For questions routed to the complementary branch (d≤γ), decode the same question with vatt alone and score the yes/no answer against ground truth; if vatt-only accuracy is near chance while still agreeing with the full-image answer, low JS does not imply correct attention. (2) Control ablations: replace Eq. 3's top-λ selection with a random λ=0.2 image-token mask, and replace the Eq. 6 threshold with random routing at the same complementary/contrastive rate, over at least 5 seeds. If either control matches MoD within one standard deviation, the attention/consistency mechanism is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 asserts that alignment between pθ(yt|v) and pθ(yt|vatt) 'indicates that the model's attention over image tokens is right,' and Eq. 6 uses d(v,vatt)≤γ to select complementary decoding. The assertion is not secure. vatt is produced by zeroing all but the top-λ attended tokens from the same forward pass (Eq. 2-3). If the model's attention is wrong but strongly committed to those tokens, the attended-only forward pass will reproduce the original hallucinated distribution, because the discarded low-attention tokens contributed little to pθ(yt|v) in the first place. Low JS then triggers the complementary branch, amplifying both the original and the masked logits, exactly when contrastive suppression is needed. Conversely, the paper's own MME 'position' result shows that zeroing low-attention tokens can remove decisive evidence when attention to it is low, inflating d and routing to contrastive. Figure 1 shows a correlation between JS and hallucination, but not that the correlation is driven by attention correctness rather than by sensitivity to any token removal. The central novelty is therefore underspecified: the method may work as a generic partial-image complement/contrast heuristic while the attention-correctness story is false.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixture of Decoding (MoD), a training-free adaptive decoding strategy for mitigating hallucinations in large vision-language models. MoD computes the Jensen-Shannon divergence between the output distribution from the original image tokens and the output distribution from a masked input retaining only the top-λ attended image tokens. If the divergence is below a threshold γ, MoD treats the model's attention as correct and combines the two logit distributions additively; otherwise it treats the attention as erroneous and applies contrastive decoding. The method is evaluated on POPE, MME, CHAIR, and AMBER across LLaVA-1.5, Qwen-VL, and LLaVA-NEXT, and is compared with sampling, VCD, M3ID, and AvisC. The paper reports consistent gains and argues that MoD is robust to hyperparameter choices.","tokens_in":22693,"tokens_out":2778,"duration_ms":28319,"significance":"If the central mechanism is validated, MoD is a useful contribution: it is training-free, model-agnostic, and the reported gains are consistent across four benchmarks and three LVLMs. The paper also provides ablations on the key hyperparameters (λ, α1, α2, γ, β) and releases code, which strengthens reproducibility. The main conceptual novelty is the adaptive switch between complementary and contrastive decoding based on a consistency criterion. However, the validity of this switch rests on the claim that consistency between original and attended-only outputs indicates correct attention. This claim is not directly tested, and the paper's SOTA claim is qualified by the DeGF comparison in Appendix B.5. These issues are substantial but addressable, so the work is promising rather than settled.","major_comments":[{"comment":"The load-bearing premise that low JS divergence between pθ(yt|v) and pθ(yt|vatt) indicates correct attention is not secure. If the model's attention is wrong but strongly committed to the top-λ tokens, then vatt keeps those tokens and pθ(yt|vatt) will closely mirror pθ(yt|v), yielding a low JS value and routing to the complementary branch—exactly when contrastive suppression is needed. Conversely, zeroing low-attention tokens can remove decisive visual evidence when attention to it was low, inflating JS and routing to contrastive even when attention is correct. Figure 1 shows that JS is correlated with hallucination, but it does not show that this correlation is driven by attention correctness rather than by sensitivity to any token removal. The authors should provide a control experiment (e.g., random token masking at the same retention rate) and, ideally, an evaluation of the gate's precision/recall against an independent measure of attention correctness.","section":"§3.3, Eq. (5)–(6)"},{"comment":"All results are reported for a single fixed random seed (seed 42) under temperature=1 and top-p=1. Since the method involves stochastic decoding, the absence of multiple seeds or confidence intervals makes it impossible to assess whether small reported gains (e.g., 86.0 vs. 85.5 for Qwen-VL on POPE random) are statistically meaningful. The authors should rerun the main comparisons over at least three seeds and report means and variances, or otherwise justify why single-seed results are representative.","section":"§4.1, 'Implementation Details'"},{"comment":"The abstract claims MoD 'significantly outperforms existing decoding methods,' but the DeGF comparison in Table 9 shows DeGF achieving a higher total MME score (643.3 vs. 638.3) on LLaVA-1.5. This directly qualifies the SOTA claim. DeGF should either be included in the main experimental tables and discussed in the abstract, or the claims should be restricted to the specific baselines evaluated in the main text. The paper should also report comparisons with DeGF on additional benchmarks and models before making an unqualified superiority claim.","section":"Appendix B.5, Table 9"},{"comment":"The attention aggregation in Eq. (2) averages attention weights uniformly over all layers and heads, and Eq. (3) then selects the top λ proportion of image tokens. This ignores the known heterogeneity of attention heads: some heads may encode visual grounding while others encode positional or textual information. The paper does not justify why uniform averaging is appropriate, nor does it test whether the conclusions change when using a different layer/head subset. The consistency criterion in §3.3 could behave differently under alternative aggregations, so an ablation on the attention aggregation would strengthen the attention-correctness story.","section":"§3.2, Eq. (2)–(3)"}],"minor_comments":[{"comment":"The model name is typeset inconsistently as 'LLaV A' (with a space) in many places; it should be 'LLaVA' throughout.","section":"Throughout"},{"comment":"The text states that γ is varied from 0.02 to 0.08, while the figure caption says the range is 0.01 to 0.10; these should be reconciled.","section":"§4.3, Figure 4"},{"comment":"The notation pθ(yt|v,vatt,x,y<t) overloads the probability symbol; it should clearly denote the modified MoD distribution, not an ordinary conditional distribution given both image representations.","section":"Eq. (6)"},{"comment":"The comparison with DeGF is reported only on MME and only for LLaVA-1.5; this is too narrow to support the general statement that MoD is preferable in terms of accuracy/cost trade-off.","section":"Appendix B.5"},{"comment":"The right panel reports a Pearson correlation of 0.85 for CHAIRi but does not specify over how many samples or which model; adding this information would improve interpretability.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The consistency criterion used in MoD is closely related to the authors' prior work (Liu et al., 2025), and the paper does not clearly position the incremental contribution relative to that work or to DeGF. The editor may wish to ask the authors to clarify novelty and to include DeGF in the main comparison before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth a serious referee, but read the adaptive-switch story as a heuristic, not as proof about attention correctness. MoD is a token-level switch: mask low-attention image tokens, run a second forward pass, and either add the masked logits to the original (low JS) or subtract them (high JS). The experiments are wide — three models, four benchmarks, consistent wins over VCD/M3ID/AvisC — and the ablations cover all four hyperparameters. Code is released. That is real work, and the method will likely become a useful baseline.\n\nThe genuinely new bit is replacing DeGF's generated image with an attention-masked copy of the same image and adding a complementary branch when the two passes agree. That is a legitimate combination of known pieces, and the paper is honest about the lineage. It is not a big conceptual leap.\n\nThe weakest point is the load-bearing interpretation. Section 3.3 claims that low JS between the original and masked outputs indicates the model's attention is right. That does not follow. If attention is wrong but strongly concentrated on the wrong tokens, masking low-attention tokens changes little, JS stays low, and MoD runs the complementary branch, amplifying hallucinated logits exactly when contrastive suppression is needed. The paper's own MME position result shows the mirror image: zeroing low-attention tokens can remove decisive evidence, inflate JS, and send you to contrastive when attention was actually useful. Figure 1 shows a correlation between JS and hallucination, not between JS and attention correctness. The mechanism as described is underspecified; the numbers support a generic partial-image complement/contrast heuristic.\n\nTwo more issues. Everything is single-seed with no error bars, which matters for a sampling-based decoder at temperature 1. And the closest adaptive baseline, DeGF, appears only in Appendix B.5, where it slightly beats MoD on MME (643.3 vs 638.3). The paper admits this and notes DeGF costs extra image generation — a fair trade-off — but the abstract's 'significantly outperforms existing decoding methods' is too strong.\n\nVerdict: conditional accept with revisions. The core result, that MoD reduces hallucination across models and benchmarks, is credible and reproducible in principle. The attention-correctness interpretation should be softened or tested directly (e.g., with misdirection cases or interventions). Add multi-seed results and move DeGF into the main table. A serious editor should send this to review, not desk-reject it.","headline":"A well-ablated decoding heuristic that reduces LVLM hallucination; the attention-correctness interpretation is not supported, and the SOTA claim is undercut by DeGF.","tokens_in":23212,"tokens_out":3598,"would_cite":true,"duration_ms":34241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixture of Decoding reduces vision-language hallucinations by comparing full-image outputs with outputs from the model's attended image tokens and switching between complementary and contrastive decoding at each step.","keywords":["hallucination mitigation","large vision-language models","mixture of decoding","contrastive decoding","attention masking","Jensen-Shannon divergence","training-free decoding","object hallucination"],"falsifier":"Build a set of images with a visually salient decoy that strongly attracts the model's attention while a different object is the correct answer, and check whether hallucinated tokens still occur with $d(v,v_{\\text{att}})\\le\\gamma$; if many do, the consistency criterion is not separating correct from incorrect attention. A quantitative form is to measure, on a large corpus with human-verified object presence, whether the $\\gamma=0.05$ threshold separates hallucinated from non-hallucinated generations better than a language-prior-only baseline.","tokens_in":22210,"feed_emoji":"🖼️","tokens_out":17290,"duration_ms":140686,"temperature":0.7,"pith_summary":"Large vision-language models routinely produce fluent descriptions that mention objects or attributes absent from the image. The paper claims that this failure can be caught during decoding by comparing the next-token distribution predicted from the full image with the distribution predicted from only the image tokens the model's attention concentrates on. When the two agree, Mixture of Decoding (MoD) treats the attention as correct and adds the two logit streams to amplify what the model is looking at; when they diverge past a fixed threshold, MoD treats the attention as misleading and subtracts the attended-token logits from the original ones. The switch is training-free, token-by-token, and uses one shared set of hyperparameters across models and benchmarks. On POPE, MME, CHAIR, and AMBER, MoD is reported to outperform VCD, M3ID, and AvisC on LLaVA-1.5, Qwen-VL, and LLaVA-NEXT, which matters because it suggests hallucination can be reduced without retraining or extra data.","feed_headline":"Attention check steers decoding to cut vision-language hallucinations","feed_subtitle":"No retraining, no per-task tuning: the model's own attention decides whether to amplify or suppress each token.","key_machinery":"The load-bearing mechanism is the attention-masked image $v_{\\text{att}}$: the original image token sequence with every token outside the top $\\lambda=0.2$ fraction of average attention (computed over all layers and heads from the final input token's attention to image tokens) zeroed out. The model is run twice—once on the full image tokens $v$ and once on $v_{\\text{att}}$—and the Jensen-Shannon divergence $d(v,v_{\\text{att}})$ between the two next-token distributions is the switch that selects between the complementary and contrastive branches of Eq. (6). Figure 1 supplies the paper's evidence that the switch is meaningful: non-hallucinated POPE outputs cluster at low divergence, and CHAIRi correlates with divergence at Pearson $r=0.85$.","core_discovery":"MoD's central claim is that a single divergence number—the Jensen-Shannon divergence between $p_\\theta(y_t\\mid v,x,y_{<t})$ and $p_\\theta(y_t\\mid v_{\\text{att}},x,y_{<t})$, where $v_{\\text{att}}$ is the image with all but the top $\\lambda$ attended tokens zeroed out—tells whether the model's visual attention is reliable at the current step. If $d(v,v_{\\text{att}})\\le\\gamma$, the outputs are consistent, the attention is judged correct, and MoD complements the logits (adding $\\alpha_1$ times the attended-token logits to the full-image logits). If $d(v,v_{\\text{att}})>\\gamma$, the attention is judged wrong, and MoD contrasts (weighting the full-image logits by $1+\\alpha_2$ and subtracting $\\alpha_2$ times the attended-token logits). This per-token switch is applied without training and with shared hyperparameters ($\\lambda=0.2$, $\\alpha_1=4$, $\\alpha_2=1$, $\\gamma=0.05$) across three LVLM architectures, and the paper reports consistent gains over VCD, M3ID, and AvisC on object, attribute, and relation hallucination benchmarks.","pith_inferences":["One extension the authors do not test: replacing the hard zero-mask with soft attention-weighted pooling could preserve spatial layout and recover some of the position-sensitive MME scores that MoD currently sacrifices.","Since the divergence is computed from two forward passes at every step, caching the attended-token pass or sharing cached attention states could cut MoD's doubling of latency, a limitation the authors acknowledge.","The same consistency criterion could serve as a standalone zero-shot hallucination detector for LVLMs, flagging tokens where the full-image and attended-image distributions diverge, without needing the complementary or contrastive branches.","A testable refinement is to make the threshold or the attended-token proportion adaptive to model or input difficulty; the paper shows stability over a range but does not explore per-image thresholds."],"forward_implications":["MoD is a training-free, model-agnostic intervention: the same hyperparameters are reported to work on three different LVLM architectures and four benchmarks, so it can be layered onto already deployed models without retraining.","Because the consistency test is applied per generated token, the method can react within a single response, suppressing a hallucinated object as it is about to be generated and returning to complementary decoding when attention is back on track.","The reported gains are largest on false-positive and object-existence errors (POPE precision, MME existence, CHAIRi), suggesting the divergence switch specifically reduces the yes-saying and object-insertion tendencies of LVLMs.","At the same two-forward-pass cost as VCD, M3ID, and AvisC, the adaptive mixture beats either branch used alone, as shown in the ablation where MoD outperforms both pure complementary and pure contrastive decoding on MME."],"supporting_citations":[{"why":"Supplies VCD, the Gaussian-noise contrastive decoding baseline that MoD is compared against and whose language-prior account MoD extends.","marker":"Leng et al., 2024"},{"why":"Supplies M3ID, the text-only contrastive baseline, and MoD is measured against it on all four benchmarks.","marker":"Favero et al., 2024"},{"why":"Supplies AvisC, the attention-based contrastive decoder that MoD refines by deciding when attended tokens should be amplified rather than suppressed.","marker":"Woo et al., 2024"},{"why":"Defines POPE, the object-hallucination benchmark used to measure MoD's discriminative accuracy and precision.","marker":"Li et al., 2023"},{"why":"Defines MME and its existence, count, position, and color scores, a primary benchmark for MoD's hallucination reduction.","marker":"Fu et al., 2023"},{"why":"Defines CHAIRs and CHAIRi, the captioning hallucination metrics used to evaluate MoD on generative outputs.","marker":"Rohrbach et al., 2018"},{"why":"Defines AMBER and its unified hallucination score covering object, attribute, and relation errors in discriminative and generative tasks.","marker":"Wang et al., 2023a"},{"why":"The LLaVA-1.5 model on which MoD is evaluated and where the attended-image visualization is produced.","marker":"Liu et al., 2024b"},{"why":"The Qwen-VL model that provides the second architecture on which MoD's model-agnostic claims are tested.","marker":"Bai et al., 2023b"},{"why":"The LLaVA-NEXT model that provides the third architecture and the anyres mechanism cited in the position-subset analysis.","marker":"Liu et al., 2024a"}],"fun_headline_variants":["Attention decides: amplify or suppress to fight LVLM hallucinations","Mixture of Decoding: attention-guided logit mixing halts hallucinations","No retraining: adaptive decoding uses attention to cut vision hallucinations","Per-token attention check picks amplify or suppress to reduce hallucination","LVLM hallucination fix: adaptive logit blending based on attention consistency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that agreement between the full-image output and the attended-image output is a reliable sign the model is attending correctly, so a fixed threshold can tell good from bad attention rather than both outputs merely reflecting the same language prior.","fun_headline_variants_meta":{"raw":{"variants":["Attention decides: amplify or suppress to fight LVLM hallucinations","Mixture of Decoding: attention-guided logit mixing halts hallucinations","No retraining: adaptive decoding uses attention to cut vision hallucinations","Per-token attention check picks amplify or suppress to reduce hallucination","LVLM hallucination fix: adaptive logit blending based on attention consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3487,"prompt_tokens":994,"completion_tokens":2493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2403}},"tokens_in":610,"tokens_out":2493,"duration_ms":16393,"temperature":1.0,"reasoning_tokens":2403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:44:29.339656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of images with a visually salient decoy that strongly attracts the model's attention while a different object is the correct answer, and check whether hallucinated tokens still occur with $d(v,v_{\\text{att}})\\le\\gamma$; if many do, the consistency criterion is not separating correct from incorrect attention. A quantitative form is to measure, on a large corpus with human-verified object presence, whether the $\\gamma=0.05$ threshold separates hallucinated from non-hallucinated generations better than a language-prior-only baseline.","supporting_citations":[],"review_version":1}