{"id":"9a0c28b5-024a-4438-b244-38310f1657d8","arxiv_id":"2411.14279","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Dual attention plus a learned null-image prompt reduces reported hallucination rates in LLaVA-1.5, but the gains rely on test-set-tuned hyperparameters.","lead":"A two-part method to cut hallucinations in vision-language models: a dual-attention block that forces deep layers to keep looking at the image, and a decoding scheme that contrasts normal outputs with outputs using a learned placeholder image. Reported gains are large, but they rest on hyperparameters tuned on the evaluation benchmarks and on an attention mask that is printed as non-causal.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1)-(3) define a text attention mask that is anti-causal as written (i <= j) and a multiplicative mask that does not actually mask logits; if implemented literally, MDA is not the architecture whose benchmark gains are reported.","rationale":"The reader's weakest assumption is the same blocker I would raise first. The anti-causal direction in Eq. (1) and the multiplicative-mask semantics in Eq. (2) mean the described MDA is not a causal plus bidirectional attention mechanism; a literal implementation would either leak future text during training or produce degenerate attention at decoding time. Since MDA is applied throughout training and inference and is the mechanism behind Table 3's FastV analysis, the reported reductions in hallucination cannot be assigned to the architecture as written. I agree with the REJECT verdict: the manuscript needs a corrected mask specification and, ideally, code verification. A secondary concern also stands even if the mask is corrected: Appendix D and Section 4.3 report picking lambda and theta by performance on LLaVA-Bench, MMBench, and MMHal-Bench themselves, so the headline numbers may reflect test-set hyperparameter selection. That concern is not the first blocker, but it should be fixed with a held-out validation split in revision.","tokens_in":21765,"tokens_out":8916,"duration_ms":89175,"concrete_test":"Download the released implementation linked from the paper and inspect the MDA mask construction: first compare the inequality in M_T (i <= j versus j <= i), and second check whether zero entries are added as -inf before softmax rather than multiplied as in Eq. (2). Then run the 7B model on Object HalBench using the literal Eq. (1)-(3) as a control. If the literal mask yields degenerate text attention, Table 1's gains cannot be attributed to the described MDA; if the released code uses the corrected mask, the paper's equations need revision before the empirical claims can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (1) defines M_T[i,j] = 1 when s_j is a text token and i <= j, which is anti-causal: query position i may attend to text positions j > i. The paper simultaneously claims causal text attention is retained. If the formula is literal, then during autoregressive decoding a generated token at step i has no text keys with j < i available, so it can attend only to itself in the text branch; coherent generation and the reported benchmark gains could not come from this mechanism. Eq. (2) is also not a well-formed mask: writing softmax(QK^T/sqrt(d_k) * M) with M in {0,1} leaves exp(0)=1 for masked positions, and if the multiplication is instead applied after softmax, the rows are no longer normalized. The central empirical claim therefore depends on an unstated corrected implementation, presumably j <= i plus additive -inf masking. The manuscript does not specify this correction, so the architecture evaluated cannot be recovered from the equations alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LACING, a framework to reduce language bias in large vision-language models. It introduces Multimodal Dual-Attention (MDA), which computes separate attention weights for visual and text tokens and combines them, and Soft-Image Guidance (SIG), which trains a learnable soft visual prompt and uses it at inference to contrast the output distributions of original and multimodal-null inputs. The method is evaluated on LLaVA-1.5 7B/13B across object hallucination, MMHal-Bench, LLaVA-Bench, MMBench, TextVQA, and MM-VET, reporting large gains such as a 40.36% relative reduction in Object HalBench response-level hallucination and a 12.11% absolute gain on LLaVA-Bench for the 7B model.","tokens_in":22041,"tokens_out":6059,"duration_ms":54571,"significance":"If the reported results are trustworthy, the framework is significant because it addresses language bias from both training and inference perspectives with only a small number of additional parameters and no extra training data. The paper also contains a self-contained Bayesian derivation for the contrastive decoding update and a useful attention-allocation analysis. However, the current manuscript does not support the reported numbers: the attention-mask equations are internally inconsistent with the claimed causal mechanism, and the main hyperparameters are selected on the same benchmarks used for the final results. These issues must be resolved before the significance of the contribution can be assessed.","major_comments":[{"comment":"The text-attention mask is defined as M_T[i,j] = 1 when s_j is a text token and i <= j, which is anti-causal: a query at position i may attend to text positions j > i. This contradicts the paper's repeated statement that MDA retains causal attention for text inputs. If the formula is interpreted literally, an autoregressive token at step i has no earlier text keys available and can attend only to itself in the text branch, which is incompatible with coherent generation and with the reported benchmark results. The experiments must therefore be using a different mask, presumably j <= i, but this correction is never stated.","section":"§3.1, Eq. (1)"},{"comment":"Equation (2) defines W_I and W_T as softmax(QK^T/sqrt(d_k) * M) with M in {0,1}. A binary mask multiplied elementwise before the softmax does not actually mask logits: masked positions contribute exp(0)=1 to both numerator and denominator. If the multiplication is instead applied after softmax, the attention rows are no longer normalized. The paper does not specify the actual implementation, such as additive negative-infinity masking, so the architecture whose results are reported cannot be recovered from the equations as written.","section":"§3.1, Eq. (2)"},{"comment":"The scaling parameter lambda and replacement probability theta are said to be determined on the validation set (§4.1), but the reported sweeps in Figures 5, 6, and 7 and in Table 8 are performed on the same benchmarks whose final scores appear in the main tables: LLaVA-Bench, MMHal-Bench, and MMBench. The headline improvements are therefore partly fitted to the evaluation benchmarks. An unbiased assessment requires either a separate validation split or a single pre-specified configuration.","section":"§4.1 and Appendix D"},{"comment":"The contrastive decoding formula rests on identifying the soft-prompt logits l_u with the unconditional log-probability log p(y_t). The paper does not provide any analysis showing that the distribution produced by the learned soft visual prompt matches the text-only unconditional distribution. Without such evidence, the Bayes-derived adjustment in Eq. (9) is an asserted approximation, and the effectiveness of SIG depends on an unvalidated quantity.","section":"§3.2, Eq. (10)"}],"minor_comments":[{"comment":"The abstract uses the acronym IFG, while the method is consistently named Soft-Image Guidance (SIG) in the body; please use one acronym throughout.","section":"Abstract and §3.2"},{"comment":"Section 4.3 refers to 'Table 2' for the main comparison of baselines, but the corresponding table is labeled Table 1; the text also cites 'VDD [2]' where the correct reference is [66].","section":"§4.3 and References"},{"comment":"The MDA + FastV row reports scores around 10.5 on LLaVA-Bench with deltas of about −50 to −72; these values look implausible and should be checked for unit errors or mislabeling.","section":"Table 3"},{"comment":"The x-axis label 'N/A' is unclear; it should be replaced with the actual baseline configuration, for example lambda = 1.","section":"Figures 5–7"},{"comment":"The human evaluation reports only majority voting without inter-annotator agreement; please report agreement statistics or at least the per-annotator breakdown.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the paper addresses an important problem, but the two load-bearing issues—the unrecoverable attention-mask specification and the selection of hyperparameters on the evaluation benchmarks—prevent the reported empirical claims from being trusted. A major revision should require a corrected and precise formulation of Eqs. (1)–(2), a clear description of the actual masking implementation, and a re-evaluation with a proper validation split or a fixed hyperparameter setting. If the authors cannot provide these, the manuscript should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing to know before you invest time: the core idea is genuinely new, but the paper as written has a load-bearing formula error and a test-set tuning problem. I'd still send it to review, but it needs major surgery.\n\nWhat's new: LACING combines a dual-attention mechanism (separate masks for visual and text tokens, bidirectional visual attention, causal text attention) with a learned soft visual prompt that is trained to replace image tokens and then used at inference as the \"null\" input for contrastive decoding. The soft prompt is more principled than removing images or adding noise, because it preserves the input pattern. The Bayesian derivation (Eqs. 4-9) is standard—it reduces to the usual contrastive scheme—but it's cleanly presented. The paper also includes ablations, human evaluation, attention maps, and a reasonable set of baselines.\n\nThe problems are real. Eq. (1) prints M_T[i,j]=1 when i <= j, which is anti-causal: query i attends to text keys j >= i. The text says causal attention is retained, so this has to be a typo for j <= i, but it's never flagged. Taken literally, autoregressive decoding would attend only to the current token in the text branch, and the reported generations couldn't happen. Eq. (2) is also not a well-formed mask: softmax(QK^T/sqrt(d_k)) * M with binary M either leaves exp(0)=1 for masked positions (before softmax) or breaks row normalization (after softmax). The paper doesn't say which is intended. Even if the intent is obvious, the equations don't define the architecture.\n\nSecond, the hyperparameters. Section 4.1 says λ and θ are chosen on the validation set, but the appendix (Figs. 5-7, Table 8) sweeps λ on LLaVA-Bench, MMHal-Bench, and MMBench, and θ on MMBench and LLaVA-Bench—the very benchmarks used for the headline numbers. That's tuning on the test set. The 12-16 point gains on LLaVA-Bench are large enough that even with some inflation the method might work, but the paper doesn't demonstrate that with held-out selection.\n\nMinor issues: Table 3 has a row where FastV scores are printed as 10.70, 10.20, etc. (clearly a typo), and reference [2] is cited for VDD instead of [66].\n\nWho should read this: researchers working on LVLM hallucination, contrastive decoding, or multimodal attention. The idea is worth a careful look, but the reported numbers shouldn't be trusted until the mask equations are corrected and the hyperparameter selection is redone on a hold-out set. I'd accept it for peer review with a strong request for revision.","headline":"A promising new combination of dual-attention and a trained soft null prompt for LVLM hallucination, but the printed attention mask is wrong and the hyperparameters are tuned on the test benchmarks, so the headline numbers cannot be taken at face value.","tokens_in":22537,"tokens_out":4928,"would_cite":false,"duration_ms":41711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LACING claims that language bias in large vision-language models can be reduced by a dual-attention mechanism plus a learned soft-image prompt, cutting object hallucination rates by about 40% and raising LLaVA-Bench scores by 12–16 points…","keywords":["large vision-language models","language bias","hallucination reduction","multimodal dual-attention","soft-image guidance","contrastive decoding","visual comprehension"],"falsifier":"Replace the trained soft visual prompt with a randomly initialized embedding of the same shape at inference and re-run Object-HalBench and LLaVA-Bench; the paper's mechanism predicts the learned prompt is doing the work, so the hallucination reduction should largely disappear, whereas if random embeddings preserve the gains, the reported improvement comes from the contrastive decoding formula itself rather than from Soft-Image Guidance.","tokens_in":21611,"feed_emoji":"🖼️","tokens_out":7070,"duration_ms":66658,"temperature":0.7,"pith_summary":"LACING argues that large vision-language models hallucinate and under-read images largely because their text-pretrained backbone dominates the smaller multimodal training stage, and because autoregressive text decoding leans on adjacent words more than on distant visual tokens. To counter the first cause, it inserts a multimodal dual-attention mechanism that computes separate attention weights for visual and text tokens and fuses them, so visual tokens get real attention in every layer while text keeps its causal mask. To counter the second cause, it trains a learnable soft visual prompt that, at inference, replaces the image to form a multimodal-null input; the model's output on this null input is treated as the text-only prior and contrasted with the normal output in a Bayesian decoding recipe. On LLaVA-1.5 backbones, the method reports a 40.36% relative drop in response-level object hallucination and a 12.11-point absolute gain on LLaVA-Bench for the 7B model, with analogous gains for 13B. If the result holds, it matters because it promises hallucination reduction and better visual comprehension without additional data, extra training stages, or RLHF-style feedback.","feed_headline":"Dual attention plus a soft-image prompt cuts LVLM hallucinations ~40%","feed_subtitle":"A two-part fix—visual attention in every layer plus a contrastive null prompt—raises LLaVA-Bench scores by up to 16 points.","key_machinery":"Two mechanisms carry the argument. Multimodal Dual-Attention (MDA) computes two masked attention maps in parallel—a visual map from $M_I$ and a text map from $M_T$—and adds them before the value projection, creating a dedicated visual attention channel that survives into deeper layers. Soft-Image Guidance (SIG) introduces a learnable soft visual prompt $\\epsilon$ of the same shape as the visual embedding; during training it replaces the real image with probability $\\theta$, and at inference the logits from this multimodal-null input are combined with normal logits via $\\ell_g = \\ell_u + (\\ell_c-\\ell_u)\\lambda$ to up-weight visual evidence. These are the mechanisms whose removal (ablation) or replacement (FastV pruning) is used to attribute the benchmark gains.","core_discovery":"The paper's central claim is that language bias in LVLMs has two separable causes and that each has a targeted fix: MDA counters training-scale dominance by computing separate attention weights for visual and text tokens and fusing them as $A=(W_I+W_T)V$, so visual tokens receive meaningful attention in every layer while text tokens keep causal attention; SIG counters short-term text dependency by training a learnable soft visual prompt $\\epsilon$ that replaces the image with probability $\\theta$, then at inference decoding from $\\ell_g = \\ell_u + (\\ell_c-\\ell_u)\\lambda$, the pragmatic form of the Bayes-derived adjustment $\\log \\tilde{p}(y_t|v) \\propto \\lambda(\\log p(y_t|v)-\\log p(y_t))+\\log p(y_t)$. On the LLaVA-1.5 backbone this combination is reported to reduce response-level object hallucination by 40.36% (7B) and 42.18% (13B) and to raise LLaVA-Bench by 12.11 and 16.28 absolute points, all without additional training data or RLHF-style feedback.","pith_inferences":["A direct check the paper does not report is whether the soft prompt transfers across model sizes or model families: transferring the 7B prompt to the 13B model would test whether the prompt encodes generic 'ignore the image' statistics or model-specific text priors.","The decoding formula is essentially an interpolation between the null-input distribution and the normal distribution; an untested extension would be to apply the same contrast with an image caption or a text description as the null input, which would measure how much of the gain is specific to the learned prompt rather than to any stable reference distribution.","Because several headline benchmarks are scored by GPT-4, part of the reported improvement may reflect what GPT-4 prefers in an answer; the paper's 60-item human evaluation is reassuring but small, so a larger human study would be needed to confirm the gains are content-level rather than stylistic.","The method's assumption about where visual tokens sit in the input sequence suggests it may generalize naturally to interleaved image-text inputs, but the paper only demonstrates it on the standard image-first setup."],"forward_implications":["Because LACING reuses the LLaVA-1.5 training recipe and adds only about 0.03% parameters, the two mechanisms could be folded into any existing visual-instruction-tuning pipeline without new data or an RLHF stage, making hallucination reduction a byproduct of standard alignment.","If the reported Object-HalBench and LLaVA-Bench gains are real, a large share of LVLM hallucination is caused by the text-pretrained backbone's dominance and by adjacent-token decoding bias rather than by a lack of vision capability, which reframes hallucination as a distributional artifact rather than an architectural limit.","The SIG contrastive formula only needs the null-input logits, so it could be applied at decoding time to already-trained models that never saw the soft prompt, although the paper's consistency argument predicts it will work better when the prompt was trained in.","The FastV-pruning result—a large performance drop when deep visual tokens are pruned after MDA training—implies that after such training, deep visual tokens carry information the model actually uses, in contrast to the standard picture where most deep-layer visual tokens are redundant.","The reported gains under both greedy and nucleus sampling, unlike the comparison methods VCD and VDD, suggest the method would also work in deterministic deployment settings where sampling noise cannot mask the bias."],"supporting_citations":[{"why":"Defines the LLaVA-1.5 baseline whose training data, hyperparameters, and architecture LACING reuses; all reported gains are deltas against it.","marker":"[30]"},{"why":"Provides the visual contrastive decoding baseline that SIG is compared against and whose noise-injection approach SIG claims to improve on.","marker":"[22]"},{"why":"Supplies the competing text-only contrast strategy and the observation that LVLMs shift attention away from visual tokens as generation progresses.","marker":"[66]"},{"why":"Contributes the attention-pruning analysis (FastV) used to show that deeper layers ignore visual tokens and, in LACING, that MDA makes them load-bearing.","marker":"[6]"},{"why":"Defines the LLaVA architecture, the LLaVA-Bench evaluation, and the visual-instruction-tuning lineage the method builds on.","marker":"[29]"},{"why":"Supplies the Object HalBench task and COCO-label comparison used for the headline hallucination reductions.","marker":"[48]"},{"why":"Provides the evaluation setup and RLHF-based baseline that LACING follows and outperforms without reinforcement learning.","marker":"[63]"},{"why":"Named the language-bias phenomenon in LVLMs that this paper sets out to reduce.","marker":"[67]"}],"fun_headline_variants":["Cut LVLM hallucinations 40% with dual attention and soft prompts","LACING: dual attention + soft-image prompt debias LVLMs","Soft-image prompt + dual attention cut LVLM hallucinations by 40%","Debias LVLMs: dual attention plus soft prompt cuts hallucinations 40%","Two-pronged fix: dual attention and soft prompt slash LVLM hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the text-attention mask written in Eq. (1) is a typo: as printed it lets each text token attend to tokens that come after it, which would leak future text during training and invalidate the causal-attention claim on which the whole method rests; the experiments must implicitly use the reversed inequality with attention only to earlier tokens.","fun_headline_variants_meta":{"raw":{"variants":["Cut LVLM hallucinations 40% with dual attention and soft prompts","LACING: dual attention + soft-image prompt debias LVLMs","Soft-image prompt + dual attention cut LVLM hallucinations by 40%","Debias LVLMs: dual attention plus soft prompt cuts hallucinations 40%","Two-pronged fix: dual attention and soft prompt slash LVLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4319,"prompt_tokens":1048,"completion_tokens":3271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":3172}},"tokens_in":664,"tokens_out":3271,"duration_ms":20389,"temperature":1.0,"reasoning_tokens":3172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:20:29.830958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the trained soft visual prompt with a randomly initialized embedding of the same shape at inference and re-run Object-HalBench and LLaVA-Bench; the paper's mechanism predicts the learned prompt is doing the work, so the hallucination reduction should largely disappear, whereas if random embeddings preserve the gains, the reported improvement comes from the contrastive decoding formula itself rather than from Soft-Image Guidance.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the LLaVA-1.5 baseline whose training data, hyperparameters, and architecture LACING reuses; all reported gains are deltas against it."},{"cited_title":"Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the visual contrastive decoding baseline that SIG is compared against and whose noise-injection approach SIG claims to improve on."},{"cited_title":"Debi- asing multimodal large language models, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the competing text-only contrast strategy and the observation that LVLMs shift attention away from visual tokens as generation progresses."},{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024","cited_arxiv_id":null,"evidence_quote":"Contributes the attention-pruning analysis (FastV) used to show that deeper layers ignore visual tokens and, in LACING, that MDA makes them load-bearing."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Defines the LLaVA architecture, the LLaVA-Bench evaluation, and the visual-instruction-tuning lineage the method builds on."},{"cited_title":"Object hallucination in image cap- tioning, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the Object HalBench task and COCO-label comparison used for the headline hallucination reductions."},{"cited_title":"Rlhf-v: Towards trust- worthy mllms via behavior alignment from fine-grained cor- rectional human feedback, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the evaluation setup and RLHF-based baseline that LACING follows and outperforms without reinforcement learning."},{"cited_title":"Mmicl: Empowering vision-language model with multi-modal in-context learning, 2024","cited_arxiv_id":null,"evidence_quote":"Named the language-bias phenomenon in LVLMs that this paper sets out to reduce."}],"review_version":1}