{"id":"b1663ca5-cd48-42e1-914c-18cc9afb1c9e","arxiv_id":"2501.12206","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PAINT reduces hallucination in LLaVA-1.5 by selectively amplifying attention to ViT-defined local and summary tokens, but the gains are selected on the test set and lack error bars.","lead":"PAINT proposes a training-free way to cut object hallucination in image-captioning models by boosting attention to two specially chosen groups of visual tokens. The reported 62% hallucination drop is undermined because the boost strengths were picked on the same test examples used to measure the improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported hallucination reductions are selected on the same 500-image test sample, so the headline 61.9% and 71.0% improvements are in-sample best-case numbers, not validated out-of-sample performance.","rationale":"The reader's overall REJECT verdict is sound, but their stated weakest assumption is about the ViT-to-LLM token mapping and layer selection. My reading identifies a more direct load-bearing problem: the evaluation protocol itself is in-sample. Tables 2-4 tune the method's only three hyperparameters on the same 500-instance validation sample that later appears in Table 1 as the headline result. Because the optimum is extremely sharp and the output length changes dramatically across configurations, the reported 61.9% and 71.0% reductions are consistent with selecting the lucky grid point rather than with a robust property of the method. This does not refute the underlying idea; it means the paper as written does not provide out-of-sample evidence for the central claim. I therefore keep the reader's REJECT verdict rather than moving to CONDITIONAL, because the requested holdout experiment is needed before the claim can be evaluated. Secondary concerns - missing layer indices for Eq. 3, the unexplained 'head-specific modulation' in Section 4.3, and the absence of the promised Recall metric in Table 1 - are real but less decisive; they would matter even after a clean holdout evaluation is supplied. The concrete test above is the minimal check that would settle whether the concern actually lands.","tokens_in":8813,"tokens_out":6412,"duration_ms":66317,"concrete_test":"Re-run the full pipeline with a strict split: select alpha, beta, and N on a development set (e.g., 300 of the 500 instances or a fresh MSCOCO sample), freeze those hyperparameters, then evaluate once on a held-out set of at least 200 unseen instances, reporting CHAIRS, CHAIRI, F1, and output length with bootstrap confidence intervals. If the gap over PAI shrinks to near zero or flips, the reported reductions are selection artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: PAINT reduces hallucination by 61.9% at the sentence level and 71.0% at the instance level while maintaining F1. For this claim to hold, the reported numbers must be honest estimates at a fixed, prespecified configuration. The paper does not establish this. Section 4.2 samples 500 validation instances from MSCOCO, and Sections 4.4.1-4.4.3 use this same sample to select alpha=0.7, beta=0.4, and N=25% by directly inspecting CHAIRS, CHAIRI, and F1. Table 1 then reports exactly those selected values. No held-out split, cross-validation, or uncertainty estimate is described, so the headline numbers are the result of a grid search on the evaluation set. The knife-edge sensitivity makes this especially serious: in Table 2, alpha=0.8 reduces CHAIRS to 4.6 but drops F1 to 51.1 and output length to 41.1; alpha=0.9 collapses output length to 10, making the low CHAIRS vacuous. The chosen operating point also trades away accuracy, with F1 dropping from 75.9 to 71.8, conflicting with the abstract's claim of maintaining accuracy. As reported, the experiments cannot distinguish a genuine mechanism from an in-sample cherry-pick.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAINT, a training-free method that reduces hallucination in large vision-language models by selectively amplifying self-attention to two groups of visual tokens, called local tokens and summary tokens, which are identified from CLS-to-patch attention maps in a Vision Transformer. The method is evaluated on 500 sampled MSCOCO instances using CHAIR and POPE metrics, reporting large reductions in hallucination (e.g., 61.9% sentence-level and 71.0% instance-level) relative to the PAI baseline. The central claim is that selective boosting with learned margins alpha and beta outperforms uniform attention boosting while preserving accuracy.","tokens_in":9086,"tokens_out":4545,"duration_ms":42078,"significance":"If the reported results were validated on out-of-sample data, PAINT would be a valuable contribution because it is a training-free, plug-and-play intervention with a conceptually interesting token taxonomy (local vs. summary). The paper includes a public code repository, which aids reproducibility. However, the current evaluation does not establish the central claim: the headline metrics are computed on the same 500-example sample used to select alpha, beta, and N, making them in-sample fits. The method also exhibits extreme sensitivity to hyperparameters, and the manuscript omits crucial implementation details about which layers and heads are modified.","major_comments":[{"comment":"The 500-instance sample described in Section 4.2 is used both for selecting the hyperparameters alpha, beta, and N (Tables 2, 3, 4) and for computing the final results in Table 1. Because the headline reductions of 61.9% and 71.0% are obtained on the same examples that determine the operating point, they are in-sample fits rather than honest estimates of out-of-sample performance. A held-out test set, cross-validation, or a separate benchmark with hyperparameters fixed before evaluation is required to support the central claim.","section":"Sections 4.2 and 4.4"},{"comment":"The chosen operating point alpha=0.7 is not robustly separated from degenerate behavior: alpha=0.8 already reduces F1 to 51.1 and output length to 41.1, and alpha=0.9 collapses F1 to 8.7 with output length 10, making the low CHAIR values vacuous. This knife-edge sensitivity means the reported improvement at alpha=0.7 could be an artifact of selecting an extreme threshold without proper validation.","section":"Section 4.4.1, Table 2"},{"comment":"The method is underspecified: Figure 3 labels the modified attention layer as 'Layer 2', but Equations (1)-(3) and Section 3.3 do not state which token positions or which LLM layers are modified, and Section 4.3 introduces 'head-specific modulation' without defining it in Section 3. This prevents reproduction and leaves the central mechanism ambiguous, especially regarding the transferability of the layer choice across architectures.","section":"Section 3.3 and Figure 3"},{"comment":"The abstract claims the method reduces hallucination 'while maintaining accuracy', but Table 1 shows F1 decreasing from 75.9 to 71.8, which is a meaningful drop. Furthermore, the Recall metric defined in Equation (6) is never reported in any table. The paper should report Recall, error bars from multiple runs, and a length-controlled analysis to rule out the possibility that reduced hallucination is simply due to shorter or different-length generated captions.","section":"Section 4.3 and Equation (6)"}],"minor_comments":[{"comment":"The abstract reports an improvement of 'up to 62.3%' while Section 5 reports 61.9% at the sentence level and 71.0% at the instance level; please reconcile these numbers and clarify the comparison baseline.","section":"Abstract and Section 5"},{"comment":"The paper states that PAINT can be extended to InstructBLIP, LLaMA-Adapter-v2, MiniGPT4, and Shikra, but no experiments on those models are reported; please provide at least one cross-architecture validation or temper the claim.","section":"Section 4.1"},{"comment":"The attention heatmaps in Figures 2 and 3 are difficult to read; please use larger fonts, a perceptually uniform colormap, and clear annotations.","section":"Figures 2 and 3"},{"comment":"References [14] and [15] appear to be the same paper (OPERA) published in two venues; please consolidate them to avoid duplicate citations.","section":"References"},{"comment":"POPE is introduced as a benchmark in Section 4.2, but no POPE results are reported in any table; please either include the results or remove the mention.","section":"Section 4.2"},{"comment":"There are minor language issues, such as 'phenomena' in Section 1 and 'demonstrate' in the abstract, which should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader that the in-sample hyperparameter selection is a serious flaw that undermines the central empirical claim. However, I consider this addressable through a proper held-out evaluation and a more thorough sensitivity analysis, rather than an irreparable error in the approach itself. The authors should also resolve the missing layer/head specification and the length confound before the paper can be considered for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you on arXiv:2501.12206. The headline claim—PAINT cuts hallucination by ~62% with modest accuracy loss—doesn't survive contact with the experimental setup. The trio of free parameters (alpha, beta, N) is tuned on the same 500-image validation sample that produces the final CHAIR/F1 numbers, so the reported improvements are in-sample best-case fits, not out-of-sample estimates. The ablation tables actually make this explicit: alpha=0.7, beta=0.4, N=25% are the grid-search winners on the test set, and neighboring values (alpha=0.8, N=30%) degrade F1 substantially or break generation. So the stress-test note is on target, and the reader's reject verdict is reasonable.\n\nThat said, there's a real idea here. The paper observes that ViT attention from the CLS token distinguishes 'local' tokens (early layers) from 'summary' tokens (deep layers), and argues that the LLM should boost these token groups with different margins rather than uniformly as PAI does. That specific move—differentiated margins per token type—is not in the cited PAI work. The CLS-to-patch taxonomy itself comes from Darcet et al.'s registers and the authors' own HIRED, but the application to hallucination mitigation with type-specific boosts is new. The analysis of attention decay in Figure 1 is plausible.\n\nThe weaknesses beyond the evaluation issue are real but secondary. The method section never states which LLM layers get modified (Figure 3 says 'Layer 2' but Eq. 3 and Section 3.3 are layer-agnostic), and 'head-specific modulation' appears in Section 4.3 without a definition. The abstract's 'maintaining accuracy' is contradicted by their own table: F1 drops from 75.9 to 71.8. And the CHAIR reductions at alpha=0.9 are vacuous because output length collapses to 10 tokens—a degenerate response. The paper would need a clean held-out split, reported error bars, and a fixed (not grid-searched) configuration to support its claims.\n\nWho gets value from this? Researchers working on training-free hallucination mitigation might find the local/summary token distinction worth probing, and the code release helps. But as it stands, it's a hypothesis sketch with an overclaimed evaluation. I'd send it to peer review only if the authors are willing to redo the evaluation properly—the core idea is testable and not silly. For now, don't build anything on the reported numbers.","headline":"In-sample tuning of alpha/beta/N on the same 500-image test set makes the headline hallucination reductions unvalidated, though the local-vs-summary token idea is worth a proper test.","tokens_in":9637,"tokens_out":3132,"would_cite":false,"duration_ms":26188,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selective boosting of local and summary visual tokens cuts LVLM hallucination by up to 62.3%.","keywords":["hallucination mitigation","large vision-language models","self-attention intervention","visual tokens","CLS-to-patch attention","training-free","LLaVA-1.5","image captioning"],"falsifier":"Run PAINT on a second LVLM family, such as InstructBLIP or LLaMA-Adapter-v2, using the same MSCOCO subset and the same Layer-2 intervention; if CHAIRS and CHAIRI improvements vanish or require a different layer, the ViT-to-LLM token mapping is architecture-specific. A second check is to compare the selected local token patches against ground-truth object masks; low overlap would show the selector is not actually grounded.","tokens_in":8588,"feed_emoji":"🎯","tokens_out":7438,"duration_ms":69063,"temperature":0.7,"pith_summary":"PAINT claims that hallucination in large vision-language models tracks the weakening of attention to visual tokens as generation moves through deeper LLM layers, and that the right fix is selective rather than uniform reinforcement. The paper identifies two roles among ViT-produced visual tokens: local tokens, selected from early CLS-to-patch attention, carry grounded object details; summary tokens, selected from late layers, carry global scene semantics. Boosting these two groups with different attention margins in the LLM's self-attention lowers sentence-level hallucination on the MSCOCO validation set from 46.2% to 17.6% and instance-level hallucination from 13.8% to 4.0%, with a modest F1 drop from 75.9% to 71.8%. If the claim holds, PAINT is a plug-and-play, training-free intervention that improves faithfulness for existing LVLM checkpoints.","feed_headline":"Selective attention boost cuts image-model hallucinations 62%","feed_subtitle":"Boosting only the informed visual tokens, not all of them, beats uniform attention on LLaVA-1.5.","key_machinery":"The load-bearing mechanism is the CLS-to-patch attention map inside the Vision Transformer, read at two depths. Eq. (1) averages attention over heads in layer 1 and takes the top N% positions as local tokens; Eq. (2) does the same in layer 24 and takes the top N% as summary tokens. Eq. (3) then amplifies exactly those positions in the LLM's self-attention output, with separate margins α and β, at the intervention layer shown as Layer 2 in Figure 3. This mechanism converts an observed attention imbalance into a targeted re-weighting that requires no gradient updates or additional training.","core_discovery":"The paper's central discovery is that not all visual tokens deserve the same attention boost during decoding. Using the ViT's CLS-to-patch attention maps, PAINT selects the top N% of tokens in the first ViT layer as local tokens and the top N% in the last layer as summary tokens, then modifies LLM self-attention with Eq. (3): local tokens receive an added margin α·|Ā| and summary tokens receive β·|Ā|. The paper reports the best balance at α=0.7, β=0.4, N=25%, where CHAIRS drops from 46.2% to 17.6% and CHAIRI from 13.8% to 4.0%. This is the claim that selective boosting of informed tokens outperforms uniform visual-token boosting and does so while keeping most task accuracy.","pith_inferences":["Inference: The two-margin design suggests a general recipe: partition visual tokens by their role in the ViT and give each role its own attention boost, a recipe that could extend to video or high-resolution inputs where local and summary roles are more pronounced.","Inference: One testable extension is to set α and β from the measured slope of attention decay across LLM layers, making the method adaptive per image rather than dependent on fixed hyperparameters.","Inference: If the mechanism generalizes, the method could combine with logit-level calibration, since attention re-weighting and logit refinement target different failure modes."],"forward_implications":["The intervention is training-free: it changes only attention weights at inference time, so any existing LLaVA-1.5-style checkpoint can adopt it without retuning.","Because local and summary tokens need different margins, the paper's results imply that treating all visual tokens uniformly leaves hallucination reduction on the table.","The paper reports that adding spatial token selection can push CHAIRS down to 15.4%, showing further headroom beyond the main configuration.","The method is designed to transfer to other LVLMs with a ViT-plus-LLM architecture, including InstructBLIP, LLaMA-Adapter-v2, MiniGPT-4, and Shikra."],"supporting_citations":[{"why":"Supplies the uniform visual-token boosting method (PAI) that PAINT extends and compares against.","marker":"[28]"},{"why":"Defines the Vision Transformer and provides the CLS-to-patch attention maps used in Eqs. (1) and (2).","marker":"[8]"},{"why":"Introduces the idea that deep ViT layers store global information in background tokens, informing the summary-token concept.","marker":"[7]"},{"why":"Cited as prior work on attention-guided visual token analysis that inspires the local and summary token distinction.","marker":"[2]"},{"why":"Defines the CHAIR metric, including CHAIRS and CHAIRI, used for all hallucination comparisons.","marker":"[33]"},{"why":"Defines the POPE benchmark and binary object-presence evaluation used for F1 and accuracy measurements.","marker":"[22]"},{"why":"Provides LLaVA-1.5, the base vision-language model on which all PAINT experiments are run.","marker":"[25]"},{"why":"Provides the MSCOCO 2014 dataset from which the 500-instance validation subset is sampled.","marker":"[24]"}],"fun_headline_variants":["Selective visual token attention cuts LVLM hallucinations 62%","Boosting only informed tokens reduces hallucinations by 62%","PAINT: target local and summary tokens to slash hallucination","Not all visual tokens equal: selective boost cuts hallucination 62%","Informed token attention beats uniform boost in LVLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that CLS-to-patch attention in the ViT reliably identifies which visual tokens carry grounded object information, and that boosting exactly those positions in one fixed LLM layer is what removes hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Selective visual token attention cuts LVLM hallucinations 62%","Boosting only informed tokens reduces hallucinations by 62%","PAINT: target local and summary tokens to slash hallucination","Not all visual tokens equal: selective boost cuts hallucination 62%","Informed token attention beats uniform boost in LVLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1404,"prompt_tokens":1036,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":652,"tokens_out":368,"duration_ms":3909,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:23:40.623781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PAINT on a second LVLM family, such as InstructBLIP or LLaMA-Adapter-v2, using the same MSCOCO subset and the same Layer-2 intervention; if CHAIRS and CHAIRI improvements vanish or require a different layer, the ViT-to-LLM token mapping is architecture-specific. A second check is to compare the selected local token patches against ground-truth object masks; low overlap would show the selector is not actually grounded.","supporting_citations":[{"cited_title":"Paying more atten- tion to image: A training-free method for alleviating halluci- nation in lvlms, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the uniform visual-token boosting method (PAI) that PAINT extends and compares against."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale, 2021","cited_arxiv_id":null,"evidence_quote":"Defines the Vision Transformer and provides the CLS-to-patch attention maps used in Eqs. (1) and (2)."},{"cited_title":"Vision transformers need registers, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the idea that deep ViT layers store global information in background tokens, informing the summary-token concept."},{"cited_title":"Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji","cited_arxiv_id":null,"evidence_quote":"Cited as prior work on attention-guided visual token analysis that inspires the local and summary token distinction."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides LLaVA-1.5, the base vision-language model on which all PAINT experiments are run."}],"review_version":1}