{"id":"b91803b9-fc04-45c1-8eee-dd3ba5094eaa","arxiv_id":"2502.01419","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SPARC selectively and progressively reinforces attention to relevant image tokens during decoding, improving both precision and recall in detailed image captioning compared to baselines and prior hallucination-mitigation methods.","lead":"SPARC is a training-free method that selectively amplifies visual attention during image caption generation in multimodal AI models, improving both the accuracy and the completeness of detailed descriptions. It could make AI captioning more reliable for screen readers, accessibility tools, and large-scale data labeling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.52-point recall gain that distinguishes SPARC from precision-only methods is reported without variance or significance testing; if within noise, the central 'improves both precision and recall' claim is unsupported.","rationale":"The reader's weakest_assumption targets the causal validity of the mid-layer attention signal used for token selection. That is a legitimate mechanistic concern, and Appendix B.1's saliency comparison plus Table 5 provide only indirect support. However, the most load-bearing issue for the stated central claim is empirical: if the 0.52 pp recall improvement is not statistically separable from zero, SPARC has not been shown to escape the precision/recall tradeoff that the paper uses to motivate its novelty. The reader's rationale does note the small recall gain and absent error bars, so there is partial agreement, but the reader did not make this the primary attack. I leave the verdict unchanged because the appropriate response is the same as the reader's: require the authors to provide significance evidence before accepting the broad claim. If a paired bootstrap reveals a null recall effect, the verdict should move to REJECT for the central claim as stated; if the effect is confirmed, the empirical claim becomes credible, though the mechanism concern would remain a secondary point.","tokens_in":25515,"tokens_out":7853,"duration_ms":79045,"concrete_test":"Using the provided code and the same 500 MS-COCO images, rerun baseline and SPARC to obtain per-image object-level CHAIR recall for both methods. Perform a paired bootstrap (or a McNemar-style test on images where the two methods disagree in recall) and report the 95% confidence interval for the mean recall difference. If the interval includes 0, the 'improves recall' portion of the central claim is not established and should be stated only as a precision/F1 effect pending larger samples; if it excludes 0, the concern is resolved. The five repeats reported in the paper should also be disclosed per run to enable a paired significance test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2/Table 2 provides the only quantitative support for the central claim that SPARC 'successfully improves both precision and recall' and is 'the only one that improves recall compared to the baseline.' The recorded recall increase is 79.46->79.98 (+0.52 pp), while precision rises by 3.02 pp. Table 13 says these CHAIR values are averages over five random 500-instance samples, yet no per-run values, standard deviations, confidence intervals, or significance tests are reported. For an object-level proportion near 0.80 with roughly 2,500-3,500 objects across 500 COCO images, the standard error of a proportion is on the order of 0.7-0.9 pp, so a 0.52 pp difference is plausibly within sampling noise; a paired per-image analysis could change the error, but the paper gives no basis for evaluating this. If the true recall effect is zero, the strongest claim in the abstract and Section 5.2—that SPARC uniquely avoids the precision/recall tradeoff—has no support, and the method reduces to a precision/F1 improvement. This is load-bearing because the entire contrast with PAI, VOLCANO, and VCD depends on the recall result being real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPARC, a training-free decoding-time method that selectively amplifies visual token attention in multimodal LLMs to improve detailed image captioning. The method selects relevant visual tokens using a Relative Activation Score computed from attention changes across decoding steps, and progressively reinforces their contribution via a cumulative selection count. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2-VL report improved CLAIR scores and a higher CHAIR F1 score, with the paper claiming SPARC is unique in improving both precision and recall.","tokens_in":25748,"tokens_out":7357,"duration_ms":58678,"significance":"If the reported results are reliable, SPARC is a simple, efficient, and practical contribution to hallucination mitigation in image captioning: it is training-free, adds only about 3% decoding overhead (Table 6), and comes with released code. The paper also provides useful ablations (Appendix C) and an analysis of attention dynamics. However, the central differentiator—the recall improvement—is not yet statistically supported, and the abstract overstates the precision/recall trade-off of prior methods.","major_comments":[{"comment":"The central claim that SPARC uniquely improves recall rests on a 0.52 percentage-point increase (79.46 to 79.98) that is reported without any variance measure or significance test. The paper states these values are averages over five random 500-instance samples, but no per-run values, standard deviations, or confidence intervals are given. Given the standard error of a proportion near 0.8 with several thousand objects, this difference is plausibly within sampling noise. Please provide per-run results and a paired statistical test (e.g., bootstrap over images or Wilcoxon signed-rank) to establish that the recall gain is not noise.","section":"§5.2 / Table 2 / Table 13"},{"comment":"The abstract and introduction claim that existing methods improve precision at the cost of recall, but Table 2 shows OPERA and VCD have lower precision than the baseline (84.54 and 83.22 vs. 84.70) as well as lower recall (78.82 and 77.50 vs. 79.46). The paper's own text in §5.2 acknowledges this ('OPERA and VCD ... fail to improve precision or recall'). This overstatement should be corrected.","section":"Abstract / Table 2"},{"comment":"The per-model hyperparameters (τ and layer l) are selected based on performance on the same evaluation sets (IIW-400, DOCCI, and MS COCO) used for the final results; Tables 8–10 in Appendix C show that these parameters are swept on these datasets to pick the reported configuration. This selection procedure risks overfitting to the benchmark. To support generalization, the authors should either fix the hyperparameters on a held-out validation split or report results across the swept range to show the main conclusions are insensitive to reasonable choices.","section":"§5.1 / Appendix C"},{"comment":"The token-selection mechanism uses raw attention weights from a single middle layer (layer 20 for LLaVA models, 18 for Qwen2-VL) averaged across heads. The only supporting evidence for this signal is the qualitative gradient-weighted saliency comparison in Figure 12, which concerns the 'noisy attention' observation rather than the token-selection premise. Please provide quantitative evidence that the selected tokens are causally relevant to the generated caption, for example by comparing selection based on raw attention against selection based on saliency-weighted attention on CHAIR/CLAIR.","section":"§4.2 / Appendix B.1"}],"minor_comments":[{"comment":"'We compar' should be 'We compare'; 'VOCANO' should be 'VOLCANO'.","section":"§5.2"},{"comment":"'β∈[0,1] isis the smoothing factor' contains a duplicated 'is'.","section":"Eq. (4)"},{"comment":"The definition of CHAIR precision is inverted: the text says precision is the proportion of objects 'that do not appear in the reference captions'; that is the hallucination rate (CHAIR_i), not precision. Precision is 1 − CHAIR_i, as correctly stated in Appendix F.","section":"§5.2"},{"comment":"'UWith' at the start of a paragraph appears to be a typo for 'With'.","section":"Appendix D"},{"comment":"The caption says 'The best scores are bolded, while the second-best scores are underlined,' but no underlining is visible in the rendered table; please check the formatting.","section":"Table 2 caption"},{"comment":"'we generat captions' should be 'we generate captions'.","section":"§5.1"},{"comment":"The ablation text refers to 'as shown in Figure 7(a)' when describing the scaling strategy for the no-selection setting; this reference is unclear because Figure 7(a) shows attention trends rather than a scaling strategy.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and generally solid engineering, but the statistical support for the headline recall improvement is missing. If the authors can provide variance estimates and a paired significance test, and correct the abstract overstatement, the paper would be a reasonable contribution. The hyperparameter selection issue should be addressed transparently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the mechanism is genuinely new and cheap: SPARC selects visual tokens by an EMA-normalized relative activation score, then scales their value vectors by a cumulative selection count during decoding. That's a sensible way to avoid the blanket-attention-boosting failure mode, and the overhead is about 3 percent. Second, the paper's headline claim--that it's the only method that improves both precision and recall--hangs on a 0.52-point recall gain (79.46 to 79.98) reported without variance, confidence intervals, or significance tests. The stress-test note is right: with roughly 3,000 objects and a standard error around 0.7-0.9 points, that difference is plausibly noise.\n\nWhat the paper does well: it surfaces a real, underappreciated tradeoff--naive attention scaling sacrifices recall--and Section 3's analyses of attention diversity, noise growth, and decay are thoughtful and backed by multiple visualizations. The ablations are unusually thorough: layer, threshold, EMA factor, and scaling factor are each varied across three models. They even check attention against gradient-weighted saliency in the appendix, which partly addresses the concern that raw attention scores are the wrong signal. The method generalizes across LLaVA-1.5, LLaVA-Next, and Qwen2-VL on CLAIR, which is more than many such papers do. Credit earned.\n\nSoft spots, in order of seriousness. First, the recall claim. Table 13 says the CHAIR numbers are averages over five 500-image samples, but no per-run values or error bars are provided. A paired analysis might give a different error, but as reported, 0.52 points is within one standard error. If the true effect is zero, the \"improves both precision and recall\" framing collapses; the method becomes a precision/F1 improvement--still useful, but not what the abstract promises.\n\nSecond, the abstract overstates the baselines: \"existing methods improve the precision... at the cost of recall.\" In Table 2, OPERA and VCD actually reduce precision as well. The body is accurate, but the abstract is not.\n\nThird, hyperparameters--tau per model, layer per model, alpha, beta--are tuned on the evaluation datasets. The ablations show consistent trends, which mitigates this, but it is still selection on the test set and should be described more carefully.\n\nThe underlying assumption, that layer-20 averaged attention reflects causally relevant visual tokens, is only partially validated. The saliency analysis helps, but the causal link between selection and caption quality is not fully closed. That is a minor-to-moderate concern; the method works empirically even if the mechanism story is incomplete.\n\nBottom line: this paper is useful for anyone working on training-free decoding or hallucination mitigation in MLLMs. It deserves a serious referee, but it needs a revision that reports per-run CHAIR values, adds significance tests, and rewrites the abstract. As it stands, I would cite the method with a caveat and would use the paper as a reading-group case study in how a small, underpowered recall difference can carry a headline claim.","headline":"Clever training-free attention recalibration, but the 'improves both precision and recall' claim rests on a 0.52pp recall gain with no error bars; the abstract also overstates the baselines.","tokens_in":26282,"tokens_out":4624,"would_cite":true,"duration_ms":37623,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPARC claims that selectively amplifying visual tokens that show relative attention jumps makes MLLM captions both more precise and more complete, with no retraining and about 3% slowdown.","keywords":["image captioning","multimodal large language models","hallucination mitigation","visual attention","precision-recall tradeoff","training-free decoding","attention recalibration","detailed captioning"],"falsifier":"Take LLaVA-1.5 and rerun SPARC with token selection inverted, choosing the visual tokens with the lowest Relative Activation Scores instead of the highest; if inverted selection still improves CHAIR F1 over the baseline, the selection signal is not carrying the benefit and the mechanism is not what the paper claims.","tokens_in":25289,"feed_emoji":"🖼️","tokens_out":6023,"duration_ms":52499,"temperature":0.7,"pith_summary":"This paper is trying to show that a training-free, decoding-time intervention can fix a specific failure of multimodal large language models: as they produce longer image captions, their attention to image content weakens and becomes noisier, so captions drift toward hallucination and omit details. The proposed method, SPARC, does not amplify all visual attention, a naive strategy that the paper shows collapses recall. Instead, it selects the image tokens whose attention has risen sharply relative to its recent history, and progressively scales up their cached value vectors. The paper reports that SPARC improves both precision and recall on the CHAIR benchmark, raises CLAIR scores on IIW-400 and DOCCI, wins human-preference comparisons, and adds only about 2.8% to per-token generation time. If this holds, it would give a practical way to improve detailed captioning, useful for accessibility and training-data generation, without retraining any model.","feed_headline":"Attention boost makes image captions both sharper and fuller","feed_subtitle":"SPARC amplifies only the visual tokens that matter, keeping recall up while cutting hallucination at near-zero cost.","key_machinery":"The carrying objects are the Relative Activation Score $r^l_{i,j} = (a^l_{i,j} - \\tilde{a}^l_{i-1,j})/\\tilde{a}^l_{i-1,j}$, where $\\tilde{a}$ is an exponential-moving-average-smoothed attention weight, which converts absolute attention magnitude into a change signal robust to scale drift; a Selection Count $c_{i,j}$ that accumulates how often each image token crosses the threshold $\\tau$; and the recalibration step $V^{(l,h)}_j \\leftarrow V^{(l,h)}_j \\cdot \\alpha$ for selected tokens, which leverages cached key-value states to amplify attention to frequently selected tokens at negligible cost. The paper extracts attention from layer 20 in LLaVA models and layer 18 in Qwen2-VL, averages over heads, and applies the value scaling across all layers.","core_discovery":"SPARC claims that detailed image captioning in MLLMs is limited by a specific attention pathology: as generation context lengthens, the proportion of attention allocated to image tokens falls, the distribution over image tokens becomes noisier, and a few static sink tokens absorb increasing weight. The paper shows that naively amplifying all image attention, the strategy behind prior work, freezes the attention pattern, lowers attention diversity, and collapses recall. SPARC instead tracks each image token's attention relative to its own smoothed history, selects tokens whose attention jumps above a threshold, and progressively multiplies their cached value vectors by $\\alpha^{c_{i,j}}$, where $c_{i,j}$ is the number of prior steps in which the token was selected. With this mechanism, the paper reports simultaneous gains in CHAIR precision (+3.02 percentage points) and recall (+0.52 percentage points), higher CLAIR scores on IIW-400 and DOCCI, and human-preference wins over both the baseline and the naive amplification approach.","pith_inferences":["Editorial inference: if the relative activation score is causal rather than correlational, the same selection rule could be ported to any autoregressive multimodal decoder, including video captioning and embodied navigation, wherever attention decay over long horizons appears.","Editorial inference: the paper's attention-diversity analysis implies a cheap diagnostic, caption recall should correlate with the mean pairwise Wasserstein distance of attention patterns, which could predict which models benefit most from SPARC without running full evaluations.","Editorial inference: since SPARC's selection uses only cached attention, it could be combined orthogonally with contrastive decoding or self-refinement to push the precision-recall frontier further; the paper does not test these combinations.","Editorial inference: the results suggest that hallucination benchmarks should report coverage metrics alongside precision, because a method that raises precision while dropping recall can look better on a single number while producing less useful captions."],"forward_implications":["SPARC raises CHAIR F1 from 81.99 to 83.67 on LLaVA-1.5 by improving both precision and recall, whereas every compared method improves precision at the cost of recall or improves neither.","The method transfers to three 7B MLLMs, LLaVA-1.5, LLaVA-NeXT, and Qwen2-VL, with consistent CLAIR gains on IIW-400 and DOCCI.","Its runtime overhead is about +2.8% per token and roughly 40 KB of extra memory, so the intervention could be deployed in production captioning pipelines without retraining.","Human annotators prefer SPARC captions over baseline and naive-amplification captions for both precision and recall, supporting the claim that the trade-off is alleviated rather than shifted.","On the POPE hallucination benchmark, SPARC improves accuracy and instruction-following over baseline and naive attention scaling."],"supporting_citations":[{"why":"Supplies the naive attention-amplification baseline that SPARC compares against and whose recall collapse motivates the method.","marker":"Liu et al., 2025"},{"why":"Defines CHAIR, the metric used to measure hallucination precision and recall.","marker":"Rohrbach et al., 2018"},{"why":"DOCCI dataset provides the detailed image-caption pairs used in attention analyses and CLAIR evaluation.","marker":"Onoe et al., 2025"},{"why":"CLAIR, the LLM-based metric used to score caption quality against references.","marker":"Chan et al., 2023"},{"why":"Supplies the attention-sink identification used to show SPARC avoids amplifying sink tokens.","marker":"Kang et al., 2025"},{"why":"Gradient-weighted saliency analysis that supports the assumption that raw attention reflects semantically relevant image regions.","marker":"Zhang et al., 2025"},{"why":"MS-COCO 2014 validation set provides the object annotations used for CHAIR precision and recall computation.","marker":"Lin et al., 2014"}],"fun_headline_variants":["Selective attention boost sharpens captions and boosts recall","SPARC recalibrates visual attention for sharper, fuller captions","Targeted visual attention improves precision and recall in captions","Boost only key visual tokens to make captions both precise and complete","Attention recalibration lifts caption precision without sacrificing recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that head-averaged attention from a single mid-layer, converted to a relative activation score, identifies the image regions the model actually relies on for the current word; if attention sinks or positional artifacts dominate those scores, SPARC would amplify the wrong tokens and could increase hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Selective attention boost sharpens captions and boosts recall","SPARC recalibrates visual attention for sharper, fuller captions","Targeted visual attention improves precision and recall in captions","Boost only key visual tokens to make captions both precise and complete","Attention recalibration lifts caption precision without sacrificing recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1553,"prompt_tokens":952,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":568,"tokens_out":601,"duration_ms":5842,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:21:40.753485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take LLaVA-1.5 and rerun SPARC with token selection inverted, choosing the visual tokens with the lowest Relative Activation Scores instead of the highest; if inverted selection still improves CHAIR F1 over the baseline, the selection signal is not carrying the benefit and the mechanism is not what the paper claims.","supporting_citations":[{"cited_title":"Paying more attention to image: A training-free method for alleviating hallucination in lvlms","cited_arxiv_id":null,"evidence_quote":"Supplies the naive attention-amplification baseline that SPARC compares against and whose recall collapse motivates the method."},{"cited_title":"Docci: Descriptions of connected and contrasting images","cited_arxiv_id":null,"evidence_quote":"DOCCI dataset provides the detailed image-caption pairs used in attention analyses and CLAIR evaluation."},{"cited_title":"MLLM s know where to look: Training-free perception of small visual details with multimodal LLM s","cited_arxiv_id":null,"evidence_quote":"Gradient-weighted saliency analysis that supports the assumption that raw attention reflects semantically relevant image regions."}],"review_version":1}