Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SPARC claims that selectively amplifying visual tokens that show relative attention jumps makes MLLM captions both more precise and more complete, with no retraining and about 3% slowdown.

desk verdict Clever training-free attention recalibration, but the 'improves both precision and recall' claim rests on a 0.52pp recall gain with no error bars; the abstract also overstates the baselines. read the letter →

arxiv 2502.01419 v2 pith:WN4NYAUM submitted 2025-02-03 cs.CV cs.AI

Mingi Jung , Saehyung Lee , Eunji Kim , Sungroh Yoon This is my paper · ORCID
classification cs.CVcs.AI
keywords imagecaptioningmultimodallargelanguagemodelshallucinationmitigationvisualattentionprecision-recalltradeofftraining-freedecodingrecalibrationdetailed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that a training-free, decoding-time intervention can fix a specific failure of multimodal large language models: as they produce longer image captions, their attention to image content weakens and becomes noisier, so captions drift toward hallucination and omit details. The proposed method, SPARC, does not amplify all visual attention, a naive strategy that the paper shows collapses recall. Instead, it selects the image tokens whose attention has risen sharply relative to its recent history, and progressively scales up their cached value vectors. The paper reports that SPARC improves both precision and recall on the CHAIR benchmark, raises CLAIR scores on IIW-400 and DOCCI, wins human-preference comparisons, and adds only about 2.8% to per-token generation time. If this holds, it would give a practical way to improve detailed captioning, useful for accessibility and training-data generation, without retraining any model.

What carries the argument

The carrying objects are the Relative Activation Score $r^l_{i,j} = (a^l_{i,j} - \tilde{a}^l_{i-1,j})/\tilde{a}^l_{i-1,j}$, where $\tilde{a}$ is an exponential-moving-average-smoothed attention weight, which converts absolute attention magnitude into a change signal robust to scale drift; a Selection Count $c_{i,j}$ that accumulates how often each image token crosses the threshold $\tau$; and the recalibration step $V^{(l,h)}_j \leftarrow V^{(l,h)}_j \cdot \alpha$ for selected tokens, which leverages cached key-value states to amplify attention to frequently selected tokens at negligible cost. The paper extracts attention from layer 20 in LLaVA models and layer 18 in Qwen2-VL, averages over heads, and applies the value scaling across all layers.

What would settle it

Take LLaVA-1.5 and rerun SPARC with token selection inverted, choosing the visual tokens with the lowest Relative Activation Scores instead of the highest; if inverted selection still improves CHAIR F1 over the baseline, the selection signal is not carrying the benefit and the mechanism is not what the paper claims.

Watch

Extended reading notes

Core claim

SPARC claims that detailed image captioning in MLLMs is limited by a specific attention pathology: as generation context lengthens, the proportion of attention allocated to image tokens falls, the distribution over image tokens becomes noisier, and a few static sink tokens absorb increasing weight. The paper shows that naively amplifying all image attention, the strategy behind prior work, freezes the attention pattern, lowers attention diversity, and collapses recall. SPARC instead tracks each image token's attention relative to its own smoothed history, selects tokens whose attention jumps above a threshold, and progressively multiplies their cached value vectors by $\alpha^{c_{i,j}}$, where $c_{i,j}$ is the number of prior steps in which the token was selected. With this mechanism, the paper reports simultaneous gains in CHAIR precision (+3.02 percentage points) and recall (+0.52 percentage points), higher CLAIR scores on IIW-400 and DOCCI, and human-preference wins over both the baseline and the naive amplification approach.

Load-bearing premise

The method assumes that head-averaged attention from a single mid-layer, converted to a relative activation score, identifies the image regions the model actually relies on for the current word; if attention sinks or positional artifacts dominate those scores, SPARC would amplify the wrong tokens and could increase hallucination.

Editorial extensions

If this is right

  • SPARC raises CHAIR F1 from 81.99 to 83.67 on LLaVA-1.5 by improving both precision and recall, whereas every compared method improves precision at the cost of recall or improves neither.
  • The method transfers to three 7B MLLMs, LLaVA-1.5, LLaVA-NeXT, and Qwen2-VL, with consistent CLAIR gains on IIW-400 and DOCCI.
  • Its runtime overhead is about +2.8% per token and roughly 40 KB of extra memory, so the intervention could be deployed in production captioning pipelines without retraining.
  • Human annotators prefer SPARC captions over baseline and naive-amplification captions for both precision and recall, supporting the claim that the trade-off is alleviated rather than shifted.
  • On the POPE hallucination benchmark, SPARC improves accuracy and instruction-following over baseline and naive attention scaling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the relative activation score is causal rather than correlational, the same selection rule could be ported to any autoregressive multimodal decoder, including video captioning and embodied navigation, wherever attention decay over long horizons appears.
  • Editorial inference: the paper's attention-diversity analysis implies a cheap diagnostic, caption recall should correlate with the mean pairwise Wasserstein distance of attention patterns, which could predict which models benefit most from SPARC without running full evaluations.
  • Editorial inference: since SPARC's selection uses only cached attention, it could be combined orthogonally with contrastive decoding or self-refinement to push the precision-recall frontier further; the paper does not test these combinations.
  • Editorial inference: the results suggest that hallucination benchmarks should report coverage metrics alongside precision, because a method that raises precision while dropping recall can look better on a single number while producing less useful captions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SPARC, a training-free decoding-time method that selectively amplifies visual token attention in multimodal LLMs to improve detailed image captioning. The method selects relevant visual tokens using a Relative Activation Score computed from attention changes across decoding steps, and progressively reinforces their contribution via a cumulative selection count. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2-VL report improved CLAIR scores and a higher CHAIR F1 score, with the paper claiming SPARC is unique in improving both precision and recall.

Significance. If the reported results are reliable, SPARC is a simple, efficient, and practical contribution to hallucination mitigation in image captioning: it is training-free, adds only about 3% decoding overhead (Table 6), and comes with released code. The paper also provides useful ablations (Appendix C) and an analysis of attention dynamics. However, the central differentiator—the recall improvement—is not yet statistically supported, and the abstract overstates the precision/recall trade-off of prior methods.

major comments (4)
  1. [§5.2 / Table 2 / Table 13] The central claim that SPARC uniquely improves recall rests on a 0.52 percentage-point increase (79.46 to 79.98) that is reported without any variance measure or significance test. The paper states these values are averages over five random 500-instance samples, but no per-run values, standard deviations, or confidence intervals are given. Given the standard error of a proportion near 0.8 with several thousand objects, this difference is plausibly within sampling noise. Please provide per-run results and a paired statistical test (e.g., bootstrap over images or Wilcoxon signed-rank) to establish that the recall gain is not noise.
  2. [Abstract / Table 2] The abstract and introduction claim that existing methods improve precision at the cost of recall, but Table 2 shows OPERA and VCD have lower precision than the baseline (84.54 and 83.22 vs. 84.70) as well as lower recall (78.82 and 77.50 vs. 79.46). The paper's own text in §5.2 acknowledges this ('OPERA and VCD ... fail to improve precision or recall'). This overstatement should be corrected.
  3. [§5.1 / Appendix C] The per-model hyperparameters (τ and layer l) are selected based on performance on the same evaluation sets (IIW-400, DOCCI, and MS COCO) used for the final results; Tables 8–10 in Appendix C show that these parameters are swept on these datasets to pick the reported configuration. This selection procedure risks overfitting to the benchmark. To support generalization, the authors should either fix the hyperparameters on a held-out validation split or report results across the swept range to show the main conclusions are insensitive to reasonable choices.
  4. [§4.2 / Appendix B.1] The token-selection mechanism uses raw attention weights from a single middle layer (layer 20 for LLaVA models, 18 for Qwen2-VL) averaged across heads. The only supporting evidence for this signal is the qualitative gradient-weighted saliency comparison in Figure 12, which concerns the 'noisy attention' observation rather than the token-selection premise. Please provide quantitative evidence that the selected tokens are causally relevant to the generated caption, for example by comparing selection based on raw attention against selection based on saliency-weighted attention on CHAIR/CLAIR.
minor comments (7)
  1. [§5.2] 'We compar' should be 'We compare'; 'VOCANO' should be 'VOLCANO'.
  2. [Eq. (4)] 'β∈[0,1] isis the smoothing factor' contains a duplicated 'is'.
  3. [§5.2] The definition of CHAIR precision is inverted: the text says precision is the proportion of objects 'that do not appear in the reference captions'; that is the hallucination rate (CHAIR_i), not precision. Precision is 1 − CHAIR_i, as correctly stated in Appendix F.
  4. [Appendix D] 'UWith' at the start of a paragraph appears to be a typo for 'With'.
  5. [Table 2 caption] The caption says 'The best scores are bolded, while the second-best scores are underlined,' but no underlining is visible in the rendered table; please check the formatting.
  6. [§5.1] 'we generat captions' should be 'we generate captions'.
  7. [Appendix C] The ablation text refers to 'as shown in Figure 7(a)' when describing the scaling strategy for the no-selection setting; this reference is unclear because Figure 7(a) shows attention trends rather than a scaling strategy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SPARC's precision/recall claims are tested against external benchmarks and do not reduce by construction to its attention-selection inputs.

full rationale

The paper's derivation chain is not circular in the sense prohibited here. SPARC computes a Relative Activation Score from temporal changes in attention (Eq. 5), selects tokens by thresholding that score (Eq. 6), and amplifies their value vectors (Eqs. 8-9). The claimed outputs—CHAIR precision/recall, CLAIR scores, and human preference—are measured on external benchmarks and human judgments, and no equation in the paper makes those metrics equal to the method's inputs by definition. CHAIR recall, for example, counts ground-truth objects from MS-COCO, whereas SPARC only modifies internal attention; whether this improves recall is an empirical question that the experiments answer, not a tautology. The self-citation of Lee et al. 2024 (same author group) appears in Section 3.3 as support for the observation that longer contexts reduce visual focus, but that observation is independently corroborated by Favero et al. 2024 and by the paper's own Figure 5, so it is not load-bearing circularity. The more substantive concern is that hyperparameters are tuned via ablations on the same evaluation datasets (e.g., alpha = 1.1 selected by CHAIR F1 in Figure 14, and tau selected on IIW-400/DOCCI in Tables 8-10), which is an overfitting and statistical-reporting risk, not a definitional circularity; the recall gain of 0.52 pp is reported without variance or significance testing, and Section F itself concedes weaker CHAIRs performance. These are correctness and robustness concerns, but the central derivation remains self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 2 assumptions · 0 invented entities

The central claim rests on four fitted hyperparameters and two domain assumptions about attention semantics. No new physical or model entities are introduced. The paper does not derive the method from first principles; it is an empirically motivated heuristic.

free parameters (4)
  • alpha (scaling factor) = 1.1
    Scaling factor for value vectors; chosen via ablation in Appendix C, Table 8, where alpha=1.1 gave highest CLAIR on IIW-400.
  • beta (EMA smoothing factor) = 0.1
    EMA smoothing factor; ablation in Appendix C, Table 8 shows best at 0.1.
  • tau (selection threshold) = 1.5 (LLaVA-1.5), 4.0 (LLaVA-Next), 3.0 (Qwen2-VL)
    Selection threshold adjusted per model to optimize CLAIR, as stated in Section 5.1 and ablation in Appendix C.
  • layer l (token selection layer) = 20 (LLaVA models), 18 (Qwen2-VL)
    Layer for attention extraction; ablation in Appendix C shows layer 20 optimal for LLaVA-1.5, chosen similarly for other models.
assumptions (2)
  • domain assumption Attention weights in mid-to-late layers are a causally relevant proxy for the model's visual grounding, and scaling value vectors of selected tokens improves caption quality without introducing errors.
    SPARC selects tokens based on layer-20/18 attention and amplifies them; if attention is not grounding-relevant, selection is misdirected. This premise is invoked in Section 4.2 and Section 5.1, partially validated by saliency analysis in Appendix B.1.
  • domain assumption The Relative Activation Score (EMA-normalized attention increase) reliably identifies contextually relevant visual tokens across diverse captions.
    The threshold-based selection assumes that relative increases in attention correspond to semantic relevance; this is a heuristic without external grounding, introduced in Section 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/WN4NYAUM

@misc{pith2026250201419,
  author       = {Pith},
  title        = {Pith review of: Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WN4NYAUM}},
  note         = {Machine review of arXiv:2502.01419}
}
read the original abstract

Detailed image captioning is essential for tasks like data generation and aiding visually impaired individuals. High-quality captions require a balance between precision and recall, which remains challenging for current multimodal large language models (MLLMs). In this work, we hypothesize that this limitation stems from weakening and increasingly noisy visual attention as responses lengthen. To address this issue, we propose SPARC (Selective Progressive Attention ReCalibration), a training-free method that enhances the contribution of visual tokens during decoding. SPARC is founded on three key observations: (1) increasing the influence of all visual tokens reduces recall; thus, SPARC selectively amplifies visual tokens; (2) as captions lengthen, visual attention becomes noisier, so SPARC identifies critical visual tokens by leveraging attention differences across time steps; (3) as visual attention gradually weakens, SPARC reinforces it to preserve its influence. Our experiments, incorporating both automated and human evaluations, demonstrate that existing methods improve the precision of MLLMs at the cost of recall. In contrast, our proposed method enhances both precision and recall with minimal computational overhead.

Figures

Figures reproduced from arXiv: 2502.01419 by the authors.

Figure 1
Figure 1. Percentage change in precision, recall, and F1-score compared to the results before applying each method. ities (Li et al., 2023a; Liu et al., 2024c; Lin et al., 2024). These models leverage the powerful language capabilities of Large Language Models (LLMs) to generate textual descrip￾tions from visual inputs (Bai et al., 2023; Touvron et al., 2023; Abdin et al., 2024; Peng et al., 2023). This capabil￾ity enables ML… view at source ↗
Figure 2
Figure 2. Visualization of image token attention at different context lengths. As the generation context length increases, image atten￾tion diminishes, reducing reliance on visual inputs. Our approach mitigates this by preserving image attention, reducing hallucina￾tions and enabling more detailed captions. terms of recall and precision. Notably, our analysis reveals a previously overlooked limitation: these methods signif￾ic… view at source ↗
Figure 3
Figure 3. Visual attention diversity comparison between (a) the baseline model and (b) the naive attention enhancement approach. The naive approach reduces visual attention diversity, indicating ineffective adaptation to important visual tokens normalized visual attention weights for the first 100 tokens of each caption. We then compute a pairwise distance ma￾trix (100 × 100) of visual attention patterns. The distance is calc… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Average attention weight trends for text and image tokens as a function of context length during caption generation. As the context length increases, the proportion of attention allocated to image tokens gradually decreases compared to text tokens. This indicates a sig…
Figure 4
Figure 4. Figure 4: Visualization of the temporal dynamics of image atten￾tion during caption generation. Early in the process, attention is focused on contextually relevant regions, but as the caption grows longer, it increasingly shifts toward noise or consistently high￾attention tokens…
Figure 7
Figure 7. Figure 7: Visual Attention Analysis. (a) Change in visual attention with increasing context length. SPARC mitigates the decline com￾pared to the baseline. (b) Ratio of attention scales between sink and non-sink visual tokens during captioning. SPARC maintains baseline-level prop…
Figure 6
Figure 6. Figure 6: Human evaluation results showing the winning ratio (%) of our method compared to (a) the baseline and (b) naive approach in terms of precision and recall. CLAIR scores for the IIW-400 and DOCCI datasets, re￾spectively. For DOCCI, we randomly select 500 samples for eval…
Figure 8
Figure 8. Figure 8: Comparison of captions generated by applying our method to LLaVA-1.5 and the baseline. Red text highlights incorrect references in the captions, while blue text indicates additional details provided by our method compared to the baseline. 14 [PITH_FULL_IMAGE:figures/f…
Figure 9
Figure 9. Figure 9: Comparison of captions generated by applying our method to LLaVA-NeXT and the baseline. Red text highlights incorrect references in the captions, while blue text indicates additional details provided by our method compared to the baseline. 15 [PITH_FULL_IMAGE:figures/…
Figure 10
Figure 10. Figure 10: Comparison of captions generated by applying our method to LLaVA-NeXT and the baseline. Red text highlights incorrect references in the captions, while blue text indicates additional details provided by our method compared to the baseline. 16 [PITH_FULL_IMAGE:figures…
Figure 11
Figure 11. Figure 11: Visual attention diversity comparison between (a) the baseline model, (b) the naive attention enhancement approach, and (c) our method. The naive approach reduces visual attention diversity, indicating ineffective adaptation to important visual tokens. Instead, our pr…
Figure 12
Figure 12. Figure 12: Comparison between visualizations of attention scores and gradient-weighted attention scores for different context lengths during caption generation. Both methods exhibit similar trends, with increasing noise as the context length grows. Longer Context, Less Visual Fo…
Figure 13
Figure 13. Figure 13: Average attention weight trends for text and image tokens as a function of context length during caption generation, computed by dividing attention by the number of respective tokens. This normalized view reveals that attention to image tokens decreases more sharply t…
Figure 14
Figure 14. Figure 14: Object-level Precision, Recall, and F1 scores for LLaVA-1.5 (a) and LLaVA-NeXT (b) with scaling parameter α D. Detailed Caption Similarity Analysis In the previous sections, we observed that naively enhancing the attention on visual tokens leads to a reduction in the …
Figure 15
Figure 15. Figure 15: (a) Comparison of caption similarity scores and number of selected tokens. (a) Caption similarity score across different methods. (b) Number of dynamically selected tokens during caption generation. Methods that identify more tokens as important tend to yield lower si…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A self-evolving multimodal model using continuous self-consistency rewards improves math reasoning by about 2–3% using only raw images, without labels or external reward models.

Reference graph

Works this paper leans on

59 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., and Shou, M. Z. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  4. [4]

    Understanding information storage and transfer in multi-modal large language models

    Basu, S., Grayson, M., Morrison, C., Nushi, B., Feizi, S., and Massiceti, D. Understanding information storage and transfer in multi-modal large language models. arXiv preprint arXiv:2406.04236, 2024

    arXiv 2024

  5. [5]

    Y., and Furlotte, N

    Belyaeva, A., Cosentino, J., Hormozdiari, F., Eswaran, K., Shetty, S., Corrado, G., Carroll, A., McLean, C. Y., and Furlotte, N. A. Multimodal llms for health grounded in individual-specific data. In Workshop on Machine Learning for Multimodal Healthcare Data, pp.\ 86--102. Springer, 2023

    work page 2023

  6. [6]

    E., Darrell, T., and Canny, J

    Chan, D., Petryk, S., Gonzalez, J. E., Darrell, T., and Canny, J. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971, 2023

    arXiv 2023

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024

    2024

  8. [8]

    Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges

    Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287, 2023

    arXiv 2023

Show all 59 references
  1. [9]

    Vision transformers need registers

    Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1

    2024

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Multi-modal hallucination control by visual information grounding

    Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14303--14312, 2024

    2024

  4. [12]

    K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R

    Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024

    2024 arXiv

  5. [13]

    Damro: Dive into the attention mechanism of lvlm to reduce object hallucination

    Gong, X., Ming, T., Wang, X., and Wei, Z. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514, 2024

    2024

  6. [14]

    Detecting and preventing hallucinations in large vision language models

    Gunjal, A., Yin, J., and Bas, E. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18135--18143, 2024

    2024

  7. [15]

    Chartllama: A multimodal llm for chart understanding and generation

    Han, Y., Zhang, C., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., and Zhang, H. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023

    2023 arXiv

  8. [16]

    A multi-modal foundation model to assist people with blindness and low vision in environmental interaction

    Hao, Y., Yang, F., Huang, H., Yuan, S., Rangan, S., Rizzo, J.-R., Wang, Y., and Fang, Y. A multi-modal foundation model to assist people with blindness and low vision in environmental interaction. Journal of Imaging, 10 0 (5): 0 103, 2024

    2024

  9. [17]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2024

  10. [18]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models

    Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-introspective decoding: Alleviating hallucinations for large vision-language models. arXiv preprint arXiv:2408.02032, 2024

    2024 arXiv

  11. [19]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  12. [20]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens

    Jiang, Z., Chen, J., Zhu, B., Luo, T., Shen, Y., and Yang, X. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724, 2024

    2024 arXiv

  13. [21]

    Kang, S., Kim, J., Kim, J., and Hwang, S. J. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321, 2025

    2025 arXiv

  14. [22]

    H., Jo, Y., and Seo, M

    Lee, S., Park, S. H., Jo, Y., and Seo, M. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362, 2023

    2023 arXiv

  15. [23]

    Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage

    Lee, S., Yoon, S., Bui, T., Shi, J., and Yoon, S. Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint arXiv:2412.15484, 2024

    2024 arXiv

  16. [24]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13872--13882, 2024

    2024

  17. [25]

    Llava-onevision: Easy visual task transfer

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 a

    2023

  19. [27]

    Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding

    Li, J., Zhang, J., Jie, Z., Ma, L., and Li, G. Mitigating hallucination for large vision language model by inter-modality correlation calibration decoding. arXiv preprint arXiv:2501.01926, 2025

    2025 arXiv

  20. [28]

    Inference-time intervention: Eliciting truthful answers from a language model

    Li, K., Patel, O., Vi \'e gas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  21. [29]

    X., and Wen, J.-R

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023 b

    2023 arXiv

  22. [30]

    Vila: On pre-training for visual language models

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26689--26699, 2024

    2024

  23. [31]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  24. [32]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024 a

    2024

  25. [33]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 b . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  26. [34]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 c

    2024

  27. [35]

    A survey on hallucination in large vision-language models

    Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., and Peng, W. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024 d

    2024 arXiv

  28. [36]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms

    Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pp.\ 125--140. Springer, 2025

    2025

  29. [37]

    Nvila: Efficient frontier visual language models

    Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024 e

    2024 arXiv

  30. [38]

    Compositional chain of thought prompting for large multimodal models

    Mitra, C., Huang, B., Darrell, T., and Herzig, R. Compositional chain of thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024

    2024

  31. [39]

    Docci: Descriptions of connected and contrasting images

    Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., et al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp.\ 291--309. Springer, 2025

    2025

  32. [40]

    A., Shalaby, M

    Osman, A. A., Shalaby, M. A. W., Soliman, M. M., and Elsayed, K. M. A survey on attention-based models for image captioning. International Journal of Advanced Computer Science and Applications, 14 0 (2), 2023

    2023

  33. [41]

    Instruction tuning with gpt-4

    Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  34. [42]

    V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C

    Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. Sam 2: Segment anything in images and videos, 2024. URL http...

    2024 arXiv

  35. [43]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., and Zhang, L. Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    2024

  36. [44]

    A., Burns, K., Darrell, T., and Saenko, K

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  37. [45]

    Wasserstein distance guided representation learning for domain adaptation

    Shen, J., Qu, Y., Zhang, W., and Yu, Y. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  39. [47]

    Calculation of the wasserstein distance between probability distributions on the line

    Vallender, S. Calculation of the wasserstein distance between probability distributions on the line. Theory of Probability & Its Applications, 18 0 (4): 0 784--786, 1974

    1974

  40. [48]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  41. [49]

    Efficient large language models: A survey

    Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023

    2023 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  43. [51]

    Don't miss the forest for the trees: Attentional vision calibration for large vision language models

    Woo, S., Kim, D., Jang, J., Choi, Y., and Kim, C. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820, 2024

    2024 arXiv

  44. [52]

    Mitigating object hallucination via concentric causal attention

    Xing, Y., Li, Y., Laptev, I., and Lu, S. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926, 2024

    2024 arXiv

  45. [53]

    Less is more: Mitigating multimodal hallucination from an eos decision perspective

    Yue, Z., Zhang, L., and Jin, Q. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14545, 2024

    2024 arXiv

  46. [54]

    MLLM s know where to look: Training-free perception of small visual details with multimodal LLM s

    Zhang, J., Khayatkhoei, M., Chhikara, P., and Ilievski, F. MLLM s know where to look: Training-free perception of small visual details with multimodal LLM s. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=DgaY5mDdmT

    2025

  47. [55]

    Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms

    Zhang, X., Quan, Y., Gu, C., Shen, C., Yuan, X., Yan, S., Cheng, H., Wu, K., and Ye, J. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms. arXiv preprint arXiv:2411.09968, 2024

    2024 arXiv

  48. [56]

    Investigating and mitigating the multimodal hallucination snowballing in large vision-language models

    Zhong, W., Feng, X., Zhao, L., Li, Q., Huang, L., Gu, Y., Ma, W., Xu, Y., and Qin, B. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.00569, 2024

    2024 arXiv

  49. [57]

    Analyzing and mitigating object hallucination in large vision-language models

    Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023

    2023 arXiv

  50. [58]

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding

    Zhu, L., Ji, D., Chen, T., Xu, P., Ye, J., and Liu, J. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476, 2024

    2024 arXiv

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.