Mitigating Image Captioning Hallucinations in Vision-Language Models

Chengcui Zhang; Fei Zhao; Runlin Zhang; Tianyang Wang; Xi Li

REVIEW 3 major objections 6 minor 26 references

Mitigating Image Captioning Hallucinations in Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that updating only LayerNorm gamma parameters with a policy-gradient step, guided by a CLIP-based dual reward, reduces object hallucination in vision-language captioning at test time, reporting CHAIR reductions of 14.1%…

desk verdict Plausible idea and honest AMBER evaluation, but the RL derivation doesn't match the algorithm and the VCD baseline is implausible—needs major revision before it can be believed. read the letter →

arxiv 2505.03420 v2 pith:7RUOSTPS submitted 2025-05-06 cs.MM cs.CV

Fei Zhao , Chengcui Zhang , Runlin Zhang , Tianyang Wang , Xi Li This is my paper · ORCID

classification cs.MMcs.CV

keywords vision-languagemodelshallucinationmitigationtest-timeadaptationreinforcementlearninglayernormalizationCLIPrewardimagecaptioningobject

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a test-time adaptation framework that reduces object hallucinations in vision-language captioning without retraining or auxiliary models. The captioning model is treated as a reinforcement-learning policy, and for each test image only the layer-normalization gamma parameters, about 0.003% of the total, are updated. The reward comes from a lightweight CLIP-based evaluator that combines a semantic-alignment score with a non-hallucination probability. On LLaVA 7B and InstructBLIP 7B, the authors report CHAIR reductions of 14.1% and 26.1%, Hal reductions of 15.4% and 17.3%, and a roughly 68% lower hallucination rate than a contrastive-decoding baseline. The paper's point is that tiny, sample-specific normalization updates can steer a frozen pretrained model away from hallucinated objects.

What carries the argument

The load-bearing machinery is the pairing of two components: first, the LayerNorm gamma parameters as the only learnable parameters of the frozen VLM, providing a low-dimensional steering mechanism during inference; second, a hallucination evaluation model built from a frozen CLIP with one learnable query token refined through N-stage cross-attention and trained with a triplet loss, which outputs the dual rewards SAS and NHP. The policy-gradient objective $L_{\mathrm{policy}} = -\mathbb{E}_{\pi_\theta}[\log \pi(y|s;\theta)\, r(y,v)]$ is what formally converts those rewards into gamma updates, with beam search supplying the candidate captions.

What would settle it

On a held-out set of AMBER images, compare three variants: the full TTA pipeline, a variant that scores the same beam candidates but simply picks the highest-reward caption without any LayerNorm update, and a variant that updates a random 0.003% of parameters instead of LayerNorm gamma. If the re-ranking or random-parameter variant matches the full pipeline's CHAIR and Hal reductions, the claimed policy-gradient-to-LayerNorm mechanism is not what is causing the gains.

Watch

Extended reading notes

Core claim

The central discovery the paper aims to establish is that object hallucinations in deployed VLMs can be mitigated at inference time by treating the VLM as a policy and applying policy-gradient updates to only the LayerNorm gamma parameters. For each test image, the policy generates B beam-search captions; a CLIP-based hallucination evaluator scores each candidate with dual rewards, the Semantic Alignment Score from cosine similarity and the Non-Hallucination Probability from a classifier logit; the aggregated reward then drives a policy-gradient loss that updates gamma. After each sample the parameters reset to their pretrained values. The paper reports consistent reductions on CHAIR, Hal, and Cog for both LLaVA 7B and InstructBLIP 7B, and claims the approach outperforms visual contrastive decoding on these metrics.

Load-bearing premise

The derivation treats a policy-gradient update computed from beam-search candidate captions as a valid estimate of the expected-reward gradient, but beam search is deterministic and is not sampling from the model's own distribution.

Editorial extensions

If this is right

Deployed captioning models can be adapted per test image without retraining, making hallucination mitigation available on demand in high-stakes applications.
Because only LayerNorm gamma changes and is reset after each sample, the pretrained model's general knowledge is preserved while sample-specific biases are not accumulated.
The dual-reward design shows that semantic alignment alone is insufficient: adding the non-hallucination classifier logit yields measurable further reductions in CHAIR, Hal, and Cog.
The framework transfers across two different 7B-scale VLMs, suggesting the parameter-efficient update is not tied to a single architecture.
The CLIP-based evaluator, trained on object, attribute, and relation hallucinations, can be reused for broader hallucination detection beyond the current scope.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the policy-gradient estimate is computed from deterministic beam-search outputs rather than sampled from $\pi_\theta$, the update may not truly maximize expected reward; a simpler best-of-B re-ranking guided by the same dual rewards might reproduce the reported gains.
The LayerNorm-gamma update may act as a test-time feature re-scaling that shifts activation distributions toward the current image, which would connect this work to test-time normalization methods even outside the RL framing.
The dual-reward evaluator could be combined with the same loop to address attribute and relation hallucinations, which the paper leaves to future work.
Varying the number of adaptation steps or the beam size could reveal whether hallucination reductions saturate or reverse, separating the contribution of the per-sample update from the evaluator's ranking quality.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear Colleague,

Quick take: the headline claim—that updating only LayerNorm gamma with a CLIP-based reward reduces object hallucination rates by 15–17% on LLaVA and InstructBLIP—may be real, but this paper doesn't yet support it. The RL derivation in III-A doesn't match Algorithm 1, and the VCD baseline in Table III is so implausible that it casts doubt on the whole comparison.

What's new: adapting a VLM at test time by updating only LayerNorm gamma (about 0.003% of parameters) to reduce hallucination is a neat, parameter-efficient idea. I haven't seen that exact combination. The custom evaluator—frozen CLIP plus one learnable query token, multi-stage cross-attention, triplet loss, and dual SAS/NHP rewards—is also a legitimate new component. Credit where due: the main results are evaluated on AMBER ground truth, not on the reward model's own scores, so the central empirical claim isn't circular. The paper is also honest about scope, limiting itself to object hallucinations.

Where it falls apart: Algorithm 1 generates candidate captions with beam search, but Section III-A defines the policy gradient as an expectation under the policy's softmax. Beam search is deterministic and does not sample from πθ, so the implemented update is not ∇θJ(θ) as stated. The authors need to either sample from πθ or reframe the loss as reward-weighted regression on beam-selected outputs. This isn't cosmetic—it changes what the method actually is. Second, Table III shows VCD on LLaVA with CHAIR 80.5 and Hal 97.5, while the unadapted LLaVA in Table I has CHAIR 7.8 and Hal 36.4. That's backwards. VCD is designed to reduce hallucination, not make it an order of magnitude worse. Either the numbers are misreported or the baseline is broken. Third, there are no error bars or significance tests on any of the improvements, and no code is released. Given the effect sizes (a few points on CHAIR), we can't tell if the gains are stable.

The paper deserves a serious referee because the idea is interesting and the flaws are identifiable and fixable. But I wouldn't accept it as-is. The authors need to correct the derivation (or reframe), rerun the VCD baseline properly, add variance estimates, and release code. Then the empirical claim can actually be checked.

Recommendation: send to peer review with expectation of major revision. If the derivation and baseline issues are addressed, this could be a useful contribution to the hallucination-mitigation subfield.

Referee Report

3 major / 6 minor

Summary. This paper proposes a test-time adaptation (TTA) method to mitigate object hallucinations in vision-language models (VLMs). The method freezes all model parameters except layer-normalization gamma parameters (approximately 0.003% of the total), generates B candidate captions via beam search, evaluates them with a custom CLIP-based hallucination evaluation model that produces Semantic Alignment Scores (SAS) and Non-Hallucination Probability (NHP) rewards, and updates the gamma parameters with a policy-gradient-style loss. Experiments on the AMBER benchmark report reductions in CHAIR, Hal, and Cog for LLaVA-7B and InstructBLIP-7B, and claim substantial improvements over the Visual Contrastive Decoding (VCD) baseline. Evaluation is performed on external AMBER ground truth, so the headline results are not circular with respect to the reward model's own outputs.

Significance. The idea of adapting only layer-normalization parameters at test time is interesting and potentially efficient, and the within-model comparisons show consistent directional improvements on CHAIR, Hal, and Cog for both models. The use of an external benchmark (AMBER) for evaluation is a strength and avoids immediate circularity. However, the paper's central mechanism is undermined by the mismatch between the stated policy-gradient derivation and the beam-search-based implementation, and the SOTA comparison contains implausible baseline numbers. If these issues are resolved, the method could be a useful lightweight hallucination-mitigation technique; as it stands, the theoretical and empirical support are insufficient.

major comments (3)

[III-A, Algorithm 1] The policy-gradient update in §III-A is not an unbiased estimator of ∇θJ(θ) because Algorithm 1 generates candidate captions with deterministic beam search rather than sampling from πθ. The objective J(θ)=Eπθ[r(y,v)] and gradient ∇θJ(θ)=Eπθ[∇θ log π(y|s;θ) r(y,v)] require samples from the policy. Beam search is a deterministic argmax-like procedure, and without importance weights or an off-policy correction, the loss Lpolicy=−logπ(y|s;θ) r(y,v) is not an unbiased estimate of the policy gradient. The beam-search selection may reinforce high-likelihood sequences independently of reward, confounding the reported gains. The authors should either sample from the policy, apply an appropriate off-policy correction, or re-derive the method as a deterministic risk/reranking objective and explain the observed improvements in those terms.
[Table III] The VCD baseline numbers in Table III are inconsistent with the unadapted model in Table I. For LLaVA-7B, Table I reports w/o TTA as CHAIR=7.8, Cover=51.0, Hal=36.4, Cog=4.2, while Table III reports w/ VCD as CHAIR=80.5, Cover=10.3, Hal=97.5, Cog=9.2. VCD is designed to mitigate hallucinations, so such a large degradation is implausible unless the VCD implementation is broken, hyperparameters are inappropriate, or the evaluation protocol differs. This invalidates the claimed 91.7% CHAIR reduction and 68.4% Hal improvement over SOTA. The authors must report a correctly configured VCD baseline with the same evaluation code, include implementation details, and verify the numbers.
[Tables I, IV; §IV hyperparameters] No error bars, confidence intervals, or significance tests are reported for any of the main results. The improvements are small in absolute terms (for example, CHAIR 7.8→6.7 for LLaVA-7B), so without repeated runs or significance testing the robustness of the claim is unclear. In addition, several hyperparameters (β, learning rates, beam size B=5, number of adaptation steps=5, and the reward normalization procedure) are set without sensitivity analysis; the method's sensitivity to these choices should be examined.

minor comments (6)

[References] Reference [8] does not point to InstructBLIP; it cites a survey. The correct InstructBLIP paper (Dai et al.) should be cited.
[III-B] The normalization of SAS and NHP scores is only described in prose as subtracting the mean across candidates; please define the normalization operator explicitly in an equation.
[IV] The description of using 'LLaMA3 to generate negative image-caption pairs' lacks details; please specify the prompts, filtering criteria, and how you ensure the generated negatives contain object hallucinations rather than merely being dissimilar.
[Algorithm 1] Step 8 says 'Generate the refined caption y* using the updated VLM' but does not specify the decoding method; please state whether beam search with the same B is used or a different decoding strategy, and ensure the evaluation protocol matches.
[Table V] The comparison between CLIP-based models and LLaVA-based models for hallucination detection is potentially misleading because LLaVA is evaluated on a binary classification task it was not trained for; please clarify the task setup or remove this comparison if the settings are not comparable.
[Abstract] The abstract claims 'outperforms state-of-the-art baselines with a 68.3% improvement', but this number depends on the implausible VCD baseline in Table III; please qualify or revise the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: headline hallucination reductions are measured on external AMBER ground truth, not on the reward model's own scores, and no load-bearing derivation step reduces to its input.

full rationale

The paper's central empirical claim is that test-time RL updates to LayerNorm gamma reduce object hallucination rates on LLaVA and InstructBLIP. The reward signal is indeed produced by a trained CLIP-based evaluator (SAS plus NHP), but that evaluator is trained on a PixelProse subset with LLaMA3-generated negative captions, whereas the headline results are evaluated with AMBER's external CHAIR/Hal/Cover ground truth. Thus the reported reductions are not measured by the fitted reward model itself, and no parameter is fitted directly to the AMBER evaluation metric. The RL objective J(θ)=Eπθ[r(y,v)] is defined independently of the later empirical validation, and the policy-gradient loss is stated in standard expectation form. The paper's self-citations ([1], [9]) are contextual prior work and do not carry the central argument. The beam-search-versus-policy-sampling mismatch in Algorithm 1 is a statistical validity concern about whether the implemented update estimates the stated gradient, but it does not make any claimed result equivalent to its input by construction; therefore it is outside this circularity pass. Overall, the derivation chain is not circular: the fitted component is used as an internal reward surrogate, while the main empirical verification is anchored to an external benchmark.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on unstated assumptions: the validity of a policy-gradient update computed from beam-search candidates, the transfer of a CLIP evaluator trained on LLaMA3 negatives to AMBER, and the assertion that LayerNorm gamma updates reduce distribution shift. No new physical entities are introduced; the learnable query token is a trained parameter inside an evaluated module.

free parameters (5)

β reward weight = 1
Section III-B sets r = norm SAS + β * norm NHP with β=1 empirically; no sensitivity analysis is provided.
Triplet loss margin α = not reported
Section III-C defines L_triplet = max(0, cos(CLS_N, Qneg_N) - cos(CLS_N, Qpos_N) + α), but the value of α is never given.
Learning rates = 2e-3 for LLaVA, 2e-4 for InstructBLIP
Section IV sets these for RL-based inference; they are chosen per model without ablation.
Beam size B and adaptation steps = B=5, steps=5
Section IV sets these values; no study of sensitivity is reported.
Number of cross-attention stages N = not reported
Section III-C defines N-stage cross-attention for the query token but never states N.

assumptions (5)

standard math Policy gradient theorem and the validity of the gradient estimator.
Section III-A uses ∇θJ(θ)=Eπθ[∇θ log π(y|s;θ) r(y,v)], which is only valid for samples from πθ.
domain assumption CLIP embeddings are a valid semantic alignment signal for image-text pairs.
The SAS score is the cosine similarity between CLIP image and text features (Section III-B); this relies on CLIP zero-shot generalization.
domain assumption The trained CLIP evaluator generalizes from LLaMA3-generated hallucinated negatives to AMBER test captions.
The reward model is trained on PixelProse positives and LLaMA3 negatives (Section IV), then applied to AMBER samples.
ad hoc to paper Updating only LayerNorm gamma can steer the frozen VLM toward lower hallucination.
This mechanism is asserted in the Abstract and Section I but is not directly measured.
ad hoc to paper Beam search candidates are a valid substitute for policy samples in policy gradient.
Algorithm 1 generates B candidates with beam search, while the loss is written as an expectation under πθ; no justification is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Image Captioning Hallucinations in Vision-Language Models." pith.science (2026). https://pith.science/paper/7RUOSTPS

@misc{pith2026250503420,
  author       = {Pith},
  title        = {Pith review of: Mitigating Image Captioning Hallucinations in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RUOSTPS}},
  note         = {Machine review of arXiv:2505.03420}
}

read the original abstract

Hallucinations in vision-language models (VLMs) hinder reliability and real-world applicability, usually stemming from distribution shifts between pretraining data and test samples. Existing solutions, such as retraining or fine-tuning on additional data, demand significant computational resources and labor-intensive data collection, while ensemble-based methods incur additional costs by introducing auxiliary VLMs. To address these challenges, we propose a novel test-time adaptation framework using reinforcement learning to mitigate hallucinations during inference without retraining or any auxiliary VLMs. By updating only the learnable parameters in the layer normalization of the language model (approximately 0.003% of the model parameters), our method reduces distribution shifts between test samples and pretraining samples. A CLIP-based hallucination evaluation model is proposed to provide dual rewards to VLMs. Experimental results demonstrate a 15.4% and 17.3% reduction in hallucination rates on LLaVA and InstructBLIP, respectively. Our approach outperforms state-of-the-art baselines with a 68.3% improvement in hallucination mitigation, demonstrating its effectiveness.

Figures

Figures reproduced from arXiv: 2505.03420 by the authors.

**Figure 1.** (a) The overall framework for mitigating object hallucinations in VLMs using TTA. The red texts indicate object hallucinations in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 9 canonical work pages

[1]

Deep multimodal data fusion,

F. Zhao, C. Zhang, and B. Geng, “Deep multimodal data fusion,” ACM Computing Surveys, vol. 56, no. 9, pp. 1–36, 2024

work page 2024
[2]

A survey on hallucination in large vision-language models,

H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,” arXiv preprint arXiv:2402.00253 , 2024

arXiv 2024
[3]

Understanding and im- proving layer normalization,

J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin, “Understanding and im- proving layer normalization,” Advances in neural information processing systems, vol. 32, 2019

work page 2019
[4]

Miti- gating object hallucinations in large vision-language models through vi- sual contrastive decoding,

S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Miti- gating object hallucinations in large vision-language models through vi- sual contrastive decoding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 872–13 882

work page 2024
[5]

Chatgpt (mar 14 version),

OpenAI, “Chatgpt (mar 14 version),” 2023, large language model. [Online]. Available: https://chat.openai.com

work page 2023
[6]

Gemini (dec 1 version),

G. DeepMind, “Gemini (dec 1 version),” 2023, large language model. [Online]. Available: https://gemini.deepmind.com

work page 2023
[7]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

2023
[8]

Visual instruction tuning towards general-purpose multimodal model: A survey,

J. Huang, J. Zhang, K. Jiang, H. Qiu, and S. Lu, “Visual instruction tuning towards general-purpose multimodal model: A survey,” arXiv preprint arXiv:2312.16602, 2023

arXiv 2023

Show all 26 references

[9]

Checkguard: Advancing stolen check detection with a cross-modal image-text bench- mark dataset,

F. Zhao, J. Chen, B. Huang, C. Zhang, and G. Warner, “Checkguard: Advancing stolen check detection with a cross-modal image-text bench- mark dataset,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , 2024, pp. 5425–5429

2024
[10]

Llava-next: Improved reasoning, ocr, and world knowledge,

H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

2024
[11]

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y . Qiao, and J. Dai, “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” arXiv preprint arXiv:2312.14238 , 2023

2023 arXiv
[12]

Prismer: A vision-language model with an ensemble of experts,

S. Liu, L. Fan, E. Johns, Z. Yu, C. Xiao, and A. Anandkumar, “Prismer: A vision-language model with an ensemble of experts,” arXiv preprint arXiv:2303.02506, vol. 3, 2023

2023 arXiv
[13]

Detecting and evaluating medical hallucinations in large vision language models,

J. Chen, D. Yang, T. Wu, Y . Jiang, X. Hou, M. Li, S. Wang, D. Xiao, K. Li, and L. Zhang, “Detecting and evaluating medical hallucinations in large vision language models,” arXiv preprint arXiv:2406.10185 , 2024

2024 arXiv
[14]

Test-time adaptation with clip reward for zero-shot generalization in vision-language models,

S. Zhao, X. Wang, L. Zhu, and Y . Yang, “Test-time adaptation with clip reward for zero-shot generalization in vision-language models,” arXiv preprint arXiv:2305.18010, 2023

2023 arXiv
[15]

Clipcap: Clip prefix for image captioning,

R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,” arXiv preprint arXiv:2111.09734 , 2021

2021 arXiv
[16]

Mitigating large vision- language model hallucination at post-hoc via multi-agent system,

C.-E. J. Yu, B. Jalaian, and N. D. Bastian, “Mitigating large vision- language model hallucination at post-hoc via multi-agent system,” in Proceedings of the AAAI Symposium Series , vol. 4, no. 1, 2024, pp. 110–113

2024
[17]

Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback,

W. Xiao, Z. Huang, L. Gan, W. He, H. Li, Z. Yu, H. Jiang, F. Wu, and L. Zhu, “Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback,” arXiv preprint arXiv:2404.14233 , 2024

2024 arXiv
[18]

Mitigating object hallucination in large vision-language models via classifier-free guidance,

L. Zhao, Y . Deng, W. Zhang, and Q. Gu, “Mitigating object hallucination in large vision-language models via classifier-free guidance,” arXiv preprint arXiv:2402.08680, 2024

2024 arXiv
[19]

Best-first beam search,

C. Meister, T. Vieira, and R. Cotterell, “Best-first beam search,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 795–809, 2020. [Online]. Available: https://aclanthology.org/2020. tacl-1.51

2020
[20]

Policy gradi- ent methods for reinforcement learning with function approximation,

R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradi- ent methods for reinforcement learning with function approximation,” Advances in neural information processing systems , vol. 12, 1999

1999
[21]

Learning transferable visual models from natural language supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

2021
[22]

Attention is all you need,

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

2017
[23]

Facenet: A unified embed- ding for face recognition and clustering,

F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embed- ding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 815– 823

2015
[24]

An llm-free multi-dimensional benchmark for mllms hallucination evaluation,

J. Wang, Y . Wang, G. Xu, J. Zhang, Y . Gu, H. Jia, M. Yan, J. Zhang, and J. Sang, “An llm-free multi-dimensional benchmark for mllms hallucination evaluation,” arXiv preprint arXiv:2311.07397 , 2023

2023 arXiv
[25]

From pixels to prose: A large dataset of dense image captions,

V . Singla, K. Yue, S. Paul, R. Shirkavand, M. Jayawardhana, A. Ganj- danesh, H. Huang, A. Bhatele, G. Somepalli, and T. Goldstein, “From pixels to prose: A large dataset of dense image captions,” arXiv preprint arXiv:2406.10328, 2024

2024 arXiv
[26]

Llama 3.2 multimodal (version 2023),

M. AI, “Llama 3.2 multimodal (version 2023),” 2023, large language model. [Online]. Available: https://ai.meta.com/llama

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Deep multimodal data fusion,

F. Zhao, C. Zhang, and B. Geng, “Deep multimodal data fusion,” ACM Computing Surveys, vol. 56, no. 9, pp. 1–36, 2024

work page 2024

[2] [2]

A survey on hallucination in large vision-language models,

H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,” arXiv preprint arXiv:2402.00253 , 2024

arXiv 2024

[3] [3]

Understanding and im- proving layer normalization,

J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin, “Understanding and im- proving layer normalization,” Advances in neural information processing systems, vol. 32, 2019

work page 2019

[4] [4]

Miti- gating object hallucinations in large vision-language models through vi- sual contrastive decoding,

S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Miti- gating object hallucinations in large vision-language models through vi- sual contrastive decoding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 872–13 882

work page 2024

[5] [5]

Chatgpt (mar 14 version),

OpenAI, “Chatgpt (mar 14 version),” 2023, large language model. [Online]. Available: https://chat.openai.com

work page 2023

[6] [6]

Gemini (dec 1 version),

G. DeepMind, “Gemini (dec 1 version),” 2023, large language model. [Online]. Available: https://gemini.deepmind.com

work page 2023

[7] [7]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

2023

[8] [8]

Visual instruction tuning towards general-purpose multimodal model: A survey,

J. Huang, J. Zhang, K. Jiang, H. Qiu, and S. Lu, “Visual instruction tuning towards general-purpose multimodal model: A survey,” arXiv preprint arXiv:2312.16602, 2023

arXiv 2023