Mitigating Object Hallucination via Robust Local Perception Search

Chaochao Lu; Chao Yang; Xing Xu; Zhanhui Zhou; Zixian Gao

REVIEW 4 major objections 5 minor 47 references

Mitigating Object Hallucination via Robust Local Perception Search

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LPS, a training-free decoding method that scores candidate sentences against the model's own localized object list, suppresses object hallucination in MLLMs, with the largest gains under adversarial image noise.

desk verdict Useful training-free hallucination mitigation with consistent but modest gains; the unmeasured reliability of the self-generated prior under adversarial noise is the load-bearing risk. read the letter →

arxiv 2506.06729 v1 pith:5D3QDOSK submitted 2025-06-07 cs.CV cs.CL

Zixian Gao , Chao Yang , Zhanhui Zhou , Xing Xu , Chaochao Lu This is my paper

classification cs.CVcs.CL

keywords objecthallucinationmultimodallargelanguagemodelsinference-timedecodinglocalperceptionprioradversarialrobustnessreward-guidedsearchPOPEbenchmarkCHAIR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Local Perception Search (LPS) aims to establish that a multimodal large language model can reduce object hallucination by using its own localized perception as a guide: before answering, the model is prompted to scan the image in four sections and list the objects it finds there. At each sentence-level decoding step, several candidate continuations are sampled and ranked by how well they match that self-generated object list, combined with a CLIP-based image-text similarity score. The paper reports consistent hallucination reductions over a CLIP-reward baseline across Qwen 2.5 VL, LLaMA 3.2 Vision, Phi 3.5 Vision, and LLaVA-1.5 (7B/13B) on the POPE and CHAIR benchmarks, with the largest margins on an adversarially perturbed version of the Multitrust dataset. The significance is practical: if the claim holds, it gives a cheap, model-agnostic, training-free lever for making vision-language outputs more trustworthy exactly where they currently fail most under corrupted or attacked visual input.

What carries the argument

The machinery is a two-term reward used inside stepwise sentence-level search. First, LPS obtains a local perception prior by prompting the model with 'Please carefully observe the top, bottom, left, and right parts of the image in sequence, and list the objects present in each section.' At each decoding step, $k$ candidate sentence continuations are sampled, and each candidate is scored by $R_t^i = \alpha \cdot \text{CosSim}(\bar{y}, c_t^i) + \beta \cdot \text{CLIP}(I, c_t^i)$, where $\bar{y}$ is the prior object list; the highest-scoring continuation is chosen. The cosine term is the load-bearing new signal: it rewards outputs that agree with the model's own localized object list, while the CLIP term keeps the output globally aligned with the image.

What would settle it

Compute the precision and recall of the local perception prior against ground-truth object annotations on the extended Multitrust attack set. If the prior's accuracy on attacked images is no better than the model's direct captioning accuracy, then the gains attributed to LPS cannot come from the local prior term in Eq. (3), and the experiment would instead point to the CLIP term or the sampling search as the active ingredient.

Watch

Extended reading notes

Core claim

The paper's discovery is that multimodal LLMs have a 'local perception ability': when asked to inspect the top, bottom, left, and right sections of an image in sequence, a model that fails on the whole image can still name the objects in each local region, even under adversarial perturbation. LPS turns this self-generated object list into a value function for inference-time search: at each sentence step it samples several candidate continuations and selects the one with highest cosine similarity to the prior plus highest CLIP image-text similarity. The authors report that this procedure lowers object hallucination on POPE and CHAIR across Qwen 2.5 VL, LLaMA 3.2 Vision, Phi 3.5 Vision, and LLaVA-1.5 (7B/13B), and that the advantage over a CLIP-only reward baseline is largest when the image is adversarially attacked.

Load-bearing premise

The method rests on the assumption that prompting the model to scan an image in sections produces a factually correct object list even when the image is noisy or adversarially attacked; if that list is wrong, the scoring rule rewards outputs that match the wrong list.

Editorial extensions

If this is right

LPS can be applied to any MLLM without retraining or external grounding detectors, needing only one extra forward pass to obtain the local perception prior.
On the extended Multitrust adversarial set, LPS improves over the CLIP-PRM baseline under both 100-step and 500-step attacks, with the largest gains on Phi 3.5 Vision.
The ablation on candidate count shows performance rises with more candidates per step and then plateaus, consistent with a search that saturates as it approaches an upper bound.
LPS also reduces CHAIR sentence-level and instance-level caption hallucination rates while preserving BLEU, so the gains are not limited to yes/no object-probing questions.
The final reward still includes a CLIP term, so LPS is not fully free of external models; the limitation section explicitly acknowledges this residual dependency and the extra inference overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable general principle behind the paper is that local attention is more robust to adversarial perturbation than global attention in MLLMs; measuring the accuracy of the local prior across attack strengths would directly probe that principle.
Because the reward is a cosine similarity to a text prior, LPS may bias generation toward generic, high-frequency object names; a caption-diversity or specificity metric would reveal whether hallucination reduction comes at the cost of informativeness.
The same recipe, self-prior via localized prompting followed by reward-guided decoding, could transfer to video or audio-language models, where global corruption is common and per-region perception may remain reliable.
An ablation that replaces the CLIP term with the model's own self-attention or likelihood score would show how much of the reported gain actually requires the external CLIP model rather than the local prior itself.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate, useful inference-time method. The core idea—generate a local perception prior by prompting the model to scan top/bottom/left/right, then use cosine similarity between candidate sentences and that prior, blended with a CLIP score, as a reward for sentence-level search—is genuinely not in the cited prior work. The paper shows consistent improvements over CLIP-PRM on POPE and CHAIR across Qwen 2.5 VL, LLaMA 3.2 Vision, and Phi 3.5 Vision, and better robustness than both CLIP-PRM and VCD on adversarial Multitrust. Code is public. That is real evidence, and the plug-and-play, training-free angle makes it practically appealing.

The main soft spot is exactly what the stress test flags: the prior is generated by the same model that is prone to hallucination under adversarial noise, and the paper never measures how often the prior itself is accurate. If the prior says 'dog' when there is no dog, Eq. (3) actively rewards candidates that also say 'dog.' The CLIP term can veto some of that, but CLIP is also susceptible to adversarial perturbation, so the safety margin is unclear. I don't think this is fatal—the aggregate results on adversarial benchmarks suggest the prior is usually good enough—but it is load-bearing and should have been measured directly (e.g., precision/recall of objects listed in the prior against the attacked images).

Other issues are smaller. There is an inconsistency: Section 4.1 says four candidate continuations per step, Appendix A.2 says six, and the ablation in Table 3 uses 1, 2, 4, 6, 8 with the best usually at 6. Presumably the main runs used 6; the text should say so. No error bars or significance tests anywhere, and several of the POPE gains are around 1%, so I'd want standard errors. The extended Multitrust dataset (100 to 1,000 images) is described vaguely—no statement about how the extra images were chosen or how ground-truth objects were obtained for the lenient 'mentions any ground truth object' metric. And the abstract's 'exceptional performance' is overclaiming; the gains are real but modest on clean images and larger under attack.

This is a solid submission, not a watershed. It deserves a serious peer review, and a good reviewer would ask for the prior reliability analysis, error bars, and a cleaned-up description of the dataset and candidate count. I'd bring it to reading group and would likely cite it if I worked on hallucination mitigation.

Referee Report

4 major / 5 minor

Summary. The paper proposes Local Perception Search (LPS), a training-free, plug-and-play decoding method for multimodal large language models (MLLMs) that aims to reduce object hallucination, especially under adversarial visual perturbations. LPS first prompts the MLLM to produce a textual 'local perception prior' by listing objects in the top, bottom, left, and right image regions, then at each decoding step samples several candidate sentence continuations and scores them with a weighted combination of cosine similarity to the prior and CLIP image-text similarity. The highest-scoring candidate is selected at each step. Experiments are reported on POPE, CHAIR, and an extended Multitrust adversarial dataset for Qwen 2.5 VL, LLaMA 3.2 Vision, Phi 3.5 Vision, and LLaVA 1.5-7B/13B, comparing against a CLIP-based reward model baseline and, on a subset, against VCD. The paper reports consistent but often modest improvements on clean benchmarks and larger gains on adversarially perturbed images.

Significance. If the central claim holds, LPS is a useful addition to inference-time hallucination mitigation: it is training-free, model-agnostic, and requires only the MLLM itself plus CLIP, without grounding detectors or trained reward models. The paper's strength is its breadth of empirical evaluation across multiple model families and scales, and the consistency of the reported improvements over CLIP-PRM. The code release is a concrete reproducibility asset. However, the load-bearing assumption — that the MLLM's self-generated local perception prior is sufficiently accurate under adversarial noise, and that Eq. (3) does not amplify hallucinated objects when the prior is wrong — is not systematically tested. This gap, together with missing statistical significance reporting and incomplete dataset-construction details, prevents the robustness claim from being fully established.

major comments (4)

[Sec. 3.3-3.4, Eq. (3)] The reward in Eq. (3) is the cosine similarity between a candidate continuation and the prior text y generated by the same MLLM using the Section 3.3 prompt. The abstract's robustness claim depends on y being factually accurate under adversarial perturbation, but the paper provides only anecdotal examples (Figs. 1, 2, 5) and does not report any systematic precision/recall of the local perception prior against ground-truth objects, on either clean or attacked images. This is load-bearing: if y contains a hallucinated object, any candidate that mentions that object receives a high R_t^LPS, so the search procedure would actively reinforce the errors it is intended to suppress. I request a quantitative prior-accuracy evaluation and an analysis of failure cases where the prior is wrong, including whether the CLIP term in Eq. (5) is sufficient to veto such candidates.
[Sec. 4.1, Table 2, and Appendix A.1(3)] The extension of the Multitrust adversarial set from 100 to 1,000 images is not described in sufficient detail: the paper does not state how the additional images were selected, how attacks were generated for them, or how ground-truth object lists were obtained for the evaluation. The metric in Appendix A.1(3) counts a response as accurate if it 'explicitly includes at least one ground truth object,' which could be satisfied by a generic or partially correct caption and does not penalize hallucinated objects that are also mentioned. This makes the reported gains in Table 2 difficult to interpret. Please provide the dataset-construction protocol, the annotation or automated extraction of ground-truth objects, and ideally additional metrics that penalize false mentions.
[Sec. 4.1, Appendix A.2, and Table 3] There is an internal inconsistency in the number of candidate continuations: Section 4.1 says 'four candidate continuations were generated' per step, while Appendix A.2 states 'we generate 6 candidate tokens at each decoding step,' and the ablation in Table 3 ranges over 1, 2, 4, 6, and 8 candidates. Since the ablation shows that performance varies non-monotonically with candidate number (e.g., Qwen 2.5 VL drops from 49.5 at 6 candidates to 49.3 at 8), the main-table results need to specify exactly which candidate count was used and why. In addition, no sensitivity analysis is reported for the weights α and β in Eq. (5), which are fixed to 1; these are free parameters and their impact on the central result should be assessed.
[Intro, Sec. 3.4, and Limitations] The paper repeatedly describes LPS as relying only on the MLLM's internal capabilities, e.g., 'without external supervision or auxiliary models' and 'without reliance on external tools or annotations,' yet Eq. (4) explicitly uses CLIP, an external model, as part of the reward, and this is acknowledged only in the Limitations. The framing is therefore contradictory. Since the main baseline is also CLIP-based (CLIP-PRM), the comparison is between two methods that both use CLIP; please clarify whether CLIP is essential to LPS, whether LPS can operate without it, and what the contribution is relative to a CLIP-only reward.

minor comments (5)

[Sec. 3.3] The text says 'as illustrated in Figure X' but no such figure is referenced; either insert the intended figure or fix the cross-reference.
[Sec. 2] The sentence 'This consistent improvement clearly demonstrates that LPS is effective...' appears in the Related Work section before any experiments are presented; this sentence should be removed or relocated.
[Sec. 4.3] The heading 'Comparsion with Other decoding method' contains a typo; it should read 'Comparison with other decoding methods.'
[Fig. 3 caption] The caption contains an apparent rendering artifact: a long sequence of '/uni00000024/...' characters. This should be removed.
[Appendix A.2] The hyperparameters are called 'a' and 'b' in the appendix but are denoted α and β in Eq. (5); please use consistent notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LPS's reliance on a self-generated prior is a robustness risk, not a derivation that reduces to its inputs.

full rationale

I find no circular step in the paper's derivation chain. The self-referential aspect—using the model's own local-perception prior y (Eq. 2) as the reward target in Eq. (3)—is an algorithmic design choice, not a fitted parameter or a prediction that is equivalent to its input by construction. The selected candidate is by definition the one maximizing the weighted reward in Eq. (5), but the paper's central claim that this reduces hallucination is not derived from that definition; it is tested against external ground-truth benchmarks (POPE, CHAIR, Multitrust) whose labels are independent of the prior. There are no load-bearing self-citations: the cited works by the same lab (Liu et al., 2024b; Zhou et al., 2024b) appear only as related-work context, and the adversarial attack reference (Chen et al., 2023) is external. The unresolved 'Figure X' placeholder in Sec. 3.3 and the lack of systematic measurement of prior precision under attack are genuine correctness and robustness limitations, but they do not constitute circularity because the evaluation protocol does not define hallucination in terms of the prior. Thus, despite the self-referential prior loop, the central empirical claim has independent content and the paper is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central claim depends on two main assumptions: the reliability of the self-generated local perception prior, and the validity of cosine similarity in embedding space as a factual alignment measure. The free parameters (alpha, beta, candidate count, step limit) are set without rigorous sensitivity analysis, and the candidate count is reported inconsistently.

free parameters (4)

alpha = 1
Weight for the local-prior cosine reward in Eq. (5), set to 1 without ablation.
beta = 1
Weight for the CLIP image-text reward in Eq. (5), set to 1 without ablation.
candidate number k = 6
Number of candidate continuations per decoding step; Table 3 shows performance peaking near 6, while Section 4.1 says 4 and Appendix A.2 says 6.
max search steps = 10
Maximum number of sentence-level decoding steps (Section 4.1).

assumptions (3)

domain assumption The cosine similarity between the MLLM's embedding of the prior text y and a candidate sentence c approximates factual agreement about image content.
Eq. (3) assumes this similarity is a good reward signal; no evidence is given that embedding proximity tracks factual correctness.
domain assumption The MLLM's local perception ability (prompting for objects in spatial regions) produces a reliable prior even under adversarial perturbation.
Section 3.2 and Fig. 2 argue this from examples, but it is not systematically validated; the entire method depends on it.
domain assumption Standard MLLM and CLIP architectures provide compatible embedding spaces for cosine similarity.
Eqs. (3) and (4) compare embeddings across model families without calibration.

invented entities (1)

Local Perception Prior
purpose: A textual list of objects in image regions generated by the MLLM itself, used as a reward signal to guide decoding.
The paper postulates that MLLMs possess a local perception ability whose output is more reliable than global decoding under adversarial noise. The only evidence is internal to the paper's own experiments; no external falsifiable prediction is made.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Object Hallucination via Robust Local Perception Search." pith.science (2026). https://pith.science/paper/5D3QDOSK

@misc{pith2026250606729,
  author       = {Pith},
  title        = {Pith review of: Mitigating Object Hallucination via Robust Local Perception Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5D3QDOSK}},
  note         = {Machine review of arXiv:2506.06729}
}

read the original abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have enabled them to effectively integrate vision and language, addressing a variety of downstream tasks. However, despite their significant success, these models still exhibit hallucination phenomena, where the outputs appear plausible but do not align with the content of the images. To mitigate this issue, we introduce Local Perception Search (LPS), a decoding method during inference that is both simple and training-free, yet effectively suppresses hallucinations. This method leverages local visual prior information as a value function to correct the decoding process. Additionally, we observe that the impact of the local visual prior on model performance is more pronounced in scenarios with high levels of image noise. Notably, LPS is a plug-and-play approach that is compatible with various models. Extensive experiments on widely used hallucination benchmarks and noisy data demonstrate that LPS significantly reduces the incidence of hallucinations compared to the baseline, showing exceptional performance, particularly in noisy settings.

Figures

Figures reproduced from arXiv: 2506.06729 by the authors.

**Figure 2.** Overall inference framework of our method. The dashed box highlights the Local Perception Prior process, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Performance comparison between VCD and LPS on the POPE and Multitrust datasets. "Adv" refers to adversarial performance, while "Ovrl" indicates overall performance. "Mul-100" and "Mul-500" correspond to Multitrust under 100-step and 500-step adversarial attacks, respectively. dataset. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance comparison of CLIP-PRM and LPS on LLaVA 1.5 models with different parameter scales (7B vs. 13B) across POPE and CHAIR datasets. Comparison Across Models with Different Parameter Scales. To further validate the generality of our proposed LPS method, we con…

**Figure 5.** Figure 5: Qualitative comparison of LPS and CLIP-PRM methods when handling adversarially perturbed images. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 8 canonical work pages

[1]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024
[2]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

2015
[3]

Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. 2023. Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint arXiv:2303.09105

arXiv 2023
[4]

Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

arXiv 2024
[5]

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

arXiv 2023
[6]

Yi Ding, Bolian Li, and Ruqi Zhang. 2024. Eta: Evaluating then aligning safety of vision language models at inference time. arXiv preprint arXiv:2410.06625

arXiv 2024
[7]

Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936

arXiv 2022
[8]

Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, and Jiawei Zhou. 2024. From uncertainty to trust: Enhancing reliability in vision-language models with uncertainty-guided dropout decoding. arXiv preprint arXiv:2412.06474

arXiv 2024

Show all 47 references

[9]

Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, and 1 others. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296

2024 arXiv
[10]

Zixian Gao, Xun Jiang, Xing Xu, Fumin Shen, Yujie Li, and Heng Tao Shen. 2024. Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26876--26885

2024
[11]

Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214

2024
[12]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[13]

Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135--18143

2024
[14]

Yangyang Guo, Fangkai Jiao, Liqiang Nie, and Mohan Kankanhalli. 2024. The vllm safety paradox: Dual ease in jailbreak attack and defense. arXiv preprint arXiv:2411.08410

2024 arXiv
[15]

Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, and Donglai Wei. 2023. Cliptrans: transferring visual knowledge with pre-trained models for multimodal machine translation. In Proceedings of the IEEE/CVF international conference on computer vision...

2023
[16]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

2024 arXiv
[17]

Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984

2023 arXiv
[18]

Junho Kim, Hyunjun Kim, Kim Yeonju, and Yong Man Ro. 2024. Code: Contrasting self-generated description to combat hallucination in large multi-modal models. Advances in Neural Information Processing Systems, 37:133571--133599

2024
[19]

Yeonju Kim, Junho Kim, Byung-Kwan Lee, Sebin Shin, and Yong Man Ro. 2023. Mitigating dataset bias in image captioning through clip confounder-free captioning network. In 2023 IEEE International Conference on Image Processing (ICIP), pages 1720--1724. IEEE

2023
[20]

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. https://arxiv.org/abs/2311.16922 Mitigating object hallucinations in large vision-language models through visual contrastive decoding . Preprint, arXiv:2311.16922

2023 arXiv
[21]

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

2024 arXiv
[22]

Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. 2024 b . Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706

2024 arXiv
[23]

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. https://arxiv.org/abs/2305.10355 Evaluating object hallucination in large vision-language models . Preprint, arXiv:2305.10355

2023 arXiv
[24]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, ...

2014
[25]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . Preprint, arXiv:2310.03744

2024 arXiv
[26]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

2023
[27]

Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. 2024 b . Inference-time language model alignment via integrated value guidance. arXiv preprint arXiv:2409.17819

2024 arXiv
[28]

Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910--10921

2023
[29]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[30]

Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. 2024. Mllm-protector: Ensuring mllm's safety without hurting performance. arXiv preprint arXiv:2401.02906

2024 arXiv
[31]

Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, and James Glass. 2024. Quantifying generalization complexity for large language models. arXiv preprint arXiv:2410.01769

2024 arXiv
[32]

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2019. https://arxiv.org/abs/1809.02156 Object hallucination in image captioning . Preprint, arXiv:1809.02156

2019 arXiv
[33]

Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. Advances in Neural Information Processing Systems, 37:52723--52748

2024
[34]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[35]

Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and 1 others. 2023. Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397

2023 arXiv
[36]

Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2024. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704

2024 arXiv
[37]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

2024 arXiv
[38]

Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. 2023. Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints, pp. arXiv--2310

2023
[39]

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and 1 others. 2024 a . Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973

2024 arXiv
[40]

Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024 b . Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms. arXiv preprint arXiv:2411.09968

2024 arXiv
[41]

Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, and 1 others. 2024 c . Multitrust: A comprehensive benchmark towards trustworthy multimodal large language models. Advances in Neural Information Processi...

2024
[42]

Henry Hengyuan Zhao, Pan Zhou, Difei Gao, Zechen Bai, and Mike Zheng Shou. 2024. Lova3: Learning to visual question answering, asking and assessment. Advances in Neural Information Processing Systems, 37:115146--115175

2024
[43]

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024 a . Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622

2024 arXiv
[44]

Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024 b . Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405.19262

2024 arXiv
[45]

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

2023 arXiv
[46]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[47]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024

[2] [2]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

2015

[3] [3]

Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. 2023. Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint arXiv:2303.09105

arXiv 2023

[4] [4]

Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

arXiv 2024

[5] [5]

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

arXiv 2023

[6] [6]

Yi Ding, Bolian Li, and Ruqi Zhang. 2024. Eta: Evaluating then aligning safety of vision language models at inference time. arXiv preprint arXiv:2410.06625

arXiv 2024

[7] [7]

Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936

arXiv 2022

[8] [8]

Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, and Jiawei Zhou. 2024. From uncertainty to trust: Enhancing reliability in vision-language models with uncertainty-guided dropout decoding. arXiv preprint arXiv:2412.06474

arXiv 2024