Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CATCH is a training-free decoding-time method that reduces hallucinations in large vision-language models by contrastively comparing SAM-decoupled complementary image views against a text-only baseline.

desk verdict CATCH is a novel training-free decoding method with big benchmark gains, but the reported numbers cannot distinguish visual correction from a conservative response shift, so the mechanistic claim is unproven. read the letter →

arxiv 2411.12713 v1 pith:5CYBEHPN submitted 2024-11-19 cs.CV cs.AI

Zhehan Kan , Ce Zhang , Zihan Liao , Yapeng Tian , Wenming Yang , Junyuan Xiao , Xu Li , Dongmei Jiang
show 2 more authors
Yaowei Wang Qingmin Liao
This is my paper · ORCID
classification cs.CVcs.AI
keywords hallucinationmitigationlargevision-languagemodelscontrastivedecodinginformationbottleneckvisualdefectSegmentAnythingModel(SAM)decoding-timeinterventionobject
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models (LVLMs) often invent objects or attributes that are not in the image, and the authors argue the root cause is a 'visual defect': the whole image overloads the model's vision-language alignment, compressing away fine-grained features and pushing the decoder onto linguistic priors. CATCH intervenes at decoding time, without any retraining or task-specific data, by segmenting the image with SAM into two complementary views (foreground-object view and residual view), running a text-only version of the same prompt, and using Jensen-Shannon divergence to decide which view carries the evidence for the next token. Depending on that comparison, it either contrastively subtracts the original (hallucination-rich) distribution or adds the decoupled view to restore diversity. On POPE, MME, and CHAIR benchmarks, CATCH reports consistent gains over two base models, including up to 8.07 accuracy points on POPE, 16% total-score improvement on MME, and a 45.8% relative reduction on CHAIR hallucination metrics. The importance is that a decoder-side correction, not more data or a bigger visual encoder, may be sufficient to curb a substantial share of LVLM hallucination.

What carries the argument

The mechanics rest on the complementarity of two views plus a divergence-based selection rule. CVD builds $z_d$ and $z_r$ so that $z_d + z_r = v$ in pixel space, each carrying a simplified subset of the visual features; the non-visual input $z_n$ serves as a language-prior reference. At each token step, the model runs multiple forward passes (original $v$, the two views, and $z_n$), and NVS selects the view with the larger Jensen-Shannon divergence from $z_n$ as the decoupled image $z$ (Equation 6). ATCD then applies Equation (8): if the chosen view's divergence from the non-visual input is greater than the original image's divergence, it samples from $\mathrm{softmax}[\alpha\,\mathrm{logit}(y|z,x) - \mathrm{logit}(y|v,x)]$ to subtract hallucinated concepts; otherwise it samples from $\mathrm{softmax}[\beta\,\mathrm{logit}(y|v,x) + \mathrm{logit}(y|z,x)]$ to add diversity. The whole mechanism is training-free and is reported to be robust across two LVLMs and three benchmarks.

What would settle it

Construct or find an image where a large, salient object (for example, a bright red ball) is present but irrelevant, while a small, easily missed object (for example, a key on the floor) is the ground-truth object the caption must mention for the next token. Segment the image so the ball forms the dual image and the key is in the residual; if the ball's distribution is farther from the text-only prior than the residual's, NVS selects the ball, and CATCH will suppress or fail to boost the correct token. A statistical version: on 1,000 random MSCOCO images, mask the ground-truth region and measure whether the view with the larger Jensen-Shannon divergence from the non-visual input gives a higher probability to the ground-truth token than the other view does; a null or negative correlation would refute the load-bearing selection rule.

Watch

Extended reading notes

Core claim

The central claim is that hallucinations in LVLMs emerge from a vision-language alignment bottleneck rather than only from statistical or language priors. The paper shows that when the visual input is decoupled into simplified complementary parts, the probability of the ground-truth token rises while the hallucinated token's probability falls, implying that extraneous visual features actively interfere with fine-grained perception. CATCH formalizes this with three components: Complementary Visual Decoupling (CVD) uses SAM to split the image into a dual image $z_d$ (top $M$ objects) and a residual image $z_r$ (the rest), and introduces a non-visual input $z_n$ that contains only the text prompt; Non-Visual Screening (NVS) computes $D_{\mathrm{JS}}(z_d,z_n)$ and $D_{\mathrm{JS}}(z_r,z_n)$ and selects whichever view is more divergent as the decoupled image $z$; Adaptive Token-level Contrastive Decoding (ATCD) then compares $D_{\mathrm{JS}}(z,z_n)$ with $D_{\mathrm{JS}}(v,z_n)$ to choose between subtractive and additive contrastive decoding. The authors report that this procedure reduces hallucination across object-existence (POPE), attribute (MME), and open-ended captioning (CHAIR) benchmarks on both LLaVA-1.5 and InstructBLIP, and that it delays cumulative hallucinations from roughly 40% of the generated sequence to about 80%.

Load-bearing premise

At each decoding step, the complementary image view whose output distribution is farthest from the text-only, non-visual input is assumed to be the one containing the visual evidence most relevant to the next token; if that ranking is wrong, the contrastive subtraction or addition targets the wrong view and can reinforce the hallucination.

Editorial extensions

If this is right

  • Without any training or data collection, CATCH can be applied to an already deployed LVLM to reduce object hallucinations in both short-answer and open-ended generation.
  • The same mechanism—splitting the image into complementary views and anchoring on a text-only prior—generalizes to different visual question-answering tasks, as the paper shows on POPE, MME, and CHAIR.
  • By shifting the onset of cumulative hallucinations from ~40% to ~80% of the sequence, CATCH reduces error propagation in long captions, making it useful for tasks that require detailed descriptions.
  • Because CATCH improves not only existence but also count, position, and color hallucinations, the mechanism addresses attribute-level fabrication, not just object presence.
  • The method is model-agnostic across the two decoders tested (LLaVA-1.5 and InstructBLIP), suggesting it targets a structural failure mode of LVLM decoding rather than one specific training scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's selection rule equates 'more divergent from a text-only prior' with 'more relevant to the next token,' but that equivalence is not proven for all images; a natural stress test is an image with a salient but irrelevant object that the language prior favors, where the rule could pick the wrong view and CATCH would amplify the hallucination.
  • The information-bottleneck framing implies that any operation that increases the mutual information between the visual evidence and the next token at decoding time—not just SAM-based decoupling—should produce a similar effect; testing CATCH against simple crops, magnification, or attention-weighted masks would isolate the role of the complementary split.
  • The method requires several forward passes per token, so its practical cost is not reported; if the selection rule proves correct, a lightweight approximation of the JSD ranking could make the approach faster without losing the core contrastive mechanism.
  • The paper analyzes cumulative hallucinations by measuring when the output distribution collapses to the non-visual distribution; a stronger test would use oracle ground-truth tokens to verify that the selected view actually lifts the probability of the correct next token at that position.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CATCH, a training-free decoding-time method to reduce hallucinations in large vision-language models (LVLMs). It first uses SAM to decouple the input image into two complementary views (dual and residual), introduces a non-visual text-only input, and at each token step computes Jensen-Shannon divergences between the output distributions of the non-visual input and the original/decoupled views. Based on these divergences, it selects one decoupled view and either contrastively subtracts its logits from the original (hallucination-existence case) or adds them to the original (diversity-insufficient case). CATCH is evaluated on POPE, MME Hallucination, and CHAIR using LLaVA-1.5 and InstructBLIP baselines, reporting consistent gains over the baselines and over prior contrastive-decoding methods. The paper also presents analyses (Figs. 3 and 5) aimed at motivating the visual-defect/information-bottleneck narrative and the claim that CATCH delays cumulative hallucinations.

Significance. If the central claim holds, CATCH would be a valuable, generally applicable, training-free intervention that reduces object hallucination in LVLMs without requiring task-specific data or fine-tuning. The paper has real strengths: it evaluates on two model families across three standard benchmarks, reports internally consistent numbers, and includes an interesting empirical observation (Fig. 1) that decoupling the image changes ground-truth versus hallucinated token probabilities. The method is simple enough to reproduce. However, the causal narrative is not yet established by the evidence: the benchmark results do not separate improvement due to genuine visual correction from a conservative response bias, and the information-bottleneck formulation is presented as a motivation rather than derived or verified. Because the manuscript's core claim rests on this distinction, the significance of the contribution currently depends on additional analysis rather than on the reported numbers alone.

major comments (4)
  1. [§3.4, Eq. (8)] There is an inconsistency between the selection rule in Eq. (6) and the decoding rule in Eq. (8). Eq. (6) selects the decoupled image z from {z_d, z_r} based on comparing d(z_d, z_n) and d(z_r, z_n); when z_r is selected, the text in §3.4 says the 'decoupled image' distribution is used for both contrastive subtraction and contrastive enhancement. However, Eq. (8) conditions the first branch on d(z_d, z_n) ≥ d(v, z_n) rather than on d(z, z_n) ≥ d(v, z_n), and the second branch uses logit_θ(y | z_d, x) even when Eq. (6) selected z_r. As written, the algorithm is ill-defined for the case z = z_r and the second branch never uses the selected decoupled image. Please correct Eq. (8) to use the selected z in both branches and to compare d(z, z_n) with d(v, z_n), or explicitly state if the design intentionally always uses z_d in the enhancement branch and justify that choice.
  2. [§3.2, Eq. (2)] The information bottleneck objective in Eq. (2), min[I(v; z) − I(z; y)], is never optimized in the paper. The text states that SAM is used to 'optimize this objective,' but no derivation is given showing that selecting the top M objects by area as z_d (and masking them to form z_r) minimizes I(v; z) − I(z; y), nor is any approximation or justification provided. As it stands, the IB formulation is a post hoc narrative rather than a derivation of CVD. Please either derive the connection between Eq. (2) and the SAM-based decoupling, or reframe the contribution to present CVD as a heuristic motivated by (but not derived from) the information bottleneck principle.
  3. [§2, Table 1 and Fig. 4] The reported improvements are not sufficient to distinguish 'visual correction' from a conservative-response artifact. On POPE, CATCH consistently increases precision while decreasing recall; for example, LLaVA on GQA-adversarial shows precision rising from 61.68 to 71.72 while recall falls from 93.13 to 91.47, and LLaVA on A-OKVQA-popular shows precision rising from 70.51 to 80.90 while recall falls from 91.67 to 90.67. The paper itself attributes the precision gain to a 'lower yes response ratio' (§2). On CHAIR, the metrics reward fewer object mentions, so a model that omits uncertain objects can improve regardless of visual perception. To support the claim that CATCH specifically restores visual evidence, please add controls: report object-level recall of ground-truth objects on CHAIR, caption length, and a POPE analysis that either matches the baseline's yes-rate through rejection sampling or compares against a constant-'no' baseline. Without such controls, the benchmark results are also compatible with a strategy that simply makes the model more conservative.
  4. [§3.3, Eq. (6)] The selection rule in Eq. (6) assumes that the view with the larger Jensen-Shannon divergence from the non-visual input is the one whose visual content is most relevant to the next token. This is a load-bearing premise, because selecting the wrong view would make the contrastive subtraction or addition amplify the wrong signal. The supporting evidence in Fig. 3 uses a masking procedure ('key visual features') that is not described in the same terms as the actual CVD (which selects top M objects by area), and it is only demonstrated on a random subset of MSCOCO, not on the heterogeneous images in POPE, MME, and CHAIR. Please either provide a per-benchmark validation of the selection rule or add an analysis (e.g., a token-prediction probe) showing that the larger-JSD view is indeed the one that improves next-token probability for ground-truth tokens.
minor comments (5)
  1. [§2, CHAIR definitions] The definitions of CHAIR_S and CHAIR_I in the text are incorrect. The paper describes CHAIR_S as 'the proportion of hallucinated sentences relative to all sentences,' but CHAIR_S is the fraction of hallucinated object instances among all mentioned object instances; similarly, CHAIR_I is the fraction of captions containing any hallucinated object, not 'the proportion of hallucinated objects relative to all generated objects.' Please correct these descriptions.
  2. [§3.4, Eq. (8)] The notation p_θ(y_t | z, v, x) on the left-hand side of Eq. (8) does not match the right-hand side, which conditions on z in the first branch and on z_d in the second branch. Please make the conditioning consistent after fixing the branch logic.
  3. [Throughout] There are several typographical errors: 'Constrastive' in the Keywords, 'Implementtation' in §3.5, 'showm' in §2, and a duplicated affiliation number 4 in the author list. These should be corrected in the final version.
  4. [Fig. 4] The figure caption states that five random seeds were averaged, but no error bars or per-seed variance are shown. Please include error bars or report the variance, since the MME subtasks contain only 60 questions each and are thus susceptible to seed noise.
  5. [§3.5] The hyperparameters α, β, and the object-count ratio M/N are fixed by default (α = 1.2, β = 3, M = N * 0.05), but no sensitivity analysis is reported. Since the method's performance is likely sensitive to these values, please report an ablation over at least α and β, and over M/N for a representative subset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the method's derivation is self-contained and its claims are evaluated against external benchmarks.

full rationale

The paper's derivation chain does not reduce to its inputs. Complementary Visual Decoupling (CVD) segments the image into complementary parts (Eq. 4), Non-Visual Screening (NVS) selects the part with larger Jensen-Shannon divergence to the non-visual input (Eq. 6), and Adaptive Token-level Contrastive Decoding (ATCD) contrastively subtracts or adds the decoupled logits (Eq. 8). These quantities are defined from SAM segmentation and model output distributions, not from the benchmark numbers reported later. The 'visual defect' narrative is motivated by Fig. 1 and Fig. 3 demonstrations of token probabilities under decoupling, rather than by fitting downstream metrics. The method is evaluated on external benchmarks (POPE, MME, CHAIR) against LLaVA-1.5 and InstructBLIP baselines, and no load-bearing self-citation appears in the references; no prior result by the same authors is invoked to force the choice of Eq. 6 or Eq. 8. Hyperparameters alpha=1.2 and beta=3 are stated as defaults rather than fitted from the reported benchmarks, so there is no exhibited reduction of a prediction to a fitted input. Possible concerns about conservative-response artifacts, recall drops on POPE, or CHAIR rewarding fewer object mentions are benchmark-validity and interpretation issues, not circularity of the derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several domain assumptions that are plausible but not proven: the visual-defect etiology, the use of the non-visual distribution as a language-prior reference, the ranking of JSDs as a proxy for visual grounding, and the correspondence between the Information Bottleneck objective and SAM decoupling. The free hyperparameters α, β, and M are hand-set without ablations. No new physical entities are introduced; 'visual defect' is a named hypothesis with only in-paper correlational support. These entries are the cost the reader pays before the benchmark gains in Tables 1 and 4 can be interpreted.

free parameters (3)
  • alpha (α) = 1.2
    Amplification factor in the contrastive subtraction branch of Eq (8); fixed in §3.5 without ablation or a stated selection procedure, so it may be tuned on the evaluation benchmarks.
  • beta (β) = 3
    Amplification factor in the diversity-enhancement branch of Eq (8); fixed in §3.5 without ablation, so may be tuned on the evaluation benchmarks.
  • object ratio M/N = 0.05 (M = N * 0.05)
    Number of SAM-segmented objects selected for the exposed image; fixed by default in §3.5. The choice of 5% of detected objects is not derived from the Information Bottleneck objective and may be tuned for performance.
assumptions (5)
  • domain assumption Hallucinations in LVLMs arise primarily from a 'visual defect' caused by an information bottleneck in vision-language alignment.
    This is the paper's central etiological claim, asserted in §1 and Discussion; it is supported by correlational evidence (masking features changes token probabilities) but not by a causal or mathematical proof.
  • domain assumption At each generation step, the decoupled image with larger Jensen-Shannon divergence from the non-visual input is the one that preserves key visual features.
    Used in §3.3 Eq (5)-(6) to select between zd and zr; this mapping from divergence ranking to visual grounding is assumed, not derived.
  • domain assumption The non-visual input distribution pθ(yt|zn,x,y<t) faithfully represents language-prior behavior.
    NVS and ATCD compare all images against this reference (§3.3-3.4); if the text-only distribution is not a stable reference, the selection and adaptive rule lose their justification.
  • domain assumption SAM segmentation into top-M objects by area produces complementary visual parts that simplify the image without discarding task-relevant information.
    CVD relies on this in §3.2 Eqs (3)-(4); no guarantee that the highest-area objects are the relevant ones for the next token.
  • ad hoc to paper Information Bottleneck objective min[I(v;z)-I(z;y)] can be implemented by SAM-based decoupling.
    The paper states the IB objective in Eq (2) but never optimizes it or connects it to the SAM heuristic; this is an asserted correspondence, not a derivation.
invented entities (1)
  • visual defect (named causal mechanism)
    purpose: Posits that hallucinations arise from an information bottleneck when mapping high-dimensional visual features into the aligned vision-language space, not from language or statistical bias alone.
    Introduced in §1 and used to motivate CVD. The in-paper evidence (Figs. 1 and 3) is correlational and only indirectly supports the mechanism; no external falsifiable prediction is given that would distinguish 'visual defect' from the previously known biases it claims to supersede.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs." pith.science (2026). https://pith.science/paper/5CYBEHPN

@misc{pith2026241112713,
  author       = {Pith},
  title        = {Pith review of: CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CYBEHPN}},
  note         = {Machine review of arXiv:2411.12713}
}
read the original abstract

Large Vision-Language Model (LVLM) systems have demonstrated impressive vision-language reasoning capabilities but suffer from pervasive and severe hallucination issues, posing significant risks in critical domains such as healthcare and autonomous systems. Despite previous efforts to mitigate hallucinations, a persistent issue remains: visual defect from vision-language misalignment, creating a bottleneck in visual processing capacity. To address this challenge, we develop Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs (CATCH), based on the Information Bottleneck theory. CATCH introduces Complementary Visual Decoupling (CVD) for visual information separation, Non-Visual Screening (NVS) for hallucination detection, and Adaptive Token-level Contrastive Decoding (ATCD) for hallucination mitigation. CATCH addresses issues related to visual defects that cause diminished fine-grained feature perception and cumulative hallucinations in open-ended scenarios. It is applicable to various visual question-answering tasks without requiring any specific data or prior knowledge, and generalizes robustly to new tasks without additional training, opening new possibilities for advancing LVLM in various challenging applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.

  2. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

29 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    In: Proceedings of the IEEE International Conference on Computer Vision, pp

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433 (2015)

    work page 2015

  2. [2]

    Nature communications 12(1), 5721 (2021)

    Gupta, A., Savarese, S., Ganguli, S., Fei-Fei, L.: Embodied intelligence via learning and evolution. Nature communications 12(1), 5721 (2021)

    2021

  3. [3]

    arXiv preprint arXiv:2406.19280 (2024)

    Chen, J., Ouyang, R., Gao, A., Chen, S., Chen, G.H., Wang, X., Zhang, R., Cai, Z., Ji, K., Yu, G., et al.: Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280 (2024)

    arXiv 2024

  4. [4]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp

    Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al.: A survey on multimodal large language models for autonomous driving. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958–979 (2024)

    work page 2024

  5. [5]

    arXiv preprint arXiv:2404.18930 (2024)

    Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., Shou, M.Z.: Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024)

    arXiv 2024

  6. [6]

    Advances in neural information processing systems 35, 23716–23736 (2022)

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, 23716–23736 (2022)

    work page 2022

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., Zhuang, Y.: Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12944–12953 (2024)

    work page 2024

  8. [8]

    In: International Conference on Multimedia Modeling, pp

    Wang, L., He, J., Li, S., Liu, N., Lim, E.-P.: Mitigating fine-grained halluci- nation by fine-tuning large vision-language models with caption rewrites. In: International Conference on Multimedia Modeling, pp. 32–45 (2024). Springer

    work page 2024

Show all 29 references
  1. [9]

    arXiv preprint arXiv:2402.14545 (2024)

    Yue, Z., Zhang, L., Jin, Q.: Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14545 (2024)

    2024 arXiv

  2. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning 15 for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  3. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306 (2024)

    2024

  4. [12]

    Advances in neural information processing systems 36 (2024)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)

    2024

  5. [13]

    arXiv e-prints, 2310 (2023)

    Zhai, B., Yang, S., Xu, C., Shen, S., Keutzer, K., Li, M.: Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints, 2310 (2023)

    2023

  6. [14]

    arXiv preprint arXiv:2401.03105 (2024)

    He, X., Wei, L., Xie, L., Tian, Q.: Incorporating visual experts to resolve the information loss in multimodal large language models. arXiv preprint arXiv:2401.03105 (2024)

    2024 arXiv

  7. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Jain, J., Yang, J., Shi, H.: Vcoder: Versatile vision encoders for multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992–28002 (2024)

    2024

  8. [16]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578 (2024)

    2024

  9. [17]

    arXiv preprint arXiv:2403.00425 (2024)

    Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., Zhou, J.: Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425 (2024)

    2024 arXiv

  10. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., Bing, L.: Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872–13882 (2024)

    2024

  11. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swami- nathan, A., Soatto, S.: Multi-modal hallucination control by visual information grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14303–14312 (2024)

    2024

  12. [20]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.-Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4026 (2023)

    2023

  13. [21]

    In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 16 2014, Proceedings, Part V 13, pp

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 16 2014, Proceedings, Part V 13, pp. 7...

    2014

  14. [22]

    https://arxiv.org/abs/2305.06500

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023). https://arxiv.org/abs/2305.06500

    2023 arXiv

  15. [23]

    See https://vicuna

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April

  16. [24]

    arXiv preprint arXiv:2305.10355 (2023)

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.-R.: Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  17. [25]

    arXiv preprint arXiv:1809.02156 (2018)

    Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T., Saenko, K.: Object hallucination in image captioning. arXiv preprint arXiv:1809.02156 (2018)

    2018 arXiv

  18. [26]

    arXiv preprint arXiv:2306.13394 (2023)

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023)

    2023 arXiv

  19. [27]

    In: European Conference on Computer Vision, pp

    Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-okvqa: A benchmark for visual question answering using world knowledge. In: European Conference on Computer Vision, pp. 146–162 (2022). Springer

    2022

  20. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual rea- soning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700–6709 (2019)

    2019

  21. [29]

    In: International Conference on Machine Learning, pp

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International Conference on Machine Learning, pp. 19730–19742 (2023). PMLR 17

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.