Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing a pre-trained vision model's visual encoder with an EAGLE-tuned encoder reduces hallucinated object descriptions in instruction-tuned vision-language models, without retraining their language components.

desk verdict Worth reading: EAGLE is a genuinely useful drop-in visual encoder for reducing hallucinations, but the paper's 'universal no-training improvement' claim is contradicted by its own tables and needs honest rescoping. read the letter →

arxiv 2501.02699 v1 pith:FMLSEVMD submitted 2025-01-06 cs.CV cs.AI

Andrés Villa , Juan León Alcázar , Motasem Alfarra , Vladimir Araujo , Alvaro Soto , Bernard Ghanem This is my paper · ORCID
classification cs.CVcs.AI
keywords hallucinationvisualgroundingvision-languagemodelsinstructiontuningcontrastivelearningencoderfine-tuninginstancesegmentationmultimodalbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large share of object hallucination in instruction-tuned vision-language models (IT-VLMs) comes from the visual encoder: the patch-level tokens feeding the language model are poorly aligned with object semantics, even though the global CLS token is well aligned. EAGLE is a post-pretraining step that fixes this by tuning a CLIP-style ViT so that the average of patch tokens inside a segmentation mask matches the frozen text embedding of that object's class name. The tuned encoder can then replace the default visual encoder in a trained IT-VLM with no retraining of the adapter or language model. Across six IT-VLMs and three hallucination benchmarks (POPE, MMVP, MERLIM), the swap reduces hallucination, with the largest relative gains on MMVP (11.2%) and MERLIM (6.3%).

What carries the argument

The load-bearing object is the masked average pooling operator $\phi(I,m)$: it zeroes out every patch token outside a binary segmentation mask and averages the rest, producing a single local visual vector for an object. EAGLE's loss then drives that vector toward the frozen text embedding of the object's class name (with the prompt prefix "This is an image of <class>") using two terms: a contrastive instance-level loss and a sigmoid cross-entropy loss that allows multiple masks to share a class. Because the CLS token is never trained, the global descriptor is preserved, and GaLore's low-rank gradient projection limits distribution shift during full-parameter fine-tuning. This mechanism is what lets the improved encoder be dropped into an existing IT-VLM with no adapter or LLM retraining.

What would settle it

Run EAGLE on OpenImages but replace each object mask with a random crop of the same size and the same class label; if the reported POPE, MMVP, and MERLIM gains are reproduced, then the claim that object-grounded alignment is the responsible mechanism would be undermined.

Watch

Extended reading notes

Core claim

The central discovery is that the feature sequence—the patch tokens an IT-VLM actually consumes—lags far behind the CLS token in language alignment, and that lag can be closed with a reformulation of contrastive pre-training. EAGLE samples one segmented object from an image, pools the ViT's patch embeddings over the object's mask, and aligns the pooled vector with the frozen text embedding of its class name. A per-instance contrastive term plus a per-class sigmoid cross-entropy term update only the visual encoder, while GaLore keeps the global feature space from drifting. The result is an encoder whose sequence embeddings carry fine-grained object information, verified by a jump in zero-shot accuracy when using sequence features on ImageNet (e.g., from 0.7% to 50.98% for the OpenAI ViT-L-14-336) and by reductions in false positives on MS-COCO, and this transfers to reduced hallucination in six IT-VLMs without any instructional training.

Load-bearing premise

The argument assumes that patch-level tokens of a pre-trained ViT, after masked average pooling over a segmentation mask, can be meaningfully aligned with the frozen text embedding of the object's class, and that this local alignment transfers to the feature sequence in a way that reduces hallucination.

Editorial extensions

If this is right

  • A trained IT-VLM can be improved by only swapping its visual encoder, with no adapter tuning, no LLM fine-tuning, and no added inference cost.
  • The patch-level feature sequence becomes a usable zero-shot classifier on its own: for the OpenAI encoder, ImageNet sequence accuracy rises from 0.7% to 50.98% while the CLS token drops only 5.09%.
  • BLIP-2 and InstructBLIP benefit more than LLaVA-v1.5, which the authors attribute to their adapter-only tuning making them more compatible with a fixed encoder.
  • EAGLE improves both MERLIM prompts without changing their relative ranking, meaning it reduces hallucinations without correcting language-side prompt bias.
  • Training the CLS token would lower false positives further but sacrifices zero-shot accuracy, so EAGLE leaves that token untouched.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not state this, but their results suggest the gains come from making local patch tokens carry nameable object information, so alternative local-alignment supervisions—caption-derived masks, self-supervised object discovery, or point prompts—might reproduce similar reductions.
  • An untested corollary: EAGLE-style encoders should also improve tasks that require localized semantics, such as referring-expression grounding, open-vocabulary detection, or questions about small objects in dense scenes.
  • A natural stress-test would combine EAGLE with a language-side hallucination regularizer, since the two target different components and their benefits may be additive.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes EAGLE, a post-pretraining method for CLIP-style vision encoders. EAGLE fine-tunes the ViT using masked-average-pooled patch tokens aligned with text embeddings of object class names from OpenImages V7, using a two-part loss (instance-level contrastive plus class-level cross-entropy) and GaLore to limit feature drift. The resulting encoder is then swapped into existing instructional VLMs without retraining their LLM or adapter. The paper reports evaluations on POPE, MMVP, and MERLIM for six IT-VLMs, plus a retrained LLaVA-1.5 variant, and claims consistent hallucination reduction and large aggregate improvements.

Significance. The idea of improving visual grounding at the encoder level in a plug-and-play way is timely and could complement decoder-side hallucination mitigation methods. The paper includes ablations (Table 6) and shows gains on most models and benchmarks, and the method is architecture-agnostic. However, the abstract's universal claim is contradicted by the paper's own data (LLaVA-1.5 POPE degrades), and the headline improvement percentages are not reproducible from the tables. The lack of uncertainty quantification and of comparisons to existing hallucination-mitigation baselines further limits the strength of the claims.

major comments (5)
  1. [Table 4 and Section 4.2] The caption of Table 4 states that "without any tuning or alignment to the LLM or its fusion model, EAGLE consistently enhances performance across all of the IT-VLMS for all metrics in both benchmarks." This is not supported by the table: for LLaVA-v1.5, POPE Accuracy drops from 69.33% to 67.53%, and F1, Precision, and Recall also drop. Additionally, InstructBLIP (Vicuna-13B) MMVP is unchanged at 24.67%, and InstructBLIP (FlanT5xl) POPE Recall drops from 98.0% to 97.93%. The universal no-training claim must be weakened and the failure cases analyzed.
  2. [Section 4.2 and Contributions] The paper reports "2.3% absolute improvement" on MMVP and "2.73% absolute improvement" on MERLIM, and the Contributions claim "11.2% relative improvement in MMVP and 6.3% relative improvement in MERLIM." Averaging the per-model deltas in Table 4 for the six plug-and-play models gives an MMVP gain of 1.67% absolute (2.19% if LLaVA-1.5* is included); for Table 5 the MERLIM average is 2.33% absolute (2.42% with LLaVA-1.5*). The reported numbers do not match any transparent computation from the tables. Please specify exactly how the aggregates are computed and whether they include the retrained LLaVA-1.5* row.
  3. [Abstract and Section 4.2 (LLaVA A Training)] The Abstract and Contribution (i) state that EAGLE reduces hallucinations "without additional instructional training" and quote the aggregate percentages. However, LLaVA-1.5* requires second-stage instructional tuning, as described in Section 4.2. The plug-and-play results should be cleanly separated from the retrained results in the abstract and in the contribution claims, so that the reader is not led to believe that the headline numbers apply uniformly to the no-training setting.
  4. [Section 4 in general] The word "significant" is used repeatedly without statistical tests or error bars. Many gains are small in absolute terms (e.g., MiniGPT-4 POPE +0.54, LLaVA-1.5 MERLIM +0.51), and no comparison is made to existing hallucination-mitigation methods such as VCD, OPERA, or other visual-encoder enhancement methods (e.g., BRAVE). At minimum, the authors should report confidence intervals or multiple seeds, and compare against at least one strong baseline on the same benchmarks before claiming significance.
  5. [Equations (2) and (3)] In Eq. (2), d_j = 1 - sigma(phi(I,m) - l(c_j)): if sigma is the element-wise sigmoid and both phi(I,m) and l(c_j) are vectors, then d_j is a vector, making the binary cross-entropy in Eq. (3) ill-defined. Moreover, c_j is used both as the class-name prompt in Eq. (1) and as the target label in Eq. (3). Please define the exact tensor shapes and the form of c_j in the loss. This is central to understanding the training objective.
minor comments (6)
  1. [Table 2] The EAGLE EVA-01 CLS row prints a delta of (-1.49%), but the baseline average is 25.18% and the EAGLE average is 26.67%, which is an improvement of +1.49%; the sign is inconsistent with the text in Section 4.1.
  2. [Section 4, Implementation Details] The text says "We train until convergence of the Lm loss," but L_m is never defined; presumably the total loss L in Eq. (4) is meant.
  3. [Table 2] The MMVP task names contain garbled glyphs (e.g., "♂search", "/sync", "/sort-numeric-up", "♂¶ap-pin"); please use plain text names or a legend.
  4. [Section 4.2] The sentence "EAGLE enhances the performance of all evaluated IT-VLMs across the three selected benchmarks" is contradicted by the POPE result for LLaVA-1.5. Please revise this sentence and the corresponding claim in the table caption.
  5. [Section 3.1] The discussion of contrastive learning states that the standard contrastive loss is incompatible because batches may contain multiple masks of the same class, but the exact definition of negatives in Lins is not given. Please clarify whether Lins uses in-batch negatives and how repeated classes are handled.
  6. [References/Section 4.2] MERLIM [37] is co-authored by two of the current authors (Villa and Ghanem), but the paper cites it as an external benchmark without noting this overlap. For transparency, please add a statement about the self-evaluation.

Circularity Check

1 steps flagged · score 2.0 of 10

EAGLE's derivation is not circular: its training loss is independent of the evaluation benchmarks. The only circularity-adjacent issue is evaluation on the authors' own MERLIM benchmark, which is minor and not load-bearing.

  1. other [Section 4.2, Hallucination Benchmarks (MERLIM paragraph); Table 5]
    "In MERLIM [37], we evaluate EAGLE using a subset of original and edited images... MERLIM incorporates open-ended questions with equivalent meanings, to inquire about all the objects present in the image."

    MERLIM [37] is authored by Andrés Villa, Juan Carlos León Alcázar, Alvaro Soto, and Bernard Ghanem, three of whom are authors of the present paper. EAGLE's loss (Eqs. 1-4) directly optimizes alignment between masked patch features and the text embedding of the object class ('This is an image of <mask class>'), while MERLIM scores how completely models list the objects present in an image. The MERLIM leg of the reported improvement therefore re-measures the exact object-name alignment capability trained, rather than an independent external generalization. This is a minor self-citation rather than load-bearing circularity, because the central claim also rests on external POPE and MMVP results.

full rationale

EAGLE's derivation chain is not circular in the equation-level sense. The training loss (Eqs. 1-4) supervises masked-average-pooled patch features with OpenImages V7 class labels and the corresponding CLIP text embeddings; the reported hallucination reductions are measured on POPE, MMVP, and MERLIM, none of which is used to fit a parameter or select the model (the paper states hyperparameters are fixed and training runs to convergence). The central claim therefore has independent empirical content. The only self-citation of note is MERLIM [37], authored by three of the same authors as this paper. Because MERLIM asks models to enumerate objects in an image and EAGLE explicitly trains patch tokens to align with object class names, that particular benchmark is close to the trained capability and is not an independent external check. However, this is a minor, non-load-bearing self-citation: POPE and MMVP are external, and the paper's headline aggregate numbers are also supported by those benchmarks. Separately, the paper's claim that EAGLE 'consistently enhances performance across all of the IT-VLMS for all metrics' is overstated—Table 4 shows LLaVA-1.5 POPE accuracy dropping from 69.33% to 67.53%—but that is an empirical-correctness issue, not circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

EAGLE introduces no new entities; it fine-tunes an existing ViT using existing segmentation data and a frozen text encoder. The essential assumptions are that patch tokens localize objects, that the frozen text encoder is a valid teacher, and that OpenImages labels transfer to hallucination benchmarks. The free parameters are standard training hyperparameters plus an under-specified mask-overlap rule.

free parameters (7)
  • GaLore rank = 128
    Set to 128 for both ViTs (Section 4 Implementation Details and Table 8). Controls the capacity of low-rank gradient projection and affects how much global feature distribution is preserved while fine-tuning local features.
  • Learning rate = 4e-6
    Chosen by hand for both ViTs (Table 8).
  • Batch size = 512
    Set to 512 (Table 8).
  • Warmup steps = 25000
    Set to 25000 (Table 8).
  • GaLore scale = 0.25
    Set to 0.25 (Table 8).
  • Mask overlap threshold
    The masked average pooling phi zeros tokens 'if they do not overlap with the mask' but the overlap criterion (e.g., center point, IoU) is not defined (Section 3.1). This choice affects which tokens represent the object.
  • Class resampling distribution
    Masks are resampled so uncommon classes are more likely to be chosen; the target distribution is not specified numerically (Section 3.1).
assumptions (5)
  • ad hoc to paper Patch-level tokens of a CLIP-style ViT encode spatially localized object information that can be extracted by masked average pooling.
    The entire loss function assumes phi(I,m) is a meaningful object-level feature; if patch tokens are not spatially localized, the alignment signal is noise. It is not verified beyond the empirical results.
  • domain assumption The frozen language encoder l from the original VLM provides a stable, semantically meaningful embedding space for object class names.
    EAGLE does not update l; the contrastive target l(c_j) is taken as ground truth, so any language-side bias transfers to the visual encoder.
  • domain assumption OpenImages V7 instance segmentation masks and their 350 class labels are sufficient supervision to improve fine-grained visual grounding for downstream IT-VLM benchmarks.
    The method uses only OpenImages V7 with 350 classes, three orders of magnitude fewer images than VLM pretraining; the paper relies on this transferring.
  • ad hoc to paper Unit weighting of Lce and Lins in L = Lce + Lins is appropriate.
    No hyperparameter search is reported for the weighting, so the equal combination is an untested assumption, although the authors claim no extra hyperparameters.
  • domain assumption GaLore low-rank updates preserve global feature distribution (zero-shot accuracy) while allowing local fine-grained changes.
    The paper relies on this to keep the encoder plug-in compatible; it is justified empirically in Table 6 but not theoretically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models." pith.science (2026). https://pith.science/paper/FMLSEVMD

@misc{pith2026250102699,
  author       = {Pith},
  title        = {Pith review of: EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FMLSEVMD}},
  note         = {Machine review of arXiv:2501.02699}
}
read the original abstract

Large language models and vision transformers have demonstrated impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE, is fully agnostic to the LLM or fusion module and works as a post-pretraining approach that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without additional instructional training. As a result, EAGLE achieves a significant reduction in hallucinations across multiple challenging benchmarks and tasks.

Figures

Figures reproduced from arXiv: 2501.02699 by the authors.

Figure 1
Figure 1. EAGLE visual encoders reduce hallucinations in IT-VLMs. We present three example scenarios, each featuring a question about an image input to a specific IT-VLM with its original visual encoder (left box in pink) and the corresponding EAGLE-tuned visual encoder (right box in orange). “IB7”, “IB13” and “IBT5” refer to InstructBLIP with Vicuna7B, Vicuna13B, and FlanT5xl, respectively. EAGLE substantially reduces halluc… view at source ↗
Figure 2
Figure 2. Overview of the EAGLE method. EAGLE reduces the hallucinations in IT-VLMs by improving the grounding of the image encoder. In the post-pretraining phase (Left), EAGLE enhances fine-grained visual representations by employing a masked average pooling (in red dashed lines). This method selects embeddings within the feature sequence corresponding to a specific object and computes an averaged representation. Subsequentl… view at source ↗
Figure 3
Figure 3. Visual Examples Demonstrating EAGLE’s Effectiveness in Reducing Hallucinations in IT-VLMs. We present three addi￾tional illustrative scenarios, each featuring a question about an image processed by a specific IT-VLM using its original visual encoder (left, pink box) and the corresponding EAGLE-tuned visual encoder (right, orange box). The models evaluated include “LLA” (LLaVA-1.5), “LLA*” (LLaVA-1.5*), and “BT5” (BL… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visual Examples of EAGLE Enhancing Visual Grounding of VLMs. We assess the ability of two VLMs, EVA-01-CLIP￾g-14 and OpenAI CLIP-L-14-336, and their corresponding EAGLE-tuned versions (blue boxes) to embed fine-grained visual details in the sequence features, using the…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Monitoring attention entropy and image-output correlation during decoding, then applying targeted contrastive corrections, reduces hallucination in multimodal LLMs without retraining.

  2. CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation

    cs.CV 2026-01 reject novelty 5.0 of 10

    CURE's curriculum-guided multi-task training improves bounding-box grounding for chest X-ray report generation, but its claimed hallucination reduction is not confirmed by the paper's full evaluation.

  3. Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Injecting an average-pooled visual embedding into every text token improves hallucination-benchmark scores of Video-LLaVA by small single-digit amounts.

  4. MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An instruction-conditioned channel gate on pre-aligned visual tokens improves grounding in LLaVA-style models on most reported benchmarks, but the mechanism is undercut by the paper's own ablation and the abstract ove...

  5. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

47 extracted references · 35 canonical work pages · cited by 5 Pith papers

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 2, 3

    arXiv 2023

  2. [2]

    From colouring-in to pointillism: revisiting semantic segmentation supervision,

    Rodrigo Benenson and Vittorio Ferrari. From colouring-in to pointillism: revisiting semantic segmentation supervision,

    work page

  3. [3]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Adv. Neural Inform. Process. Syst., 33:1877–1901, 2020. 2

    work page 1901

  4. [4]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th Interna- tional Conference on Machine Learning , pages 1597–1607. PMLR, 2020. 2

    work page 2020

  5. [5]

    Pali: Scaling language-image learning in 100+ languages

    Xi Chen and Xiao Wang. Pali: Scaling language-image learning in 100+ languages. In Adv. Neural Inform. Process. Syst., 2022. 2

    work page 2022

  6. [6]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2818–2829, 2023. 2

    work page 2023

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2, 8

    work page 2023

  8. [8]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    work page 2024

Show all 47 references
  1. [9]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Adv. Neural Inform. Process. Syst., 36, 2024. 2, 3, 5

    2024

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    2018 arXiv

  3. [11]

    Internlm-xcomposer2: Mastering free-form text- image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Ji- aqi ...

    2024 arXiv

  4. [12]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  5. [13]

    A unified continual learn- ing framework with general parameter-efficient tuning

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learn- ing framework with general parameter-efficient tuning. In Int. Conf. Comput. Vis., pages 11483–11493, 2023. 5

    2023

  6. [14]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural Info...

    2020

  7. [15]

    Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  8. [16]

    Dimension- ality reduction by learning an invariant mapping

    Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimension- ality reduction by learning an invariant mapping. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1735–1742. IEEE,

  9. [17]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2020. 2

    2020

  10. [18]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Int. Conf. Learn. Represent., 2022. 5

    2022

  11. [19]

    BRA VE: Broadening the visual encoding of vision-language models

    O ˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRA VE: Broadening the visual encoding of vision-language models. In Eur. Conf. Comput. Vis., 2024. 2, 3

    2024

  12. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Int. Conf. on Mach. Learning , pages 19730–19742. PMLR,

  13. [21]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Conf. on Empirical Methods in Natural Language Processing , pages 292–305, Singapore,

  14. [22]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., pages 740–755. Springer, 2014. 5

    2014

  15. [23]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2, 3, 5

    2023

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Adv. Neural Inform. Process. Syst., 2023. 2, 3 9

    2023

  17. [25]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    1907 arXiv

  18. [26]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 3, 6

    2023

  19. [27]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 8

    2023 arXiv

  20. [28]

    Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. InInt. Conf. Learn. Represent., 2024. 2

    2024

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Int. Conf. on Mac...

    2021

  22. [30]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(1):5485–5551, 2020. 2

    2020

  23. [31]

    xgen-mm-phi3-mini-instruct model card, 2024

    Salesforce AI Research. xgen-mm-phi3-mini-instruct model card, 2024. 2

    2024

  24. [32]

    A-okvqa: A benchmark for visual question answering using world knowl- edge, 2022

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge, 2022. 6

    2022

  25. [33]

    Eva-clip: Improved training techniques for clip at scale,

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale,

  26. [34]

    Eyes wide shut? exploring the vi- sual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the vi- sual shortcomings of multimodal llms. In IEEE Conf. Com- put. Vis. Pattern Recog., pages 9568–9578, 2024. 2, 3, 5, 6, 8

    2024

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  28. [36]

    Pivot: Prompting for video con- tinual learning

    Andr ´es Villa, Juan Le ´on Alc´azar, Motasem Alfarra, Kumail Alhamoud, Julio Hurtado, Fabian Caba Heilbron, Alvaro Soto, and Bernard Ghanem. Pivot: Prompting for video con- tinual learning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 24214–24223, 2023. 5

    2023

  29. [37]

    Behind the magic, merlim: Multi- modal evaluation benchmark for large image-language mod- els, 2024

    Andr ´es Villa, Juan Carlos Le ´on Alc ´azar, Alvaro Soto, and Bernard Ghanem. Behind the magic, merlim: Multi- modal evaluation benchmark for large image-language mod- els, 2024. 2, 3, 6, 7

    2024

  30. [38]

    Corda: Context-oriented decomposition adaptation of large language models

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models. arXiv preprint arXiv:2406.05223, 2024. 5

    2024 arXiv

  31. [39]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Int. Conf. Comput. Vis., pages 11975–11986, 2023. 2

    2023

  32. [40]

    Galore: Memory- efficient llm training by gradient low-rank projection, 2024

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- efficient llm training by gradient low-rank projection, 2024. 5

    2024

  33. [41]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. Int. Conf. Learn. Represent., 2024. 2

    2024

  34. [42]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2, 5

    2023 arXiv

  35. [43]

    LLA” (LLaV A-1.5), “LLA*

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once.Adv. Neural Inform. Process. Syst., 36, 2024. 6 10 EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instru...

    2024

  36. [45]

    We compare the zero-shot and linear probing performance of EAGLE-tuned VLMs against the original models

    Linear Probing of EAGLES-Tuned Models We evaluate whether EAGLE effectively preserves the transfer capability of the features in the original VLM. We compare the zero-shot and linear probing performance of EAGLE-tuned VLMs against the original models. As shown in Table 1, EAGL...

  37. [46]

    Figure 3 presents 3 additional scenarios when EAGLE effectively reduces the hallucinations of the IT-VLMs

    Qualitative Examples Reducing Hallucinations in IT-VLMs. Figure 3 presents 3 additional scenarios when EAGLE effectively reduces the hallucinations of the IT-VLMs. Each scenario features a question about an image by a specific IT-VLM Figure 4. Visual Examples of EAGLE Enhancin...

  38. [47]

    The same hyperparameters are applied to both VLMs, EV A01-CLIP- g-14 and OpenAI CLIP-L-14-336

    Training Hyperparameters Table 8 summarizes the additional hyperparameters intro- duced for EAGLE training, building on the original settings provided by [33] for training EV A01-CLIP-g-14. The same hyperparameters are applied to both VLMs, EV A01-CLIP- g-14 and OpenAI CLIP-L-...

  39. [2023]

    Association for Computational Linguistics. 2, 6

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.