{"id":"854d7c06-ac6c-45ee-bdcc-2db1440fd10a","arxiv_id":"2501.02699","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Tuning a CLIP-style visual encoder with masked-pooling contrastive losses on instance segmentation data reduces hallucinations in downstream instruction-tuned vision-language models when the encoder is swapped in without retraining.","lead":"EAGLE tunes the visual encoder of instruction-tuned vision-language models on object segmentation data, then swaps the tuned encoder into existing models without retraining the LLM or adapter. The authors report fewer object hallucination errors on POPE, MMVP, and MERLIM across six IT-VLM architectures, with the largest gains on fine-grained visual benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvement percentages are inflated by the retrained LLaVA-1.5* variant, and the unretrained LLaVA-1.5 actually degrades on POPE, contradicting the universal 'no-training' hallucination-reduction claim.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption focused on whether patch tokens encode object-level semantics. That is a reasonable concern about the training signal, but the most load-bearing issue for the central claim is internal to the reported results: the largest advertised gains come from LLaVA-1.5*, which required additional instructional training, and the one plug-and-play model evaluated on POPE (LLaVA-1.5) shows a clear decline. The paper's own tables therefore fail to support the abstract's universal, training-free claim. I agree with the CONDITIONAL verdict because the method may still be valuable for the models that do improve and the underlying idea is plausible, but the paper cannot be accepted without correcting the overclaims, separating retrained variants from plug-in results, and reporting error bars/significance. My disagreement with the reader is partial: I am pointing to a more direct empirical contradiction rather than the patch-token assumption, though that assumption remains a valid secondary concern.","tokens_in":15535,"tokens_out":6398,"duration_ms":56665,"concrete_test":"Recompute the average MMVP and MERLIM improvements from Tables 4 and 5 using only the six directly-replaced models (exclude LLaVA-1.5*). Also run the POPE benchmark for LLaVA-1.5 with the original and EAGLE-tuned OpenAI ViT-L-14-336 encoders under the same protocol. If the recomputed averages remain ≈1.67 and ≈2.33 rather than 2.3 and 2.73, and/or the POPE drop is reproducible, then the headline claim is overstated and must be corrected to specify which models improve and by how much.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Abstract; §4.2) is that EAGLE reduces hallucinations across six IT-VLMs without any instructional training. Two issues undercut this. First, the reported aggregate improvements (2.3% MMVP, 2.73% MERLIM) match the averages computed only when including the LLaVA-1.5* row, which was retrained on LLaVA instructional data with the EAGLE encoder (§4.2, 'LLaVA A Training'). Excluding LLaVA-1.5*, the six plug-and-play models improve by only ~1.67% on MMVP and ~2.33% on MERLIM. Second, Table 4 itself contradicts the claim of 'consistently enhances performance across all of the IT-VLMS for all metrics': for LLaVA-1.5 with the EAGLE encoder, POPE Accuracy drops from 69.33% to 67.53%, and F1, Precision, and Recall also drop. Thus the strongest form of the claim—universal improvement without retraining—is not supported by the paper's own tables.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EAGLE, a post-pretraining method for CLIP-style vision encoders. EAGLE fine-tunes the ViT using masked-average-pooled patch tokens aligned with text embeddings of object class names from OpenImages V7, using a two-part loss (instance-level contrastive plus class-level cross-entropy) and GaLore to limit feature drift. The resulting encoder is then swapped into existing instructional VLMs without retraining their LLM or adapter. The paper reports evaluations on POPE, MMVP, and MERLIM for six IT-VLMs, plus a retrained LLaVA-1.5 variant, and claims consistent hallucination reduction and large aggregate improvements.","tokens_in":15822,"tokens_out":10059,"duration_ms":84874,"significance":"The idea of improving visual grounding at the encoder level in a plug-and-play way is timely and could complement decoder-side hallucination mitigation methods. The paper includes ablations (Table 6) and shows gains on most models and benchmarks, and the method is architecture-agnostic. However, the abstract's universal claim is contradicted by the paper's own data (LLaVA-1.5 POPE degrades), and the headline improvement percentages are not reproducible from the tables. The lack of uncertainty quantification and of comparisons to existing hallucination-mitigation baselines further limits the strength of the claims.","major_comments":[{"comment":"The caption of Table 4 states that \"without any tuning or alignment to the LLM or its fusion model, EAGLE consistently enhances performance across all of the IT-VLMS for all metrics in both benchmarks.\" This is not supported by the table: for LLaVA-v1.5, POPE Accuracy drops from 69.33% to 67.53%, and F1, Precision, and Recall also drop. Additionally, InstructBLIP (Vicuna-13B) MMVP is unchanged at 24.67%, and InstructBLIP (FlanT5xl) POPE Recall drops from 98.0% to 97.93%. The universal no-training claim must be weakened and the failure cases analyzed.","section":"Table 4 and Section 4.2"},{"comment":"The paper reports \"2.3% absolute improvement\" on MMVP and \"2.73% absolute improvement\" on MERLIM, and the Contributions claim \"11.2% relative improvement in MMVP and 6.3% relative improvement in MERLIM.\" Averaging the per-model deltas in Table 4 for the six plug-and-play models gives an MMVP gain of 1.67% absolute (2.19% if LLaVA-1.5* is included); for Table 5 the MERLIM average is 2.33% absolute (2.42% with LLaVA-1.5*). The reported numbers do not match any transparent computation from the tables. Please specify exactly how the aggregates are computed and whether they include the retrained LLaVA-1.5* row.","section":"Section 4.2 and Contributions"},{"comment":"The Abstract and Contribution (i) state that EAGLE reduces hallucinations \"without additional instructional training\" and quote the aggregate percentages. However, LLaVA-1.5* requires second-stage instructional tuning, as described in Section 4.2. The plug-and-play results should be cleanly separated from the retrained results in the abstract and in the contribution claims, so that the reader is not led to believe that the headline numbers apply uniformly to the no-training setting.","section":"Abstract and Section 4.2 (LLaVA A Training)"},{"comment":"The word \"significant\" is used repeatedly without statistical tests or error bars. Many gains are small in absolute terms (e.g., MiniGPT-4 POPE +0.54, LLaVA-1.5 MERLIM +0.51), and no comparison is made to existing hallucination-mitigation methods such as VCD, OPERA, or other visual-encoder enhancement methods (e.g., BRAVE). At minimum, the authors should report confidence intervals or multiple seeds, and compare against at least one strong baseline on the same benchmarks before claiming significance.","section":"Section 4 in general"},{"comment":"In Eq. (2), d_j = 1 - sigma(phi(I,m) - l(c_j)): if sigma is the element-wise sigmoid and both phi(I,m) and l(c_j) are vectors, then d_j is a vector, making the binary cross-entropy in Eq. (3) ill-defined. Moreover, c_j is used both as the class-name prompt in Eq. (1) and as the target label in Eq. (3). Please define the exact tensor shapes and the form of c_j in the loss. This is central to understanding the training objective.","section":"Equations (2) and (3)"}],"minor_comments":[{"comment":"The EAGLE EVA-01 CLS row prints a delta of (-1.49%), but the baseline average is 25.18% and the EAGLE average is 26.67%, which is an improvement of +1.49%; the sign is inconsistent with the text in Section 4.1.","section":"Table 2"},{"comment":"The text says \"We train until convergence of the Lm loss,\" but L_m is never defined; presumably the total loss L in Eq. (4) is meant.","section":"Section 4, Implementation Details"},{"comment":"The MMVP task names contain garbled glyphs (e.g., \"♂search\", \"/sync\", \"/sort-numeric-up\", \"♂¶ap-pin\"); please use plain text names or a legend.","section":"Table 2"},{"comment":"The sentence \"EAGLE enhances the performance of all evaluated IT-VLMs across the three selected benchmarks\" is contradicted by the POPE result for LLaVA-1.5. Please revise this sentence and the corresponding claim in the table caption.","section":"Section 4.2"},{"comment":"The discussion of contrastive learning states that the standard contrastive loss is incompatible because batches may contain multiple masks of the same class, but the exact definition of negatives in Lins is not given. Please clarify whether Lins uses in-batch negatives and how repeated classes are handled.","section":"Section 3.1"},{"comment":"MERLIM [37] is co-authored by two of the current authors (Villa and Ghanem), but the paper cites it as an external benchmark without noting this overlap. For transparency, please add a statement about the self-evaluation.","section":"References/Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's headline numbers are not reproducible from its own tables, and the universal no-training claim is contradicted by the LLaVA-1.5 POPE decrease. The core idea is interesting and orthogonal to existing methods, but the presentation and the strength of the claims need substantial revision. The self-evaluation on MERLIM should also be disclosed in the main text or at least clearly flagged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Juan, this one is worth a look, but read the tables before the abstract. The core idea is solid: take a CLIP ViT, apply masked average pooling over instance masks to get per-object patch-token features, align those with the text embedding of the class name, and use GaLore fine-tuning so the global CLS representation stays intact. You then swap the encoder into an IT-VLM without retraining the LLM or adapter. That is a plausible, cheap, and modular way to attack hallucinations, and the VLM-level evidence supports it: MS-COCO false positives drop by 10-20 points, and sequence-token zero-shot and linear probing on ImageNet hold up.\n\nThe soft spots are real but manageable. The paper claims 'consistently enhances performance across all of the IT-VLMs for all metrics.' It does not. LLaVA-1.5 with the EAGLE encoder drops on POPE accuracy from 69.33 to 67.53, with F1, precision, and recall all down. The abstract's headline improvements also lean on LLaVA-1.5*, the variant retrained with the EAGLE encoder. Excluding that row, the six plug-and-play models improve MMVP by about 1.67% on average, not 2.3%; the MERLIM table shows 2.33% for those six, and the claimed 2.73% does not match the numbers in the table. The loss equation (2) also has a vector-scalar mismatch as written—phi and l are vectors, sigmoid of their difference is elementwise, and then you feed that into a binary cross-entropy with a scalar class label. It needs a proper definition, probably cosine similarity or a projection. There are no error bars, significance tests, or comparisons to existing hallucination-mitigation baselines. The MERLIM co-authorship is a mild conflict, but the training signal comes from OpenImages, so I do not see circularity.\n\nNone of this kills the method. The idea is new, the drop-in evaluation across six IT-VLMs is useful, and the gains, while small, are consistent in most cases and large on MS-COCO false positives. The paper deserves a serious referee, but the authors need to fix the equations, rescope the claims from 'universal' to 'most models,' and report the unretrained averages honestly. I would send it to review with that expectation.","headline":"Worth reading: EAGLE is a genuinely useful drop-in visual encoder for reducing hallucinations, but the paper's 'universal no-training improvement' claim is contradicted by its own tables and needs honest rescoping.","tokens_in":16339,"tokens_out":4080,"would_cite":true,"duration_ms":35483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing a pre-trained vision model's visual encoder with an EAGLE-tuned encoder reduces hallucinated object descriptions in instruction-tuned vision-language models, without retraining their language components.","keywords":["hallucination","visual grounding","vision-language models","instruction tuning","contrastive learning","visual encoder fine-tuning","instance segmentation","multimodal benchmarks"],"falsifier":"Run EAGLE on OpenImages but replace each object mask with a random crop of the same size and the same class label; if the reported POPE, MMVP, and MERLIM gains are reproduced, then the claim that object-grounded alignment is the responsible mechanism would be undermined.","tokens_in":15338,"feed_emoji":"🖼️","tokens_out":6729,"duration_ms":59472,"temperature":0.7,"pith_summary":"This paper claims that a large share of object hallucination in instruction-tuned vision-language models (IT-VLMs) comes from the visual encoder: the patch-level tokens feeding the language model are poorly aligned with object semantics, even though the global CLS token is well aligned. EAGLE is a post-pretraining step that fixes this by tuning a CLIP-style ViT so that the average of patch tokens inside a segmentation mask matches the frozen text embedding of that object's class name. The tuned encoder can then replace the default visual encoder in a trained IT-VLM with no retraining of the adapter or language model. Across six IT-VLMs and three hallucination benchmarks (POPE, MMVP, MERLIM), the swap reduces hallucination, with the largest relative gains on MMVP (11.2%) and MERLIM (6.3%).","feed_headline":"Visual-encoder swap cuts hallucinations in six vision-language models","feed_subtitle":"EAGLE aligns patch-level image features with object labels, then drops into trained models with no retraining.","key_machinery":"The load-bearing object is the masked average pooling operator $\\phi(I,m)$: it zeroes out every patch token outside a binary segmentation mask and averages the rest, producing a single local visual vector for an object. EAGLE's loss then drives that vector toward the frozen text embedding of the object's class name (with the prompt prefix \"This is an image of <class>\") using two terms: a contrastive instance-level loss and a sigmoid cross-entropy loss that allows multiple masks to share a class. Because the CLS token is never trained, the global descriptor is preserved, and GaLore's low-rank gradient projection limits distribution shift during full-parameter fine-tuning. This mechanism is what lets the improved encoder be dropped into an existing IT-VLM with no adapter or LLM retraining.","core_discovery":"The central discovery is that the feature sequence—the patch tokens an IT-VLM actually consumes—lags far behind the CLS token in language alignment, and that lag can be closed with a reformulation of contrastive pre-training. EAGLE samples one segmented object from an image, pools the ViT's patch embeddings over the object's mask, and aligns the pooled vector with the frozen text embedding of its class name. A per-instance contrastive term plus a per-class sigmoid cross-entropy term update only the visual encoder, while GaLore keeps the global feature space from drifting. The result is an encoder whose sequence embeddings carry fine-grained object information, verified by a jump in zero-shot accuracy when using sequence features on ImageNet (e.g., from 0.7% to 50.98% for the OpenAI ViT-L-14-336) and by reductions in false positives on MS-COCO, and this transfers to reduced hallucination in six IT-VLMs without any instructional training.","pith_inferences":["The authors do not state this, but their results suggest the gains come from making local patch tokens carry nameable object information, so alternative local-alignment supervisions—caption-derived masks, self-supervised object discovery, or point prompts—might reproduce similar reductions.","An untested corollary: EAGLE-style encoders should also improve tasks that require localized semantics, such as referring-expression grounding, open-vocabulary detection, or questions about small objects in dense scenes.","A natural stress-test would combine EAGLE with a language-side hallucination regularizer, since the two target different components and their benefits may be additive."],"forward_implications":["A trained IT-VLM can be improved by only swapping its visual encoder, with no adapter tuning, no LLM fine-tuning, and no added inference cost.","The patch-level feature sequence becomes a usable zero-shot classifier on its own: for the OpenAI encoder, ImageNet sequence accuracy rises from 0.7% to 50.98% while the CLS token drops only 5.09%.","BLIP-2 and InstructBLIP benefit more than LLaVA-v1.5, which the authors attribute to their adapter-only tuning making them more compatible with a fixed encoder.","EAGLE improves both MERLIM prompts without changing their relative ranking, meaning it reduces hallucinations without correcting language-side prompt bias.","Training the CLS token would lower false positives further but sacrifices zero-shot accuracy, so EAGLE leaves that token untouched."],"supporting_citations":[{"why":"Supplies the base visual encoder (OpenAI CLIP-L-14-336) whose sequence features EAGLE tunes.","marker":"[29]"},{"why":"Supplies the second base encoder (EVA-CLIP ViT-g-14) and the training and linear-probing protocol EAGLE builds on.","marker":"[33]"},{"why":"Provides InstructBLIP, one of the six IT-VLMs evaluated, and the instruction-tuning baseline.","marker":"[9]"},{"why":"Provides BLIP-2, the IT-VLM whose architecture EAGLE is plugged into.","marker":"[20]"},{"why":"Provides LLaVA-v1.5, the IT-VLM whose adapter and LLM are co-trained; the EAGLE variant LLaVA-v1.5* retrains its second stage.","marker":"[23]"},{"why":"Defines the POPE benchmark used to measure object hallucination with yes/no questions.","marker":"[21]"},{"why":"Defines the MMVP and MMVP-VLM benchmarks used to test fine-grained visual-pair discrimination.","marker":"[34]"},{"why":"Defines the MERLIM benchmark and the 'hidden hallucination' concept used to evaluate removed objects.","marker":"[37]"},{"why":"Supplies the OpenImages V7 instance-segmentation dataset used as EAGLE's training data.","marker":"[2]"},{"why":"Supplies GaLore, the memory-efficient low-rank gradient projection that prevents zero-shot drift during EAGLE tuning.","marker":"[40]"}],"fun_headline_variants":["Visual-encoder swap cuts hallucinations in six models","Patch-level alignment reduces vision-language hallucinations","EAGLE upgrades visual encoder to slash hallucinations","Retrain-free visual grounding reduces model hallucinations","Enhancing patch tokens minimizes multimodal hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that patch-level tokens of a pre-trained ViT, after masked average pooling over a segmentation mask, can be meaningfully aligned with the frozen text embedding of the object's class, and that this local alignment transfers to the feature sequence in a way that reduces hallucination.","fun_headline_variants_meta":{"raw":{"variants":["Visual-encoder swap cuts hallucinations in six models","Patch-level alignment reduces vision-language hallucinations","EAGLE upgrades visual encoder to slash hallucinations","Retrain-free visual grounding reduces model hallucinations","Enhancing patch tokens minimizes multimodal hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1421,"prompt_tokens":943,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":559,"tokens_out":478,"duration_ms":5169,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:07:24.840897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EAGLE on OpenImages but replace each object mask with a random crop of the same size and the same class label; if the reported POPE, MMVP, and MERLIM gains are reproduced, then the claim that object-grounded alignment is the responsible mechanism would be undermined.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the base visual encoder (OpenAI CLIP-L-14-336) whose sequence features EAGLE tunes."},{"cited_title":"Instructblip: Towards general- purpose vision-language models with instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides InstructBLIP, one of the six IT-VLMs evaluated, and the instruction-tuning baseline."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Provides BLIP-2, the IT-VLM whose architecture EAGLE is plugged into."},{"cited_title":"Improved baselines with visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Provides LLaVA-v1.5, the IT-VLM whose adapter and LLM are co-trained; the EAGLE variant LLaVA-v1.5* retrains its second stage."},{"cited_title":"Evaluating object hallucination in large vision-language models","cited_arxiv_id":null,"evidence_quote":"Defines the POPE benchmark used to measure object hallucination with yes/no questions."},{"cited_title":"Eyes wide shut? exploring the vi- sual shortcomings of multimodal llms","cited_arxiv_id":null,"evidence_quote":"Defines the MMVP and MMVP-VLM benchmarks used to test fine-grained visual-pair discrimination."},{"cited_title":"Behind the magic, merlim: Multi- modal evaluation benchmark for large image-language mod- els, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the MERLIM benchmark and the 'hidden hallucination' concept used to evaluate removed objects."},{"cited_title":"From colouring-in to pointillism: revisiting semantic segmentation supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenImages V7 instance-segmentation dataset used as EAGLE's training data."},{"cited_title":"Galore: Memory- efficient llm training by gradient low-rank projection, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies GaLore, the memory-efficient low-rank gradient projection that prevents zero-shot drift during EAGLE tuning."}],"review_version":1}