Pith. sign in

REVIEW 4 major objections 4 minor 23 references

A model's wrong answer in a vision-language Q&A can be traced to one of four failure points before the model generates a single word.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:27 UTC pith:MHAHMI7K

load-bearing objection Solid descriptive core on pre-generation failure attribution, but the intervention gains are uncontrolled and can't yet be credited to routing. the 4 major comments →

arxiv 2607.04683 v3 pith:MHAHMI7K submitted 2026-07-06 cs.CV cs.CL

Failing to See or Failing to Know? Attributing Errors in Vision-Language Models

Khang Nhat Hoang Vo , Artem Vazhentsev , Artem Shelmanov , Timothy Baldwin , Yova Kementchedjhieva This is my paper
classification cs.CV cs.CL
keywords vision-language modelsknowledge-intensive VQAfailure attributionpre-generation representationsentity recognitionfactual accesstargeted interventionsoperational diagnostics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when a vision-language model gives a wrong answer to a knowledge-based question about an image, the error is not one monolithic failure. Sometimes the model never recognizes the thing in the picture; sometimes it recognizes the thing but cannot pull up the fact; sometimes it would succeed if the entity were named; sometimes the image itself is too degraded. The paper builds a decision tree that sorts each wrong answer into one of these categories using simple behavioral tests, then shows that signals hidden inside the model — before it generates any text — can predict which category an example will fall into. It then demonstrates that routing each predicted failure to a targeted fix (repairing the image, naming the entity, rewriting the question, adding a fact) lifts accuracy by 30–39 percentage points.

Core claim

The central claim is that VLM errors in knowledge-intensive visual question answering are not monolithic but decompose along a small set of operational decisions: whether the entity is recognized, whether controlled visual degradation breaks recognition, whether the original question is answered correctly after recognition, and whether the answer becomes correct when the entity is named explicitly. Each decision leaves a distinct trace in the model's pre-generation hidden states, so a separate lightweight probe per decision can predict the failure before decoding begins. The paper shows that visual-token hidden states are the strongest signal for image-side decisions, while prompt-boundary h

What carries the argument

The attribution tree is the central object: four local binary decisions arranged hierarchically, mapping each image-question pair to one of five operational outcomes (visual-evidence failure, unknown entity, success, unknown fact, unrecallable fact). Its work is to turn an unstructured wrong answer into a structured prediction target that can use different internal signals at different nodes, and to enable targeted interventions rather than a single abstention decision. The probes that do the predicting read pre-generation features — the final visual-token hidden state (VIS), the final prompt-token hidden state (EOP), the concatenated hidden states of the last eight prompt tokens (LAST8), an

Load-bearing premise

The labels produced by the controlled behavioral tests faithfully represent the true failure causes — that the yes/no recognition probe with distractors really measures entity recognition, that the entity-explicit rewrite really separates missing knowledge from failed recall, and that synthetic degradation stands in for natural visual failure; if any proxy is unfaithful, every downstream attribution and intervention result is conditional on the labeling protocol.

What would settle it

Take a sample of wrong answers where humans independently label the underlying cause (entity not seen, fact absent from the model, image degraded in a natural way such as occlusion or glare) and compare against the tree's operational labels on the same model outputs; if the tree's predicted failure type disagrees with the human diagnosis on a substantial fraction, the framework is measuring its own probes rather than the failure process. Concretely, test the visual-evidence probes on naturally degraded images — if the 97.0 PR-AUC achieved on synthetic corruption drops to chance under occlusion

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Wrong answers can be sorted into pre-recognition and post-recognition branches, and the two branches require different remedies.
  • Pre-decoding diagnostics can replace blanket abstention: instead of refusing to answer, the model can be routed to a targeted fix.
  • Visual-token features carry a strong signal for recognition failures, while prompt-boundary hidden states carry a useful but weaker signal for whether an answer will be factually correct.
  • Factual-access attribution (unknown fact vs. unrecallable fact) remains hard; only a weak pre-generation signal exists, close to what answer-level uncertainty provides.
  • The labels are model-specific: the same image-question pair can receive different failure labels for different target models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests a deployment pattern where every VLM answer carries a predicted 'failure type', letting an interactive system decide whether to ask a clarifying question, retrieve a fact, or show a repaired image — rather than silently accepting a wrong answer.
  • The visual-evidence branch is built from synthetic corruption (Gaussian blur, noise, JPEG compression, downscaling); natural failures such as occlusion, glare, and unusual viewpoint may not be detectable by the same feature, so the 97.0 PR-AUC on controlled degradation may not transfer to the wild.
  • The entity-explicit rewrite test may not cleanly separate 'fact absent from memory' from 'fact present but not recalled'; the weak factual-access signal could reflect this labeling ambiguity rather than an intrinsic limit of the model's internals.
  • A natural extension is to couple this attribution tree with retrieval-augmented generation: use the predicted failure type to decide when to retrieve external evidence, which is currently applied indiscriminately in many systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an operational attribution tree for failures in knowledge-intensive VQA, assigning each incorrect answer to one of four failure classes (visual-evidence failure, unknown entity, unknown fact, unrecallable fact) or success, using model-specific behavioral tests. The authors extract pre-generation features (visual-token states, prompt-boundary hidden states, attention maps) and train local binary probes at each tree node. They report that visual-token features best predict recognition-side decisions, prompt-boundary hidden states best predict answer success, factual-access prediction is weak, tree-composed prediction beats a flat 5-label classifier, and attribution-guided routing to targeted interventions yields 30.8–39.3 point accuracy gains. The paper is unusually honest about the operational, model-relative nature of its labels and about the weak factual-access signal.

Significance. If the central empirical pattern is robust, the paper makes a useful contribution: it moves beyond monolithic hallucination/abstention to a fine-grained, model-specific failure taxonomy, and it provides concrete evidence that different pre-generation representation families carry decision-dependent information. The multi-model, multi-dataset design and the explicit acknowledgment that factual-access attribution is weak are strengths. The intervention claim, however, is the part that would most distinguish this framework, and the current experiment is not controlled enough to support the routing conclusion. The taxonomy itself is also only as informative as its operational tests, which are acknowledged but not validated. As such, the paper is a promising diagnostic study whose main applied claim needs substantially stronger evidence.

major comments (4)
  1. [§5, Table 4] The attribution-guided intervention experiment lacks the controls needed to attribute the 30.8–39.3 point gains to the routing mechanism. Table 4 only compares original accuracy to accuracy after GPT-5 executes the branch selected by the predicted label. Because every non-visual branch augments the prompt with GPT-5 output (Appendix G) and each intervention is intended to be helpful, the gains could largely occur even under random or uniform routing. Please add: (i) random routing among the four interventions for non-SUCCESS examples; (ii) best-single intervention applied to all non-SUCCESS examples; (iii) apply-all (providing all supports at once); and (iv) a generic prompt-augmentation control with no branch-specific support. Without these, the paper's strongest applied claim — that fine-grained pre-generation attribution enables the improvement — is unsupported.
  2. [§3.2–§3.3, Limitations] The operational labels are defined by the target VLM's own behavior: UNKNOWN ENTITY is determined by yes/no probe answers to one ground-truth and three distractor prompts; VISUAL-EVIDENCE FAILURE is defined as a flip under a synthetic corruption schedule; UNKNOWN FACT vs. UNRECALLABLE FACT is determined by whether an entity-explicit rewrite succeeds. The paper states these are diagnostic indicators, not validated causal ground truth, but all downstream conclusions (probe rankings, tree-vs-flat comparison, intervention gains) are conditional on this protocol. The manuscript should either empirically validate the protocol (e.g., human sanity-checking of predicted labels, robustness of the probe rankings to alternative distractor counts/thresholds or to natural corruptions like occlusion/viewpoint) or explicitly restrict the claims to the operational definitions. Currently, the framework's
  3. [Tables 1, 2, and 4] No error bars, confidence intervals, or seed variance are reported anywhere. Several key comparisons are close: in Table 2, LAST8 Transformer for factual access is 53.5 versus ORACLE-UQ 52.4 and random 49.8; in Table 1, recognition differences between VIS and other features are smaller for some models. The qualitative claim that visual-token features are best for image-side decisions and prompt-boundary states are best for post-recognition decisions would be more convincing with multiple seeds or bootstrap intervals, and the weak factual-access claim should be framed with explicit uncertainty. Without variance estimates, a reader cannot tell whether the decision-dependent pattern is reliable across feature families, especially for the 1–2 point gaps.
  4. [Table 3] The tree-vs-flat comparison is not fully matched. Tree composition selects the best local probe per node using validation PR-AUC, while the flat classifier is a single model with its own hyperparameter grid. The flat classifier may be disadvantaged by the representational bottleneck, but it may also be disadvantaged by hyperparameter selection for a different objective (macro PR-AUC). Please report flat-classifier results using the same feature families and heads, or at least a per-feature comparison (e.g., LAST8 linear flat vs. tree-composed with LAST8 linear at all nodes), to show that the improvement comes from tree structure rather than from model-selection freedom or the ability to use different features at different nodes.
minor comments (4)
  1. [§1, Fig. 1] The example in the bottom panel says 'No, it’s the Forbidden City' for a Mukden Palace recognition check, which is a plausible distractor, but the figure caption could clarify that this is illustrative rather than an actual model output. Minor but helpful for reproducibility.
  2. [§3.2, Visual-evidence failure] The corruption schedule in Table 12 applies a fixed bundle of blur, noise, JPEG, and downscale at each level. It would be useful to report how many examples flip at each level, and whether the flip is dominated by one operation, since the operational definition of 'visual-evidence failure' depends on this schedule.
  3. [Appendix I.5] The balancing of visual-evidence failures to match unknown-entity counts is mentioned, but the exact post-balancing class sizes per model are not given. Since class balance affects PR-AUC and the random baseline in Tables 1–2, reporting these numbers would increase transparency.
  4. [Appendix G] For the UNRECALLABLE FACT branch, the target VLM is first asked to identify the entity, then GPT-5 rewrites the question. If the entity-identification step is wrong, the rewritten question may be misleading. Please report the entity-identification accuracy of the target VLMs, or at least discuss this potential error source in the limitations.

Circularity Check

1 steps flagged

Probe analysis is self-contained; one intervention branch is the same operation that defines its label, making part of the routing gain definitional.

specific steps
  1. self definitional [§3.2 (Unrecallable fact vs. Unknown fact) and §5 (Attribution-Guided Intervention, UNRECALLABLE FACT bullet)]
    "If the model fails on the original question but succeeds on the entity-explicit version, we assign UNRECALLABLE FACT... Under UNRECALLABLE FACT, no external factual evidence is added. We first ask the target VLM to identify the entity, GPT-5 then rewrites the question to state the predicted entity explicitly, without adding factual information or answering the question."

    The label UNRECALLABLE FACT is defined by success on an entity-explicit rewrite. The intervention for that branch is the same entity-explicit rewrite (using the target VLM's predicted entity). For examples where the predicted entity matches the ground-truth entity used in labeling, the intervention is exactly the operation that produced the label, so the observed gain on this branch is inherited from the labeling protocol rather than providing independent evidence for attribution-guided routing. The paper acknowledges labels are operational, but Table 4/Figure 4 still count this branch's gain as support for the routing claim.

full rationale

The core probe experiments are not circular: features (VIS/EOP/LAST8/ATTN) are defined independently of the labels and are evaluated on held-out examples, so high PR-AUC is an empirical result rather than an identity. The labels are admittedly operational diagnostics, and predicting a model's own behavior from its hidden states is a legitimate supervised-probing setup. Self-citations (Shelmanov et al. 2025, Vazhentsev et al. 2025, Vashurin et al. 2025) appear only as baselines/feature-motivation and are not load-bearing. The intervention experiment, however, has one definitional component: the UNRECALLABLE FACT intervention is the same entity-explicit rewrite used to define that label. The absence of random-routing/single-intervention controls is a separate experimental-design concern, not a circularity. Overall, the central attribution-signal claim retains independent content, so the circularity is partial rather than total.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 1 invented entities

The framework rests on behavioral-test definitions (Section 3.2), a two-hop task decomposition (Sections 2/3.1), a synthetic degradation schedule (Table 12), an 8B LLM judge (Appendix B), and the assumption that late-layer pre-generation representations encode the operational decisions (Section 3.3). None of these are externally validated against human causal annotation; the paper repeatedly disclaims causal status. Free parameters are modest and mostly protocol choices, but they directly shape the label distribution and therefore the headline probe scores.

free parameters (5)
  • Corruption schedule (blur sigma, noise std, JPEG quality, downscale per level) = Levels 1-4: sigma=2.5-5.5, noise=25-60, JPEG=60-25, downscale=0.75-0.40 (Table 12)
    Hand-chosen severity levels define the visual-evidence failure boundary; different schedules would change the label distribution and all probe scores on that branch.
  • Recognition protocol thresholds = yes to 1 ground-truth prompt, no to all 3 distractors
    Choice of distractor count and the strict conjunction determines the UNKNOWN ENTITY label; a looser or tighter rule would shift the recognition split.
  • Visual-evidence balancing target = Downsample so visual-evidence count matches unknown-entity count per model
    Class balance in the recognition-failure split is forced to match the other class, affecting reported PR-AUC for recognition-side decisions.
  • Feature geometry choices = LAST8 window=8 tokens, attention lookback k=4, layer offsets {-1,-4,-8,-12}, top-m probabilities
    Ad hoc geometry inherited from prior probing work; results may shift with other window sizes or layer selections.
  • Probe hyperparameters = Grid in Appendix E.1 (lr 1e-4..1e-3, batch 64-256, epochs 5-30)
    Selected on validation PR-AUC; no seed averaging is reported, so validation-selection noise is unquantified.
axioms (7)
  • domain assumption Knowledge-intensive VQA is a two-hop process: recognition then factual access.
    The whole tree structure depends on this decomposition (attributed to Venhoff et al. 2025). If failures intertwine, the tree misattributes them.
  • domain assumption Forced-choice yes/no recognition probes with distractors measure recognition reliability.
    Section 3.2: the authors abandoned open-question probes because they were hard to evaluate; labels are therefore protocol-dependent.
  • domain assumption Entity-explicit question rewrites separate missing knowledge from failed recall.
    Section 3.2/I.7, following Cohen et al. 2025. The paper admits rewrites may not fully separate the two causes.
  • domain assumption Synthetic degradation (blur/noise/JPEG/downscale) represents visual-evidence failures.
    Section 3.2/I.5; explicitly acknowledged as not covering occlusion, viewpoint, cropping, or domain artifacts.
  • domain assumption A Meta-Llama-3-8B text-only judge gives valid correctness labels.
    Appendix B: answer-success and factual-access labels all filter through this judge; judge errors propagate into every label.
  • domain assumption White-canvas ablation identifies insufficiently image-grounded questions.
    Section 3.2/I.3: items any model answers without the image are removed, shaping the curated set and the reported low success rates.
  • domain assumption Late-layer hidden states of the target VLM are probeable proxies for the operational decisions.
    Section 3.3/Appendix D: the entire method presumes the signal is accessible at pre-generation prompt-boundary representations.
invented entities (1)
  • Five-label operational attribution taxonomy (VISUAL-EVIDENCE FAILURE, UNKNOWN ENTITY, SUCCESS, UNKNOWN FACT, UNRECALLABLE FACT) no independent evidence
    purpose: Organize VLM errors into routable decision leaves for targeted interventions
    Explicitly operational diagnostics, not independently validated causal ground truth. The labels are defined by the target model's own behavioral tests, so no falsifiable handle exists outside the paper's protocol.

pith-pipeline@v1.3.0-alltime-deepseek · 26320 in / 16403 out tokens · 145494 ms · 2026-08-04T04:27:20.325947+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) can recognize entities in clear images yet still fail when answering questions that require factual knowledge beyond what is directly observable. Prior work has either examined individual failure modes in isolation or treated incorrect answers as monolithic, binary failures. We propose a tree-structured framework that organizes failures in knowledge-intensive visual question answering into model-specific operational outcomes. Across two datasets and four VLMs, we observe consistent distributions of operational outcomes: some failures occur before entity recognition, while others persist after the relevant entity is recognized. Visual token representations are most informative for recognition-related decisions. Prompt hidden states predict answer success more effectively, although factual-access attribution remains difficult and exhibits only a weak signal. These pre-generation signals support attribution-guided routing to targeted interventions, including image repair, entity support, question rewriting, and factual evidence.

Figures

Figures reproduced from arXiv: 2607.04683 by Artem Shelmanov, Artem Vazhentsev, Khang Nhat Hoang Vo, Timothy Baldwin, Yova Kementchedjhieva.

Figure 1
Figure 1. Figure 1: Examples of two attribution outcomes in knowledge-intensive VQA. For each image, we eval￾uate the target VLM with two independent checks: a factual VQA question and an entity-recognition probe. Top: the model answers the factual question incorrectly but recognizes the entity, so the error is attributed to UN￾KNOWN FACT. Bottom: the model answers the factual question incorrectly and also fails the recogniti… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of attribution outcomes across [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Operational attribution tree for knowledge [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Composition of probe-guided mitigation gains. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of attribution outcomes across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Confusion matrix for the direct 5-label classi [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Composition of attribution-guided interven [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Confusion matrix for the direct 5-label clas [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: Confusion matrix for the direct 5-label classi [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Confusion matrix for the direct 5-label clas [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: Confusion matrix for the direct 5-label classi [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 7 linked inside Pith

  1. [1]

    Each probe is trained only on examples reaching its decision node in the attribution tree

    Prediction targets.For each local decision d∈ {R, D, S, F}, let y(d) i ∈ {0,1}denote the corre- sponding operational label: entity recognition for R, visual-evidence attribution for D, answer suc- cess for S, and factual access for F . Each probe is trained only on examples reaching its decision node in the attribution tree. Feature families.We extract fe...

    2025

  2. [4]

    InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 1419–1436

    Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 1419–1436. Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes

    2024

  3. [5]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing

    Active retrieval augmented generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing. Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du

    2023

  4. [6]

    InFind- ings of the Association for Computational Linguistics: EMNLP 2024, pages 5042–5063, Miami, Florida, USA

    FaithScore: Fine-grained evaluations of hallu- cinations in large vision-language models. InFind- ings of the Association for Computational Linguistics: EMNLP 2024, pages 5042–5063, Miami, Florida, USA. Association for Computational Linguistics. Omri Kaduri, Shai Bagon, and Tali Dekel

    2024

  5. [9]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee

    A survey on halluci- nation in large vision-language models.Preprint, arXiv:2402.00253. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee

    Pith/arXiv arXiv

  6. [11]

    InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pages 5769–5790

    Benchmarking vision language models for cultural understanding. InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pages 5769–5790. Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang

    2024

  7. [12]

    Xin Qiu and Risto Miikkulainen

    RoRA-VLM: Robust retrieval-augmented vision lan- guage models.arXiv preprint arXiv:2410.08876. Xin Qiu and Risto Miikkulainen

    Pith/arXiv arXiv

  8. [13]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35712–35731, Suzhou, China

    A head to predict and a head to question: Pre-trained uncertainty quantification heads for hallucination detection in LLM outputs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35712–35731, Suzhou, China. Association for Computational Linguistics. Philip Wootaek Shin, Jack Sampson, Vijaykrishnan Narayanan,...

    2025

  9. [14]

    Losing the plot: How VLM re- sponses degrade on imperfect charts.arXiv preprint arXiv:2509.18425. Settaluri Lakshmi Sravanthi, Pulkit Agarwal, Debjyoti Mondal, Rituraj Singh, Subhadarshi Panda, Ankit Mishra, Kiran Pradeep, Srihari K B, Godawari Sud- hakar Rao, and Pushpak Bhattacharyya

    arXiv

  10. [15]

    In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 24048–24060, Suzhou, China

    RG- VQA: Leveraging retriever-generator pipelines for knowledge intensive visual question answering. In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 24048–24060, Suzhou, China. Association for Computational Linguistics. Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soh...

    2025

  11. [16]

    InFind- ings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110, Bangkok, Thailand

    Aligning large multimodal models with factually augmented RLHF. InFind- ings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110, Bangkok, Thailand. Association for Computational Linguistics. Gemma Team

    2024

  12. [17]

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786. Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie

    Pith/arXiv arXiv

  13. [18]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35673–35694, Suzhou, China

    Unconditional truthfulness: Learning unconditional uncertainty of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35673–35694, Suzhou, China. Association for Computational Linguistics. Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip H. S. Torr, and Neel Nanda

    2025

  14. [19]

    Yutong Xie, Zhenglin Hua, Ran Wang, Wing W

    AM- BER: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation.Preprint, arXiv:2311.07397. Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y . Ng, Xizhao Wang, and Yuheng Jia

    Pith/arXiv arXiv

  15. [23]

    Blur” is the Gaussian blur σ, “Noise

    and the iNaturalist subset of Encyclo- pedic VQA (Van Horn et al., 2018; Mensink et al., 2023). PopVQA contains factual questions associ- ated with four broad entity categories: celebrities, landmarks, logos, and paintings. The iNaturalist subset contains fine-grained factual questions about plant and animal species. We initially sample 6,300 image–questi...

    2018

  16. [2018]

    Given that PopVQA and iNaturalist differ in both entity domain and failure-mode distribution, we evaluate whether the attribution probes transfer between the two

    and cross- dataset visual-language distributions (Akula et al., 2021). Given that PopVQA and iNaturalist differ in both entity domain and failure-mode distribution, we evaluate whether the attribution probes transfer between the two. We use the best feature family for each local decision (§4.3) and train probes on PopVQA data only. Table 11 reports PopVQA...

    2021

  17. [2020]

    InProceedings of the 34th International Conference on Neural Information Pro- cessing Systems, NIPS ’20, Red Hook, NY , USA

    Retrieval-augmented generation for knowledge- intensive NLP tasks. InProceedings of the 34th International Conference on Neural Information Pro- cessing Systems, NIPS ’20, Red Hook, NY , USA. Curran Associates Inc. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and...

    2023

  18. [2021]

    InProceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2148–2166, Online and Punta Cana, Dominican Re- public

    CrossVQA: Scalably generating bench- marks for systematically testing VQA generalization. InProceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2148–2166, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi

    2021

  19. [2022]

    InProceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254, Abu Dhabi, United Arab Emirates

    Retrieval augmented visual question answering with outside knowledge. InProceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun

    2022

  20. [2023]

    Association for Computational Linguis- tics

    Can pre-trained vision and language models answer visual information-seeking questions? InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14948–14968, Singapore. Association for Computational Linguis- tics. Zijie Cheng, Ariel Yuhan Ong, Siegfried K. Wagner, David A. Merle, Lie Ju, Hanyuan Zhang, Ruinian Chen...

    2023

  21. [2024]

    InThe Twelfth International Conference on Learning Representations

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhi- fang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong ...

    Pith/arXiv arXiv 2025

  22. [2025]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi

    CultureVLM: Char- acterizing and improving cultural understanding of vision-language models for over 100 countries.arXiv preprint arXiv:2501.01282. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi

    Pith/arXiv arXiv

  23. [2026]

    Accepted at the 43rd International Conference on Machine Learning

    Finding the correct visual evidence without forgetting: Mitigat- ing hallucination in LVLMs via inter-layer visual attention discrepancy.Preprint, arXiv:2605.20965. Accepted at the 43rd International Conference on Machine Learning. Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, an...

    Pith/arXiv arXiv