Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

How Do Vision-Language Models Process Conflicting Information Across Modalities?

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models answer from whichever modality their hidden representations encode more saliently, and individual attention heads can be manipulated—and transferred across datasets—to shift that preference.

desk verdict A plausible and useful head-level story about modality bias in VLMs, but the headline correlation and head-selection are statistically softer than the paper lets on; still, this deserves a serious referee. read the letter →

arxiv 2507.01790 v1 pith:AVSRZS5C submitted 2025-07-02 cs.CL cs.AIcs.CVcs.LG

Tianze Hua , Tian Yun , Ellie Pavlick This is my paper
classification cs.CLcs.AIcs.CVcs.LG
keywords vision-languagemodelsconflictinginputsmodalitybiasrepresentationalsalienceattentionheadinterventionrouterheadsV-Measurecross-modalinterference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models presented with an image and a conflicting caption often answer from the wrong modality, and the paper argues this is not random: a model tends to report whichever modality is encoded more saliently in its hidden representations. Probing shows the models separately encode both modalities and can detect the inconsistency, so the failure is not a simple inability to read the inputs. K-Means clustering with V-Measure shows that the gap between target-modality and non-target-modality salience in late-layer activations predicts accuracy across model–dataset–task triples ($r = 0.94$). The paper then identifies individual attention heads—a modality-agnostic router head and modality-specific promotion heads—that, when scaled, shift that salience and the behavior, and the same heads generalize across datasets. If right, this makes modality preference in multimodal models a localized, manipulable mechanism rather than a fixed property.

What carries the argument

The paper's mechanism has two linked parts. Representational salience is measured by fitting K-Means clustering with as many clusters as there are classes to the hidden representations of inconsistent inputs, then scoring how well the clusters align with image labels versus caption labels using V-Measure; the target-minus-non-target V-Measure difference is the proxy for which modality dominates. The causal part is head intervention: the output of one attention head at the last prompt-token position is multiplied by a scalar $\alpha$ ranging from $-10$ to $10$, and the resulting change in answers classifies the head as a modality-agnostic router head (it promotes whichever modality the prompt requests), an image promotion head, or a caption promotion head. Intervening on the same heads also shifts the V-Measure gap, which is the evidence that the heads restructure the representations rather than only changing the output token.

What would settle it

On Qwen2.5-VL with CIFAR-100, compute the V-Measure gap from an intermediate layer that does not feed the output distribution, then scale the router head L11H14 and check whether the mid-layer gap moves with accuracy; if it does not, the final-layer salience measure is reading off the imminent answer instead of explaining the behavior.

Watch

Extended reading notes

Core claim

The central claim is that when a vision-language model is asked to report one modality from an inconsistent image–caption pair, its behavior tracks the relative representational salience of the two modalities, and that salience is controlled by specific attention heads. The supporting evidence has three layers: linear probes show both image and caption information is encoded and the inconsistency is detectable; K-Means clustering with V-Measure shows the last-layer salience gap between target and non-target modality strongly predicts behavioral accuracy; and scaling individual attention-head outputs by a constant $\alpha$ reveals heads that either promote whichever modality the instruction requests (router heads) or promote one fixed modality (image or caption promotion heads). The router head and promotion heads found on one dataset alter performance in the expected direction on other datasets, and their intervention moves the V-Measure gap in the matching direction. The paper therefore concludes that modality selection in vision-language models is attributable to identifiable, manipulable components.

Load-bearing premise

The load-bearing assumption is that clustering the model's final-layer hidden states measures an internal cause of the model's answer rather than simply echoing the answer the model is about to produce.

Editorial extensions

If this is right

  • Models that fail to reorganize their representations toward the requested modality will systematically under-report that modality, so the V-Measure gap can identify failing model–dataset combinations before behavioral testing.
  • Scaling a single router head found on Pascal VOC improves target-modality accuracy on most other datasets tested, so modality-selection ability is at least partly portable through specific heads.
  • Amplifying an image or caption promotion head helps reporting of that modality and hurts reporting of the other, indicating these heads implement a fixed modality prior.
  • Because the salience gap correlates with accuracy at $r = 0.94$, representational-salience measurements can serve as a proxy for behavior in settings where evaluating many prompts is expensive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same scaling intervention were applied to larger or differently trained multimodal models, one would expect analogous router heads to exist, making this a general recipe for locating modality-selection circuitry rather than a description of one model family.
  • The salience-versus-accuracy logic should transfer to other modality pairs such as audio–text or video–text, where the V-Measure gap could serve as a diagnostic for which stream a model will trust under conflict.
  • A practical extension the paper does not pursue is using router-head amplification as an inference-time steering control to raise target-modality accuracy without fine-tuning or retraining.
  • The fact that consistency is detectable in the representations but not always used suggests future interventions could target the layer where consistency information is integrated, before the answer is formed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how vision-language models (VLMs) process conflicting information across modalities by constructing inconsistent image-caption pairs and prompting models to report information from a designated target modality. The authors evaluate four 7B-scale VLMs on five datasets, finding that models exhibit varying modality biases and that accuracy drops under conflict. They then use linear probes to show that both modality-specific information and inter-modality consistency are encoded in the hidden representations, and apply K-Means clustering with V-Measure to argue that the relative representational salience of the target versus non-target modality predicts behavioral accuracy (Section 3.3, r = 0.94). Finally, they perform attention-head interventions, identifying modality-agnostic 'router heads' and modality-specific 'promotion heads' that, when scaled, shift model behavior toward one modality; they report cross-dataset transfer of three heads selected on Pascal VOC (Sections 4.1-4.3).

Significance. If the causal claims are substantiated, the paper offers a mechanistic account of modality selection in VLMs and a practical handle for controlling it, which would be a meaningful contribution to interpretability and robustness in multimodal models. The strengths are the breadth of behavioral evaluation across four models and five datasets, the inclusion of the reciprocal direction (reporting caption content under conflicting images), the careful construction of inconsistent pairs, and the public release of code. The cross-dataset head-intervention results, despite the caveats discussed below, suggest a potentially generalizable phenomenon. However, the load-bearing mechanistic claims currently rest on statistical and design choices that need to be tightened before the conclusions can be accepted.

major comments (4)
  1. [Section 4.1, Table 1] The head selection and the headline evaluation are performed on the same Pascal VOC data: 100 conflicting image-caption pairs (200 samples) are used both to identify the router/promotion heads and to report the Pascal VOC rows in Table 1. Because heads are also ranked by 'intervenability' (Appendix F.2), a head that happens to produce a large accuracy swing on this small sample will be selected regardless of whether it specifically controls modality routing. This selection-on-the-test-set risk makes the Pascal VOC columns in Table 1 uninterpretable as evidence. Please hold out a separate split for head selection, evaluate on a disjoint test set, report confidence intervals or bootstrap intervals over multiple data samples and random seeds, and include a control condition with randomly chosen heads (or heads selected on a different dataset).
  2. [Section 3.3, Figure 4(b)] The r = 0.94 correlation between V-Measure and behavioral accuracy is computed on the final-layer activations, which feed directly into the output token distribution used to score accuracy. This makes the correlation partly mechanical: the same representation that determines the next-token prediction is clustered and compared with the prediction. The claim that 'the structure of the representations predicts behavior' is therefore not a test of an independent internal cause. Please report the correlation for earlier layers (e.g., layers that do not directly project to the unembedding), or explicitly reframe the result as a descriptive consistency check rather than mechanistic evidence. If the correlation persists in middle layers, the interpretation is considerably stronger.
  3. [Section 4.3, Table 1] The cross-dataset generalization claim rests on only three heads, with no confidence intervals, significance tests, or multiple-seed variation. The single router head L11H14 actually degrades caption accuracy on CUB-Color by 4.9%, which the paper attributes to 'suboptimal image unimodal performance' without a quantitative test. Please provide error bars for all entries in Table 1, evaluate a larger set of candidate heads (including heads that do not meet the classification criteria), and test whether the observed cross-dataset improvements are significantly larger than those from random heads. As written, the exception on CUB-Color and the absence of variance estimates leave the universality of the router-head claim unsubstantiated.
  4. [Appendix F.1 and F.2] The procedure for classifying heads is not fully specified: the monotonicity epsilon is never given, and the final choice of L11H14, L19H26, and L13H26 is described as prioritizing 'intervenability' on the same data used for evaluation. This makes the reported head identities hard to reproduce and contributes to the selection concern in Major Comment 1. Please provide the exact numerical criteria, the full set of heads that meet each criterion, and a held-out validation of the selection rule.
minor comments (5)
  1. [Section 2] Typographical errors: 'thr VLM' should be 'the VLM', and 'ImagetNet100' should be 'ImageNet100'.
  2. [Appendix D.1] The caption of Figure D.1 contains 'acorss', which should be 'across'.
  3. [Related Work] The reference to Tenney et al. is malformed: 'Tenney et al., Belinkov, 2022' is missing the year for the first citation and the formatting should be corrected.
  4. [Section 4.1] The paper reports the number of sampled pairs and total samples (100 and 200) but does not state how many random seeds were used for the intervention sweeps or whether the sample was stratified by class; please add this information for reproducibility.
  5. [Figure 4(a)] The V-Measure curves are averaged over three K-Means initializations, but no variance or error bars are shown; adding them would strengthen the claim that the CIFAR-100 versus Pascal VOC difference is robust.

Circularity Check

2 steps flagged · score 4.0 of 10

Cross-dataset head transfer is genuinely external, but the r=0.94 representational-salience correlation and the same-dataset Pascal VOC head results are partly self-referential.

  1. other [Section 3.3, Figure 4(b)]
    "Figure 4(b) plots the difference in V-Measure of the target and non-target modality against behavioral accuracy on the target modality for everymodel–dataset–modality triplet, with those hidden activations from the last layer of the models. The strong positive correlation (r = 0.94 , p <10−3) confirms that the structure of the representations predicts behavior: models tend to answer from whichever modality their top-layer representations encode most distinctly."

    The V-Measure is computed on final-layer hidden activations, and behavioral accuracy is the next-token prediction produced from exactly those same activations through the output head. The two quantities therefore share their input representation and label set: a final layer whose clusters align with a modality's class labels is the same layer that directly determines whether the model emits that modality's label. The paper presents this partly mechanical self-consistency as an independent confirmation that 'the structure of the representations predicts behavior,' but the correlation does not establish an independent causal window into processing.

  2. fitted input called prediction [Section 4.2, Appendix F.2, Table 1]
    "For the following sections, we pick the attention heads which brings significant performance boosts after the intervention. ... one focuses on the intervenability of the heads – if a head can be intervened to improve unimodal or cross-modal performance beyond the unintervened baseline, we consider that head to be 'intervenable'"

    The heads L11H14, L19H26, and L13H26 are selected on a 100-pair Pascal VOC sample because intervening on them yields the largest accuracy improvements on that sample. Table 1 then reports Pascal VOC accuracy gains for those same heads, so the Pascal VOC row is a selected maximum rather than an out-of-sample prediction. The cross-dataset rows are genuinely external and limit the circularity, but the same-dataset portion of the headline result is statistically forced by the intervenability selection criterion.

full rationale

The paper's cross-dataset transfer results (Pascal VOC heads applied to CIFAR-10, CIFAR-100, ImageNet100, and CUB-Color) provide real external grounding, and the CUB-Color caption drop shows the generalization claim is not vacuous. No load-bearing self-citation or imported uniqueness theorem is used; the head taxonomy is operationalized from intervention patterns rather than derived from prior work. However, two load-bearing presentations are partially circular. First, the r=0.94 correlation between V-Measure on the last layer and behavioral accuracy is a self-consistency check: the final layer directly feeds the output distribution that defines accuracy, so the two measures are not independent evidence of a causal salience mechanism. Second, the Pascal VOC columns of Table 1 report improvements on the same 100-pair sample used to select heads by intervenability, making those rows a selected maximum rather than a prediction. The absence of confidence intervals on the 100-pair intervention sample compounds this, though it is a statistical concern rather than a further circularity. Overall, the central causal claim retains independent content through its out-of-sample transfer evaluations, so the circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The paper's central measurements rest on several standard interpretability assumptions: linear probe accuracy indicates accessible information, K-Means salience indicates driver modality, and attention output scaling isolates head function. None are proven within the paper, and the final-layer V-Measure is the most fragile. Free parameters include k for K-Means, alpha for intervention, and an unspecified trend-tolerance epsilon; alpha is selected on Pascal VOC before cross-dataset testing. No genuinely new physical entities are postulated; router/promotion heads are functional labels for existing attention heads with cross-dataset evidence.

free parameters (3)
  • K-Means cluster count k = |C| = number of classes per dataset (10, 20, 100, 15)
    Chosen per dataset as the number of label classes; V-Measure values are sensitive to k, and no sensitivity analysis over k is reported.
  • Attention output scaling alpha = alpha = 10 for cross-dataset intervention
    Selected from a scan of 21 values on Pascal VOC based on largest performance boost; then fixed at 10 for Tables 1 and 2. This is a hand-chosen intervention strength, not independently predicted.
  • Monotonicity epsilon for head classification = not specified
    Appendix F.1 allows small local drops with an epsilon threshold; threshold value is not reported, so head taxonomy depends on an unspecified tolerance.
assumptions (5)
  • domain assumption High linear probe accuracy implies the model has access to the probed class information during processing
    Section 3.1 concludes from probe accuracy that both modalities are "sufficiently well encoded"; linear probes can decode information a network does not actually use, so this supports availability, not causal use.
  • domain assumption K-Means V-Measure on final-layer activations is a faithful measure of which modality drives behavior
    Section 3.3 uses this as the key proxy; because final-layer activations feed the unembedding that produces the answer token, the proxy is partly confounded with the output itself.
  • domain assumption Scaling one attention head's output by alpha is a clean causal intervention
    Section 4.1 attributes changes in next-token accuracy to the isolated head, ignoring nonlinear interactions with other heads, MLPs, and normalization layers.
  • domain assumption First-token accuracy with shared-first-token filtering measures correct modality reporting
    Section 4.1 and Appendix D score only the first predicted token; full generations could diverge in ways that change the conclusion.
  • domain assumption The four 7B open-weight models are representative of VLM behavior
    Section 2 and Appendix A restrict the study to 7B-scale MLLMs; the authors themselves list model scale as a limitation.
invented entities (2)
  • Modality-agnostic router head (functional role) independent evidence
    purpose: Attention head that promotes the answer of the target modality when amplified
    Introduced as a functional class of existing attention heads; independent evidence comes from cross-dataset transfer results in Table 1, though no circuit-level proof of mechanism is given.
  • Modality-specific promotion head (image or caption) independent evidence
    purpose: Attention head that always promotes one modality's answer regardless of the requested modality
    Functional labels for existing attention heads; cross-dataset effects in Table 1 provide an out-of-sample handle, but the mechanism is not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Vision-Language Models Process Conflicting Information Across Modalities?." pith.science (2026). https://pith.science/paper/AVSRZS5C

@misc{pith2026250701790,
  author       = {Pith},
  title        = {Pith review of: How Do Vision-Language Models Process Conflicting Information Across Modalities?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVSRZS5C}},
  note         = {Machine review of arXiv:2507.01790}
}
read the original abstract

AI models are increasingly required to be multimodal, integrating disparate input streams into a coherent state representation on which subsequent behaviors and actions can be based. This paper seeks to understand how such models behave when input streams present conflicting information. Focusing specifically on vision-language models, we provide inconsistent inputs (e.g., an image of a dog paired with the caption "A photo of a cat") and ask the model to report the information present in one of the specific modalities (e.g., "What does the caption say / What is in the image?"). We find that models often favor one modality over the other, e.g., reporting the image regardless of what the caption says, but that different models differ in which modality they favor. We find evidence that the behaviorally preferred modality is evident in the internal representational structure of the model, and that specific attention heads can restructure the representations to favor one modality over the other. Moreover, we find modality-agnostic "router heads" which appear to promote answers about the modality requested in the instruction, and which can be manipulated or transferred in order to improve performance across datasets and modalities. Together, the work provides essential steps towards identifying and controlling if and how models detect and resolve conflicting signals within complex multimodal environments.

Figures

Figures reproduced from arXiv: 2507.01790 by the authors.

Figure 1
Figure 1. Examples of inconsistent image and caption pairs. Left: image and caption disagrees on the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Model performance on reporting target modality information under unimodal inputs and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of (a) unimodal information probe and (b) consistency probe on Pascal VOC [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Representational salience and behavioral accuracy. (a) Layer-wise V-Measure of Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Examples of different types of attention heads in Qwen2.5-VL found on Pascal VOC. First [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Attending to Multimodal Generation One Token at a Time

    cs.CV 2026-07 accept novelty 7.0 of 10

    MLLMs attend to the relevant modality at the exact generation steps that need it; causal blocking of those peaks breaks answers while timed boosting improves multimodal performance.

  2. Linguistic Context Recodes Visual Representations in Vision-Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Goal-directed language prompts make VLMs add a transferable goal-relevant marker to selected image objects and amplify those objects' queried attributes in later layers, and both effects causally influence answers.

  3. Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.

Reference graph

Works this paper leans on

41 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Multimodal biomedical ai

    Juli \'a n N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical ai. Nature medicine, 28 0 (9): 0 1773--1784, 2022

    work page 2022

  2. [2]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016

    arXiv 2016

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    work page 2022

  4. [4]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  5. [5]

    Probing classifiers: Promises, shortcomings, and advances

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 0 (1): 0 207--219, March 2022. doi:10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7/

    doi:10.1162/coli_a_00422 2022

  6. [6]

    On the robustness of large multimodal models against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625--24634, 2024

    work page 2024

  7. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Proc...

    work page 2023

  8. [8]

    Words or vision: Do vision-language models have blind faith in text?, 2025

    Ailin Deng, Tri Cao, Zhirui Chen, and Bryan Hooi. Words or vision: Do vision-language models have blind faith in text?, 2025. URL https://arxiv.org/abs/2503.02199

    arXiv 2025

Show all 41 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee, 2009

    2009

  2. [10]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 0 303--338, 06 2010. doi:10.1007/s11263-009-0275-4

    2010 doi

  3. [11]

    Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts

    Michal Golovanevsky, William Rudman, Michael Lepori, Amir Bar, Ritambhara Singh, and Carsten Eickhoff. Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts. arXiv preprint arXiv:2505.17127, 2025 a

    2025

  4. [12]

    What do VLM s NOTICE ? a mechanistic interpretability pipeline for G aussian-noise-free text-image corruption and evaluation

    Michal Golovanevsky, William Rudman, Vedant Palit, Carsten Eickhoff, and Ritambhara Singh. What do VLM s NOTICE ? a mechanistic interpretability pipeline for G aussian-noise-free text-image corruption and evaluation. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceed...

    2025

  5. [13]

    How does GPT -2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT -2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=p4PckNQR8k

    2023

  6. [14]

    Michael V Heinz, Daniel M Mackin, Brianna M Trudeau, Sukanya Bhattacharya, Yinzhou Wang, Haley A Banta, Abi D Jewett, Abigail Salzhauer, Tess Griffin, and Nicholas C Jacobson. Evaluating therabot: A randomized control trial investigating the feasibility and effectiveness of a ...

    2024

  7. [15]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  8. [16]

    Learning multiple layers of features from tiny images.(2009), 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.(2009), 2009

    2009

  9. [17]

    Beyond the doors of perception: Vision transformers represent relations between objects

    Michael Lepori, Alexa Tartaglini, Wai Keen Vong, Thomas Serre, Brenden M Lake, and Ellie Pavlick. Beyond the doors of perception: Vision transformers represent relations between objects. Advances in Neural Information Processing Systems, 37: 0 131503--131544, 2024

    2024

  10. [18]

    LL a VA -onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LL a VA -onevision: Easy visual task transfer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/for...

    2025

  11. [19]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id...

    2023

  12. [20]

    Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness, 2025

    Chenfei Liao, Kaiyu Lei, Xu Zheng, Junha Moon, Zhixiong Wang, Yixuan Wang, Danda Pani Paudel, Luc Van Gool, and Xuming Hu. Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness, 2025. URL https://arxiv.org/abs/2503.18445

    2025 arXiv

  13. [21]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  14. [22]

    The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability

    Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv pr...

    2024

  15. [23]

    Towards interpreting visual information processing in vision-language models

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chanJGoa7f

    2025

  16. [24]

    Same task, different circuits: Disentangling modality-specific mechanisms in vlms

    Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. Same task, different circuits: Disentangling modality-specific mechanisms in vlms. arXiv preprint arXiv:2506.09047, 2025

    2025

  17. [25]

    Introducing operator

    OpenAI . Introducing operator. https://openai.com/index/introducing-operator/, January 2025. Published January 23, 2025; accessed 2025-05-14

    2025

  18. [26]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  19. [27]

    Interpreting the linear structure of vision-language model embedding spaces

    Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Naomi Saphra, Sham Kakade, and Stephanie Gil. Interpreting the linear structure of vision-language model embedding spaces. arXiv preprint arXiv:2504.11695, 2025

    2025 arXiv

  20. [28]

    V -measure: A conditional entropy-based external cluster evaluation measure

    Andrew Rosenberg and Julia Hirschberg. V -measure: A conditional entropy-based external cluster evaluation measure. In Jason Eisner, editor, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning...

    2007

  21. [29]

    On the Adversarial Robustness of Multi-Modal Foundation Models

    Christian Schlarmann and Matthias Hein. On the Adversarial Robustness of Multi-Modal Foundation Models . In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3679--3687, Los Alamitos, CA, USA, October 2023. IEEE Computer Society. doi:10.1109/IC...

    2023

  22. [30]

    What do you learn from context? probing for sentence structure in contextualized word representations

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conferen...

  23. [31]

    Bert rediscovers the classical nlp pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, 2019

    2019

  24. [32]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Ne...

    2020

  25. [33]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez, editors, Proceedings of the 57th Annual Meeti...

    2019 doi

  26. [34]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Jul 2011

    2011

  27. [35]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  28. [36]

    Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models, 2025

    Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang, Yang Zhao, Xinze Guan, Ching-Chen Kuo, and Xin Eric Wang. Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models, 2025. URL https://arxiv.org/abs/2502.16033

    2025 arXiv

  29. [37]

    Characterizing mechanisms for factual recall in language models

    Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924--9959, Singapor...

    2023 doi

  30. [38]

    Does vision-and-language pretraining improve lexical grounding? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4357--4366, 2021

    Tian Yun, Chen Sun, and Ellie Pavlick. Does vision-and-language pretraining improve lexical grounding? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4357--4366, 2021

    2021

  31. [39]

    Emergence of abstract state representations in embodied sequence modeling

    Tian Yun, Zilai Zeng, Kunal Handa, Ashish Thapliyal, Bo Pang, Ellie Pavlick, and Chen Sun. Emergence of abstract state representations in embodied sequence modeling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12190--12205, 2023

    2023

  32. [40]

    Calling a spade a heart: Gaslighting multimodal large language models via negation, 2025

    Bin Zhu, Huiyan Qi, Yinxuan Gui, Jingjing Chen, Chong-Wah Ngo, and Ee-Peng Lim. Calling a spade a heart: Gaslighting multimodal large language models via negation, 2025. URL https://arxiv.org/abs/2501.19017

    2025

  33. [41]

    Unraveling cross-modality knowledge conflict in large vision-language models

    Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. Unraveling cross-modality knowledge conflict in large vision-language models. arXiv preprint arXiv:2410.03659, 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.