Pith. sign in

REVIEW 2 major objections 5 minor 73 references

Multimodal models encode coarse visual evidence they fail to act on; the vision-versus-prior trade-off is a controllable post-perceptual property.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:08 UTC pith:RXKEBUEL

load-bearing objection An empirically strong paper whose headline 'present but not used' claim is probably right but not yet airtight; worth referee time. the 2 major comments →

arxiv 2607.26326 v1 pith:RXKEBUEL submitted 2026-07-28 cs.CV

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Jiaang Li , Chengzu Li , Zhaochong An , Yifei Yuan , Xi Liu , Serge Belongie , V\'esteinn Sn{\ae}bjarnarson This is my paper
classification cs.CV
keywords multimodal large language modelsvisual context sensitivityperception vs utilizationcounterfactual benchmarkimage reconstruction probingactivation patchingsteering vectorlanguage prior conflict
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a multimodal model answers 'four' to a picture of a five-legged horse, either the visual signal never reached the reasoning stage or it arrived but was overridden by what the model knows. This paper argues that, for coarse attributes such as color, count, size, weight, and spatial-temporal relations, the signal does arrive: counterfactual attributes can be reconstructed from final-layer image tokens of frozen models. The failures are post-perceptual. The paper then shows this vision-versus-prior sensitivity is unstable, can be improved by fine-tuning and by a single learned steering vector used without instructions, and is much easier to control when the counterfactual is stated as text than when it appears in an image.

Core claim

The central claim is that for coarse-grained attributes, the visual evidence survives all the way through the language backbone: a decoder trained only on real-world images can redraw counterfactual attributes from the final-layer image tokens of frozen multimodal models, so the contested content was present and available. What is missing is a consistent, controllable policy for using or ignoring that evidence. The paper introduces WhatIfVis, a benchmark of 3,049 manually verified counterfactual image–text pairs, and shows that vanilla models rarely satisfy both 'follow the image' and 'ignore the image' intents on the same example. Supervised fine-tuning on a single attribute family transfer

What carries the argument

Reconstruction probe: a connector and image decoder trained only on real-world images redraw a scene from final-layer image tokens; because counterfactuals never appear in training, a correct counterfactual reconstruction shows the attribute was genuinely encoded in those tokens. WhatIfVis: 3,049 manually verified counterfactual image/text pairs across five attribute types, each with context and prior answers and strict conjunctive PairAcc metrics requiring both intents to succeed. Activation patching: replacing attention outputs at the last-token position in a contiguous layer band to locate where the vision-versus-prior trade-off is resolved. Learned steering subspace: a unit vector in the

Load-bearing premise

The reconstruction probe must be a faithful readout of what the final-layer image tokens contain: a decoder trained only on real images could, in principle, draw a plausible counterfactual from general object priors or low-level artifacts, and the 'seen but unused' conclusion would then not follow.

What would settle it

Randomize or replace the final-layer image tokens with tokens from an unrelated image, run the same connector and decoder, and check whether counterfactual attributes (e.g., an anomalous leg count) are still reconstructed at the same rate; if they are, the probe is manufacturing attributes from decoder priors rather than reading the tokens, and the paper's separation of perception from utilization fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For coarse attributes, failures on visual-prior conflicts will not be fixed by stronger vision encoders, because the evidence is already present in the LLM backbone; control interventions are the relevant remedy.
  • Visual context sensitivity is a learnable, general skill: supervised fine-tuning on only the spatial-temporal task lifts controllability on held-out color and size tasks, though count and weight remain resistant because they require reasoning beyond the pixels.
  • The vision-versus-prior trade-off is localized to a narrow band of layers at architecture-specific depths, so targeted activation editing can modulate the behavior without wholesale retraining.
  • A single learned steering vector, applied with no intent instruction, improves controllability over the fully prompted vanilla model, while a random vector in the same subspace disrupts it.
  • The modality gap in controllability is almost entirely a failure to follow visual evidence, not a failure to override it, and this gap widens with model scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reconstruction probe is accepted, many MLLM hallucinations and visual-prior conflicts become partly a control problem, which suggests that benchmarks separating perception from utilization would be more informative than end-to-end accuracy alone.
  • A natural extension is to apply the same steering approach to fine-grained attributes or small objects; the paper only claims the coarse-grained regime, so the boundary where genuine perception failure begins is left open.
  • The text-vs-image asymmetry points to a concrete training objective: make visual evidence behave more like a stated sentence. A testable prediction is that interventions which close the context-intent gap on images will also close the overall pair-accuracy gap.
  • The steering vector could serve as a lightweight post-hoc calibration tool for deployed models, since it requires no prompt change and no weight update once learned.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper investigates whether multimodal LLMs fail on visual questions because they do not perceive the evidence (H1) or because they perceive it but cannot reliably control their reliance on it (H2). It proposes a reconstruction probe that decodes images from the final-layer image tokens of frozen MLLMs, introduces the WhatIfVis benchmark of counterfactual visual questions with paired textual statements, and develops an activation-patching and subspace-steering framework (CVCS) to localize and control the vision-versus-prior trade-off. The authors report three findings: (i) coarse-grained attributes (count in the main text; color, size, weight in the appendix) are decodable from the final-layer image tokens, which they interpret as evidence that H1 is false for these attributes; (ii) vanilla models show unstable visual context sensitivity that is substantially improved by SFT on a single task and that transfers across tasks; and (iii) a learned one-dimensional steering vector, applied without any intent instruction, improves controllability over the vanilla baseline. The paper argues that the bottleneck is post-perceptual utilization, not perceptual encoding.

Significance. If the central inference holds, the paper makes a valuable contribution by relocating the failure mode of MLLMs from perception to utilization, and by providing a reusable benchmark (WhatIfVis) and a control method (CVCS) that could inform future interpretability and intervention work. The empirical scope is unusually broad for a study of this type: six models spanning three architecture families, two size tiers each, with SFT, activation patching, and steering evaluated on both image and text channels. The paper is also commendably honest in framing the steering vector as a reachable control rather than a pre-existing discovery, and it includes a random-rank-1 control for the steering experiment. The benchmark curation is careful, with human verification and matched text/image pairs. However, the paper's headline claim depends on the faithfulness of the reconstruction probe, and that assumption is not yet sufficiently controlled; this is the main weakness.

major comments (2)
  1. [Section 3, Tables 1–2] The central inference from 'the counterfactual attribute is reconstructed' to 'the attribute is present in the backbone' rests on the assertion that the connector and decoder, trained only on real images, are a faithful readout 'not a decoder manufacturing the attribute from its prior.' This assertion is not adequately supported. The decoder SANA is pretrained on massive image-text data and can plausibly generate counterfactual objects (e.g., a five-legged horse) even from weak or proxy signals, and fine-tuning on ImageNet does not erase that prior. The cross-reference in Table 2 shows that reconstructions from counterfactual tokens are closer to the counterfactual than to the matched real image, which is good evidence that the tokens distinguish the two conditions, but it does not rule out the decoder using a correlated artifact (pose, background, texture) rather than the attribute itse
  2. [Appendix D, Table 4; Abstract/Conclusion] The main text reports the reconstruction evidence only for the Count attribute, which is the only one with matched natural pairs. Appendix D extends to Color, Size, and Weight, but with only 50 samples per attribute and no paired real-image control. The abstract and conclusion nevertheless state that failures on 'these attributes' (spatial-temporal, color, count, size, weight) point to post-perceptual utilization. This generalization is under-supported for the three additional attributes, especially Size and Weight, which the paper itself classifies as requiring reasoning beyond directly visible features and which have the lowest reconstruction fidelity (SSIM 0.64–0.72). At minimum, please either provide matched-pair cross-reference controls for the additional attributes or restrict the headline claim to the attributes with direct evidence. The current wording overstates the reach of the
minor comments (5)
  1. [Section 3, first paragraph] The phrase 'the recovery is a faithful readout... not a decoder manufacturing the attribute from its prior' is a strong claim that is repeated almost verbatim but never empirically demonstrated. Even if a control is added, please soften the wording to match the strength of the evidence.
  2. [Section 2, Eq. (4)] The notation c(w) is introduced as a 'model-specific scalar for each intent w', but the text later says 'per-model multipliers tuned on the Spatial-Temporal validation set.' Please clarify whether c(w) is a single scalar per intent or a per-layer/per-head set of multipliers, and report how it is tuned (grid range, selection criterion).
  3. [Section 4.3, Figure 5 caption] The caption says 'All pair-accuracies are probabilities unless noted.' It would help to state explicitly that the probability scorer in Eq. (6) is used, and to define 'probability scorer' in the caption or in a footnote, since the figure is one of the most important in the paper.
  4. [Throughout] Typos and grammatical errors: 'the itunreliably' (Introduction), 'We then analysis this downstream behavior' (Conclusions), 'a issue ofvisual context sensitivity' (Introduction), 'Contex t-domina nt' (Figure 9), and inconsistent capitalization of 'SANA' / 'Sana-0.6B' in Appendix C. Please copyedit.
  5. [Appendix B, Table 3] The total sample count is 3,049, but the sum of the five task counts is 1,537+282+390+773+67 = 3,049. That is internally consistent; no issue. However, the text in Section 4.2 says 'With 3,049 manually-inspected counterfactual samples' — please clarify that only the generated subset and a portion of the existing subsets were manually verified with the double-top-rating criterion, as described in Appendix B, since 'manually-inspected' could be read as applying to all 3,049.

Circularity Check

0 steps flagged

No significant circularity; central derivation is self-contained and OOD-validated.

full rationale

The paper's derivation chain is not circular. Section 3's reconstruction probe is an empirical measurement: the connector and decoder are trained only on real ImageNet images, and counterfactual recovery is assessed on held-out counterfactual inputs. The statement that recovery is a 'faithful readout' is a measurement assumption vulnerable to SANA prior leakage, but it is not equivalent to the conclusion by construction. Section 4's WhatIfVis benchmark is a new task with matched text statements and conjunctive metrics; no fitted quantity is renamed as a prediction. Section 5's steering vector is trained to maximize source-answer probability, but the paper explicitly disclaims the pre-existing-discovery interpretation: 'Since u is learned rather than pre-existing, the claim is not that the vanilla model was already steerable but that the vision-versus-prior trade-off is a reachable, controllable vector.' The OOD transfer to four held-out tasks, the random rank-1 control, and seed stability provide independent support. The only overlapping-author citation (Stoehr et al. 2024, used for the last-token patching location) is minor and not load-bearing. No self-definitional, fitted-input-as-prediction, uniqueness-imported, or ansatz-smuggled-via-self-citation step is present.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claim rests on the reconstruction probe's validity and on the assumption that coarse attributes are representable; the only fitted scalars are per-model steering multipliers and a patching threshold. No new physical or ontological entities are introduced; CVCS is a name for an adapted framework and the steering vector is a learned parameter.

free parameters (2)
  • c_ctx/c_prior (per-model steering multipliers) = not reported; tuned per model on the Spatial-Temporal validation set
    Static steering (Eq. 4) sets the one-dimensional intent component to a scalar c(w); the reported 37.7% steering result depends on these tuned scalars.
  • Activation patching threshold = 0.85
    Appendix G: the contiguous layer band is selected by a binary range search with threshold 0.85; the localization claim depends on this choice.
axioms (5)
  • domain assumption Coarse attributes (identity, color, count, size, weight) are reliably encoded by vision encoders, unlike fine-grained textures/small objects.
    Section 3 restricts the study to this regime to make the H1 vs H2 separation interpretable.
  • domain assumption A decoder trained only on real images can serve as a faithful probe of information present in final-layer image tokens.
    Section 3: 'the recovery is a faithful readout... not a decoder manufacturing the attribute from its prior.' The H1/H2 separation depends on this.
  • domain assumption Last-token residual-stream and MHA positions are the correct locus for the intent/context circuit.
    Section 4.4 follows Minder et al. (2025) and prior causal tracing; patching at this locus grounds the localization and steering results.
  • domain assumption SFT on one task split (Spatial-Temporal) teaches a general 'be steerable by visual context' capability.
    Section 4.3 trains only on Spatial-Temporal and evaluates on four held-out tasks; the transfer claim assumes the capability is not task-specific.
  • domain assumption The matched text statement is an instruction-following upper bound and a same-mechanism reference for the visual channel.
    Section 6 and Appendix H explicitly state this; the reported text-vs-image gap is interpreted under this assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 25761 in / 14061 out tokens · 130267 ms · 2026-08-01T00:08:00.877414+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Figures

Figures reproduced from arXiv: 2607.26326 by Chengzu Li, Jiaang Li, Serge Belongie, V\'esteinn Sn{\ae}bjarnarson, Xi Liu, Yifei Yuan, Zhaochong An.

Figure 1
Figure 1. Figure 1: One MLLM, two opposite failures. A single model lacks a stable setpoint on the visual context￾sensitivity axis between its parametric prior and the image, giving rise to two opposite failure modes: (Left) Prior-dominant failures, where strong parametric priors overwhelm anomalous visual input; and (Right) Context￾dominant failures, where the model over-indexes on visual features even when instructed to ign… view at source ↗
Figure 2
Figure 2. Figure 2: The counterfactual attribute survives to the final layer and is visible in the reconstruction. Species identity, pose, structure, and dominant colors are preserved, including the counterfactual attribute (e.g. the anomalous limb count), while high-frequency texture softens [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation pipeline for controllable multimodal context sensitivity. We probe a model’s sensitivity to visual and textual contexts separately through targeted instruction following. Given a visual (left) or textual (right) context, the model must answer a question by either strictly grounding its reasoning in the explicit context ("Consider only...") or suppressing the context to recall standard world know… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the five WhatIfVis tasks across text and vision modalities. Each WhatIfVis example pairs a counterfactual image with a matched text statement asserting the same content. Contextual answers (CTX) are inferred from the visual evidence in the input image, whereas prior answers (PRI) rely on the model’s internal world knowledge. This distinction can reveal the visual context preference of MLLMs… view at source ↗
Figure 5
Figure 5. Figure 5: SFT lifts visual controllability across every model and task. Top row: pair-accuracy per task. Bottom row: the same runs as context-accuracy (x) against prior-accuracy (y); points above the diagonal follow the prior over the image. SFT lifts controllability consistently, with the largest gains on Color and Spatial-Temporal and the smallest on Weight and Count. All pair-accuracies are probabilities unless n… view at source ↗
Figure 6
Figure 6. Figure 6: Activation patching localizes the vision-versus-prior trade-off to a narrow layer band. The trade￾off initiates within the patched window (marked) and amplifies sharply in downstream layers. ∆(Answer probability)=Sourceprobability − Targetprobability, which is the larger the better patched. to near zero for weight and count tasks. On color subset, however, the strongest models approach 50. The remaining tw… view at source ↗
Figure 7
Figure 7. Figure 7: An instruction-free steering knob recovers controllability over the vanilla model, and the visual channel stays harder to control than text. Per-model macro pair-accuracy (probability scorer); left: image, right: text. Steering lifts every model; text is uniformly higher (Appendix E). for the source answer and red for the target answer, with the shaded band marking the patched layers, where ∆ is the probab… view at source ↗
Figure 8
Figure 8. Figure 8: Metaquery Framework. Without learnable queries, we adapt the Metaquery to only use the image tokens of the final layers of a MLLM [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The SFT lift reproduces under exact-match at lower absolute values. Per-task pair-accuracy (exact￾match scorer, image channel). Orange: baseline (with intent instructions); blue: SFT (LoRA, trained on Spatial￾Temporal only; all other tasks are out-of-distribution). The qualitative pattern, with large gains on the perceivable attributes and marginal gains on the reasoning-heavy ones, matches the probability… view at source ↗
Figure 10
Figure 10. Figure 10: Cross-direction patching curves for Qwen3.5-4B (image channel). ∆(answer probability) = source − target probability across layers; the shaded band marks the patched layers. These curves complement the main results in [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Cross-direction patching curves for Gemma-4-E2B-IT (image channel). ∆(answer probability) = source − target probability across layers; the shaded band marks the patched layers. These curves complement the main results in [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cross-direction patching curves for the Qwen2.5-VL models (image channel). Each pair shows both directions (ctx→pri and pri→ctx). ∆(answer probability) = source − target probability across layers; the shaded band marks the patched layers. These curves complement the main results in [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Pair-accuracy on both channels for the Spatial-Temporal dataset. Baseline, SFT, and steering results are shown for the image (left) and text (right) channels. Qwen2.5-VL 3B Qwen2.5-VL 7B Qwen3.5 4B Qwen3.5 9B Gemma-4 E2B Gemma-4 E4B Color Image Setting — Pair Accuracy (Probability) Baseline (w/ intent instruction) SFT (w/ intent instruction) Steering (w/o instruction) Qwen2.5-VL 3B Qwen2.5-VL 7B Qwen3.5 4… view at source ↗
Figure 14
Figure 14. Figure 14: Per-task pair-accuracy on both channels for the Color dataset. Baseline, SFT, and steering per task are displayed. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Pair-accuracy on both channels for the Size dataset. The image (left) and text (right) channels are compared across Baseline, SFT, and steering. Qwen2.5-VL 3B Qwen2.5-VL 7B Qwen3.5 4B Qwen3.5 9B Gemma-4 E2B Gemma-4 E4B Count Image Setting — Pair Accuracy (Probability) Baseline (w/ intent instruction) SFT (w/ intent instruction) Steering (w/o instruction) Qwen2.5-VL 3B Qwen2.5-VL 7B Qwen3.5 4B Qwen3.5 9B G… view at source ↗
Figure 16
Figure 16. Figure 16: Pair-accuracy on both channels for the Count dataset. Baseline, SFT, and steering results are shown. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Pair-accuracy on both channels for the Weight dataset. The image and text channels are compared for Baseline, SFT, and steering. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 3 canonical work pages

  1. [1]

    von Helmholtz, Hermann , year =

  2. [2]

    arXiv preprint arXiv:2607.02770 , year =

    Pith/arXiv arXiv

  3. [3]

    2025 , url=

    Shuai Bai and Keqin Chen and Xuejing Liu and Jialin Wang and Wenbin Ge and Sibo Song and Kai Dang and Peng Wang and Shijie Wang and Jun Tang and Humen Zhong and Yuanzhi Zhu and Mingkun Yang and Zhaohai Li and Jianqiang Wan and Pengfei Wang and Wei Ding and Zheren Fu and Yiheng Xu and Jiabo Ye and Xi Zhang and Tianbao Xie and Zesen Cheng and Hang Zhang and...

    2025

  4. [4]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Learning Transferable Visual Models From Natural Language Supervision , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  5. [5]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages=

    Sigmoid Loss for Language Image Pre-Training , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages=. 2023 , url=

    2023

  6. [6]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Improved Baselines with Visual Instruction Tuning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages=. 2024 , url=

    2024

  7. [7]

    (No Title) , year=

    In the blink of an eye: how vision sparked the big bang of evolution , author=. (No Title) , year=

  8. [8]

    2022 , editor =

    Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven , booktitle =. 2022 , editor =

    2022

  9. [9]

    Flamingo:

    Jean-Baptiste Alayrac and Jeff Donahue and Pauline Luc and Antoine Miech and Iain Barr and Yana Hasson and Karel Lenc and Arthur Mensch and Katherine Millican and Malcolm Reynolds and Roman Ring and Eliza Rutherford and Serkan Cabi and Tengda Han and Zhitao Gong and Sina Samangooei and Marianne Monteiro and Jacob Menick and Sebastian Borgeaud and Andrew B...

    2022

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Obelics: An open web-scale filtered dataset of interleaved image-text documents , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    Advances in Neural Information Processing Systems , editor =

    Visual Instruction Tuning , author =. Advances in Neural Information Processing Systems , editor =. 2023 , url =

    2023

  12. [12]

    2023 , editor =

    Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven , booktitle =. 2023 , editor =

    2023

  13. [13]

    arXiv preprint arXiv:2502.14786 , year=

    Tschannen, Michael and Gritsenko, Alexey and Wang, Xiao and Naeem, Muhammad Ferjad and Alabdulmohsin, Ibrahim and Parthasarathy, Nikhil and Evans, Talfan and Beyer, Lucas and Xia, Ye and Mustafa, Basil and H. arXiv preprint arXiv:2502.14786 , year=

    Pith/arXiv arXiv

  14. [14]

    2024 , url=

    Wu, Penghao and Xie, Saining , booktitle=. 2024 , url=

    2024

  15. [15]

    Eyes Wide Shut?

    Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining , booktitle=. Eyes Wide Shut?. 2024 , url=

    2024

  16. [16]

    2025 , url=

    Yang, Le and Zheng, Ziwei and Chen, Boxu and Zhao, Zhengyu and Lin, Chenhao and Shen, Chao , booktitle=. 2025 , url=

    2025

  17. [17]

    2026 , url=

    Lin, Yangguang and Fang, Quan and Li, Yufei and Sun, Jiachen and Gao, Junyu and Sang, Jitao , booktitle=. 2026 , url=

    2026

  18. [18]

    arXiv preprint arXiv:2603.03276 , year=

    Beyond language modeling: An exploration of multimodal pretraining , author=. arXiv preprint arXiv:2603.03276 , year=

    arXiv

  19. [19]

    The Fourteenth International Conference on Learning Representations , year=

    Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models , author=. The Fourteenth International Conference on Learning Representations , year=

  20. [20]

    arXiv preprint arXiv:2512.02014 , year=

    Liu, Zhiheng and Ren, Weiming and Liu, Haozhe and Zhou, Zijian and Chen, Shoufa and Qiu, Haonan and Huang, Xiaoke and An, Zhaochong and Yang, Fanny and Patel, Aditya and Atliha, Viktar and Ng, Tony and Han, Xiao and Zhu, Chuyan and Zhang, Chenyang and Liu, Ding and Perez-Rua, Juan-Manuel and He, Sen and Schmidhuber, J. arXiv preprint arXiv:2512.02014 , year=

    arXiv

  21. [21]

    2024 , doi =

    Tong, Shengbang and Brown, Ellis and Wu, Penghao and Woo, Sanghyun and Middepogu, Manoj and Akula, Sai Charitha and Yang, Jihan and Yang, Shusheng and Iyer, Adithya and Pan, Xichen and Wang, Austin and Fergus, Rob and LeCun, Yann and Xie, Saining , booktitle =. 2024 , doi =

    2024

  22. [22]

    arXiv preprint arXiv:2510.11690 , year=

    Diffusion Transformers with Representation Autoencoders , author=. arXiv preprint arXiv:2510.11690 , year=

    Pith/arXiv arXiv

  23. [23]

    Hidden in plain sight:

    Stephanie Fu and Tyler Bonnen and Devin Guillory and Trevor Darrell , booktitle=. Hidden in plain sight:. 2025 , url=

    2025

  24. [24]

    2026 , url=

    Liu, Zhiheng and Ren, Weiming and Huang, Xiaoke and Chen, Shoufa and Li, Tianhong and Chen, Mengzhao and Ji, Yatai and He, Sen and Schult, Jonas and Zeng, Belinda and Xiang, Tao and Chen, Wenhu and Luo, Ping and Zettlemoyer, Luke and Cong, Yuren , journal=. 2026 , url=

    2026

  25. [25]

    Context-faithful Prompting for Large Language Models

    Zhou, Wenxuan and Zhang, Sheng and Poon, Hoifung and Chen, Muhao. Context-faithful Prompting for Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.968

    doi:10.18653/v1/2023.findings-emnlp.968 2023

  26. [26]

    The Fourteenth International Conference on Learning Representations , year=

    Vision Language Models are Biased , author=. The Fourteenth International Conference on Learning Representations , year=

  27. [27]

    The Eleventh International Conference on Learning Representations , year=

    Quantifying Memorization Across Neural Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  28. [28]

    Copyright Violations and Large Language Models

    Karamolegkou, Antonia and Li, Jiaang and Zhou, Li and S gaard, Anders. Copyright Violations and Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.458

    doi:10.18653/v1/2023.emnlp-main.458 2023

  29. [29]

    Findings of the association for computational linguistics: ACL 2023 , pages=

    Large language models with controllable working memory , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    2023

  30. [30]

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

    Leng, Sicong and Zhang, Hang and Chen, Guanzheng and Li, Xin and Lu, Shijian and Miao, Chunyan and Bing, Lidong. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024

    2024

  31. [31]

    The Thirteenth International Conference on Learning Representations , year=

    Controllable Context Sensitivity and the Knob Behind It , author=. The Thirteenth International Conference on Learning Representations , year=

  32. [32]

    Glass and Pengcheng He , booktitle=

    Yung-Sung Chuang and Yujia Xie and Hongyin Luo and Yoon Kim and James R. Glass and Pengcheng He , booktitle=. 2024 , url=

    2024

  33. [33]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  34. [34]

    Publications Manual , year = "1983", publisher =

    1983

  35. [35]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  36. [36]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  37. [37]

    Dan Gusfield , title =. 1997

    1997

  38. [38]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  39. [39]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  40. [40]

    2024 , doi=

    Liu, Yuan and Duan, Haodong and Zhang, Yuanhan and Li, Bo and Zhang, Songyang and Zhao, Wangbo and Yuan, Yike and Wang, Jiaqi and He, Conghui and Liu, Ziwei and Chen, Kai and Lin, Dahua , booktitle=. 2024 , doi=

    2024

  41. [41]

    2024 , url=

    Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and Stevens, Samuel and Jiang, Dongfu and Ren, Weiming and Sun, Yuxuan and Wei, Cong and Yu, Botao and Yuan, Ruibin and Sun, Renliang and Yin, Ming and Zheng, Boyuan and Yang, Zhenzhu and Liu, Yibo and Huang, Wenhao and Sun, Huan and Su, Yu and Chen, Wenhu , booktit...

    2024

  42. [42]

    Transfer between Modalities with

    Pan, Xichen and Shukla, Satya Narayan and Singh, Aashu and Zhao, Zhuokai and Mishra, Shlok Kumar and Wang, Jialiang and Xu, Zhiyang and Chen, Jiuhai and Li, Kunpeng and Juefei-Xu, Felix and Hou, Ji and Xie, Saining , journal=. Transfer between Modalities with. 2025 , url=

    2025

  43. [43]

    2025 , url=

    Enze Xie and Junsong Chen and Junyu Chen and Han Cai and Haotian Tang and Yujun Lin and Zhekai Zhang and Muyang Li and Ligeng Zhu and Yao Lu and Song Han , booktitle=. 2025 , url=

    2025

  44. [44]

    2026 , url=

    Hao Tang and Chen-Wei Xie and Xiaoyi Bao and Tingyu Weng and Pandeng Li and Yun Zheng and Liwei Wang , booktitle=. 2026 , url=

    2026

  45. [45]

    2025 , url=

    Hongyang Wei and Baixin Xu and Hongbo Liu and Size Wu and Jie Liu and Yi Peng and Peiyu Wang and Zexiang Liu and Jingwen He and Yidan Xietian and Chuanxin Tang and Zidong Wang and Yichen Wei and Liang Hu and Boyi Jiang and Wei Li and Ying He and Yang Liu and Xuchen Song and Yangguang Li and Yahui Zhou , journal=. 2025 , url=

    2025

  46. [46]

    Auto-encoding variational

    Kingma, Diederik P and Welling, Max , journal=. Auto-encoding variational. 2013 , url=

    2013

  47. [47]

    International Conference on Learning Representations , volume=

    Deep compression autoencoder for efficient high-resolution diffusion models , author=. International Conference on Learning Representations , volume=

  48. [48]

    The Fourteenth International Conference on Learning Representations , year=

    Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris Coll-Vinent and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman R. The Fourteenth Internation...

  49. [49]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  50. [50]

    The Fourteenth International Conference on Learning Representations , year=

    Jiaang Li and Yifei Yuan and Wenyan Li and Mohammad Aliannejadi and Daniel Hershcovich and Anders S. The Fourteenth International Conference on Learning Representations , year=

  51. [51]

    VL ind-Bench: Measuring Language Priors in Large Vision-Language Models

    Lee, Kang-il and Kim, Minbeom and Yoon, Seunghyun and Kim, Minsung and Lee, Dongryeol and Koh, Hyukhun and Jung, Kyomin. VL ind-Bench: Measuring Language Priors in Large Vision-Language Models. Findings of the Association for Computational Linguistics: NAACL 2025. 2025. doi:10.18653/v1/2025.findings-naacl.231

    doi:10.18653/v1/2025.findings-naacl.231 2025

  52. [52]

    2025 , url=

    Chenfei Wu and Jiahao Li and Jingren Zhou and Junyang Lin and Kaiyuan Gao and Kun Yan and Sheng-ming Yin and Shuai Bai and Xiao Xu and Yilei Chen and Yuxiang Chen and Zecheng Tang and Zekai Zhang and Zhengyi Wang and An Yang and Bowen Yu and Chen Cheng and Dayiheng Liu and Deqing Li and Hang Zhang and Hao Meng and Hu Wei and Jingyuan Ni and Kai Chen and K...

    2025

  53. [53]

    and Bar, Amir and Singh, Ritambhara and Eickhoff, Carsten

    Golovanevsky, Michal and Rudman, William and Lepori, Michael A. and Bar, Amir and Singh, Ritambhara and Eickhoff, Carsten. Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1262

    doi:10.18653/v1/2025.emnlp-main.1262 2025

  54. [54]

    ROME : Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

    Zhou, Kankan and Lai, Eason and Yeong, Wei Bin Au and Mouratidis, Kyriakos and Jiang, Jing. ROME : Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.683

    work page doi:10.18653/v1/2023.findings-emnlp.683 2023

  55. [55]

    Characterizing Mechanisms for Factual Recall in Language Models

    Yu, Qinan and Merullo, Jack and Pavlick, Ellie. Characterizing Mechanisms for Factual Recall in Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.615

    doi:10.18653/v1/2023.emnlp-main.615 2023

  56. [56]

    Activation Scaling for Steering and Interpreting Language Models

    Stoehr, Niklas and Du, Kevin and Sn bjarnarson, V \'e steinn and West, Robert and Cotterell, Ryan and Schein, Aaron. Activation Scaling for Steering and Interpreting Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.479

    doi:10.18653/v1/2024.findings-emnlp.479 2024

  57. [57]

    A Glitch in the Matrix ? Locating and Detecting Language Model Grounding with Fakepedia

    Monea, Giovanni and Peyrard, Maxime and Josifoski, Martin and Chaudhary, Vishrav and Eisner, Jason and Kiciman, Emre and Palangi, Hamid and Patra, Barun and West, Robert. A Glitch in the Matrix ? Locating and Detecting Language Model Grounding with Fakepedia. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    doi:10.18653/v1/2024.acl-long.369 2024

  58. [58]

    International Conference on Machine Learning (

    Unveiling the Visual Counting Bottleneck in Vision-Language Models , author=. International Conference on Machine Learning (. 2026 , url=

    2026

  59. [59]

    2009 , pages=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li , booktitle=. 2009 , pages=. doi:10.1109/CVPR.2009.5206848 , url=

    arXiv 2009

  60. [60]

    Interpretability in the Wild:

    Kevin Ro Wang and Alexandre Variengien and Arthur Conmy and Buck Shlegeris and Jacob Steinhardt , booktitle=. Interpretability in the Wild:. 2023 , url=

    2023

  61. [61]

    Probing Visual Language Priors in

    Luo, Tiange and Cao, Ang and Lee, Gunhee and Johnson, Justin and Lee, Honglak , booktitle =. Probing Visual Language Priors in. 2025 , editor =

    2025

  62. [62]

    arXiv preprint arXiv:2308.10248 , year=

    Steering Language Models With Activation Engineering , author=. arXiv preprint arXiv:2308.10248 , year=

    Pith/arXiv arXiv

  63. [63]

    and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. ...

    2023

  64. [64]

    Proceedings of the Third Conference on Causal Learning and Reasoning , pages =

    Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations , author =. Proceedings of the Third Conference on Causal Learning and Reasoning , pages =. 2024 , editor =

    2024

  65. [65]

    Entity-Based Knowledge Conflicts in Question Answering

    Longpre, Shayne and Perisetla, Kartik and Chen, Anthony and Ramesh, Nikhil and DuBois, Chris and Singh, Sameer. Entity-Based Knowledge Conflicts in Question Answering. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.565

    doi:10.18653/v1/2021.emnlp-main.565 2021

  66. [66]

    Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models , year=

    Building and better understanding vision-language models: insights and future directions , author=. Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models , year=

  67. [67]

    Understanding Retrieval Robustness for Retrieval-augmented Image Captioning

    Li, Wenyan and Li, Jiaang and Ramos, Rita and Tang, Raphael and Elliott, Desmond. Understanding Retrieval Robustness for Retrieval-augmented Image Captioning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.503

    work page doi:10.18653/v1/2024.acl-long.503 2024

  68. [68]

    Position:

    Huh, Minyoung and Cheung, Brian and Wang, Tongzhou and Isola, Phillip , booktitle =. Position:. 2024 , editor =

    2024

  69. [69]

    Do Vision and Language Models Share Concepts?

    Li, Jiaang and Kementchedjhieva, Yova and Fierro, Constanza and S. Do Vision and Language Models Share Concepts?. Transactions of the Association for Computational Linguistics , volume =. 2024 , doi =

    2024

  70. [70]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

    Tong, Shengbang and Fan, David and Li, Jiachen and Xiong, Yunyang and Chen, Xinlei and Sinha, Koustuv and Rabbat, Michael and LeCun, Yann and Xie, Saining and Liu, Zhuang , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2025 , pages =

    2025

  71. [71]

    F oodie QA : A Multimodal Dataset for Fine-Grained Understanding of C hinese Food Culture

    Li, Wenyan and Zhang, Crystina and Li, Jiaang and Peng, Qiwei and Tang, Raphael and Zhou, Li and Zhang, Weijia and Hu, Guimin and Yuan, Yifei and S gaard, Anders and Hershcovich, Daniel and Elliott, Desmond. F oodie QA : A Multimodal Dataset for Fine-Grained Understanding of C hinese Food Culture. Proceedings of the 2024 Conference on Empirical Methods in...

    doi:10.18653/v1/2024.emnlp-main.1063 2024

  72. [72]

    What if Othello -Playing Language Models Could See?

    Chen, Xinyi and Yuan, Yifei and Li, Jiaang and Belongie, Serge and de Rijke, Maarten and S gaard, Anders. What if Othello -Playing Language Models Could See?. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.673

    work page doi:10.18653/v1/2025.findings-emnlp.673 2025

  73. [73]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Onestory: Coherent multi-shot video generation with adaptive memory , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=