pith. sign in

arxiv: 2605.24624 · v1 · pith:WLOGG3E5new · submitted 2026-05-23 · 💻 cs.CV

Vision-Language Binding in In-Context Image Generation

Chris Ge , Rohit Gandikota , Antonio Torralba , Tamar Rott Shaham This is my paper

Pith reviewed 2026-06-30 13:58 UTC · model grok-4.3

classification 💻 cs.CV
keywords cross-modal bindingin-context image generationtext tokensreference image conditioningmultimodal DiTattention interventionsFLUX.2visual properties routing
0
0 comments X

The pith

Text tokens in unified-attention image models absorb reference image content like color and style and carry it to the output image.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that in models such as FLUX.2, where text, reference image, and noise tokens share a single attention stream, text tokens form an implicit binding with the reference image during processing. Visual properties are absorbed into the text tokens and then causally shape the generated image, while exact pixel matches travel directly through image-to-image paths. Three interventions—decoding text activations via a text-to-image route, severing specific attention connections, and copying activations between runs—map this flow across thousands of editing tasks. The binding concentrates in the padding tokens of the text sequence. This positions text tokens as an active routing channel rather than passive prompt containers.

Core claim

An implicit cross-modal binding emerges between the text tokens and the reference image: the text tokens absorb visual reference content during the forward pass, and that absorbed content causally influences the generated output. Properties like color, style, and scene setting are first written into the text tokens, which carry them to the generated image; pixel-exact properties bypass the text tokens and flow directly from reference to image through image-to-image attention. The binding localizes to padding tokens.

What carries the argument

The implicit cross-modal binding between text tokens and reference image, isolated by T2I Lens decoding of intermediate activations, Attention Knockout of specific edges, and I2I-to-I2I Patching of token activations across editing runs.

If this is right

  • Color, style, and scene setting from a reference image route through text tokens to the output.
  • Pixel-exact details such as specific faces or identities travel directly via image-to-image attention.
  • The reference-text binding occurs specifically at padding tokens within the text sequence.
  • Text tokens function as a structured channel that actively transports reference content rather than serving only as prompt holders.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The division of labor between token-mediated and direct paths may appear in other DiT-based multimodal generators that concatenate modalities in one attention stream.
  • Targeted editing of padding-token activations could allow selective transfer of stylistic properties without altering identity.
  • The observed routing suggests that modality-specific token roles shape information flow even when all inputs share unified attention.

Load-bearing premise

The three causal interventions isolate the model's natural information flow without introducing artifacts or rerouting that would not occur in normal operation.

What would settle it

If text-token activations decoded through a text-to-image path show no reconstruction of reference color or style, or if severing text-to-reference attention edges leaves those properties unchanged in the output, the binding claim fails.

Figures

Figures reproduced from arXiv: 2605.24624 by Antonio Torralba, Chris Ge, Rohit Gandikota, Tamar Rott Shaham.

Figure 1
Figure 1. Figure 1: Our three causal intervention methods on text tokens. (1) In T2I Lens, we patch the text residual stream after a given MM-DiT or DiT block into the raw text embeddings of a reference-free, unconditional text-to-image generation with an empty prompt, decoding the text-token activations at that point back into pixel space. (2) In Attention Knockout [5], we mask out the attention paid by text tokens to refere… view at source ↗
Figure 2
Figure 2. Figure 2: T2I Lens. I2I Baseline is generated using the edit instruction on the Reference image. We then apply T2I Lens to the intermediate-layer text tokens of this baseline to visually reveal their encoded information. Despite the edit instructions not revealing information about the reference image, the T2I Lens outputs generically match the reference setting (parking garage, sports field, fancy backyard, cockpit… view at source ↗
Figure 3
Figure 3. Figure 3: Attention Knockout. I2I Baseline is generated using the edit prompt on the Reference. In KOref→img, we knock out attention paid by the image tokens to the reference tokens and repeat the editing task, and similarly for KOref→txt. In the first four columns, KOref→img does not prevent the reference’s color or style from appearing in the generated output, while KOref→txt fully prevents it, keeping the pillow … view at source ↗
Figure 4
Figure 4. Figure 4: I2I-to-I2I Patching. The Source I2I Baseline is generated using the edit on the Source Reference, and similarly for the Target I2I Baseline. Text-token activations are patched from the Source to Target I2I generations in corresponding layers, yielding the Target I2I Patched output. Patching successfully transfers the style of the reference onto the image of the boy in the first column, while preserving oth… view at source ↗
Figure 5
Figure 5. Figure 5: T2I Lens on only padding or content text tokens. We repeat our T2I Lens technique on all tasks, but only patching over a subset of the text tokens. The padding tokens on their own consistently encode reference scene information. Unlike T2I Lens on all text tokens, T2I Lens on the padding tokens does not always include the edit: there is no camera or hot air balloon in the second and third examples. In cont… view at source ↗
read the original abstract

In-context image generation models such as FLUX.2 take a text prompt and an optional reference image as visual conditioning for the output. Internally, all three inputs -- text, reference image, and the noise tokens -- are concatenated and processed through a single attention stream, where all tokens can attend to one another. This leaves open how reference information flows through the model to produce the output image. We show that an implicit cross-modal binding emerges between the text tokens and the reference image: the text tokens absorb visual reference content during the forward pass, and that absorbed content causally influences the generated output. We surface this binding with three causal interventions on FLUX.2: T2I Lens, which decodes intermediate text-token activations through a text-to-image path; Attention Knockout, which severs specific attention edges; and I2I-to-I2I Patching, which copies text token activations between editing runs. Across 2,875 editing tasks on various images, including SUN397 and DreamBench++ datasets and images collected online, we observe a consistent division of labor: properties of the reference image, like color, style, and scene setting, are first written into the text tokens, which carry them to the generated image; pixel-exact properties like a specific face or instance identity bypass the text tokens and flow directly from reference to image through image-to-image attention. We further localize the reference-text binding to the padding tokens of the text sequence. These results show that text tokens in a multimodal DiT are not just prompt holders, but a structured channel for reference image content. More broadly, they suggest that even in unified-attention multimodal generative models, token modality structures how conditioning information is represented and routed across the network.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper claims that in in-context image generation models such as FLUX.2, an implicit cross-modal binding emerges between text tokens and the reference image: text tokens absorb visual reference content during the forward pass and causally influence the generated output. Properties like color, style, and scene setting are written into text tokens (localized to padding tokens), which carry them to the output, while pixel-exact properties bypass via direct I2I attention. This is surfaced via three causal interventions (T2I Lens decoding, Attention Knockout, I2I-to-I2I Patching) across 2,875 editing tasks on SUN397, DreamBench++, and other images, showing text tokens as a structured channel in multimodal DiTs.

Significance. If the causal claims hold, the work provides concrete empirical evidence on information routing in unified-attention multimodal generative models, showing that token modality structures conditioning flow rather than treating text tokens as passive prompt holders. The scale (2,875 tasks on named datasets) and intervention-based approach strengthen the observations if controls confirm no artifacts; this could inform model interpretability and design in DiT-based systems.

major comments (1)
  1. Abstract and methods description: the central claim that the three interventions (T2I Lens, Attention Knockout, I2I-to-I2I Patching) reveal natural information flow without artifacts depends on unshown quantitative controls, such as activation statistics pre/post-intervention or ablations on non-binding edges. Without these, it remains unclear whether observed effects (e.g., division of labor between text-routed and direct I2I properties) match unmodified forward passes or reflect intervention-induced routing changes.
minor comments (1)
  1. Clarify exact definitions of the 2,875 tasks and provide a table summarizing per-dataset breakdowns and success metrics for each intervention to aid reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the major comment below and agree that additional controls will strengthen the manuscript.

read point-by-point responses

  1. Referee: [—] Abstract and methods description: the central claim that the three interventions (T2I Lens, Attention Knockout, I2I-to-I2I Patching) reveal natural information flow without artifacts depends on unshown quantitative controls, such as activation statistics pre/post-intervention or ablations on non-binding edges. Without these, it remains unclear whether observed effects (e.g., division of labor between text-routed and direct I2I properties) match unmodified forward passes or reflect intervention-induced routing changes.

    Authors: We agree this is a valid concern: the manuscript does not currently include quantitative controls such as pre/post-intervention activation statistics or ablations on non-binding edges. In the revised version we will add these, including L2-norm and cosine-similarity comparisons of text-token and image-token activations before versus after each intervention, plus targeted ablations that sever non-binding attention edges while preserving the reported pathways. These results will be reported in an expanded Methods section and supplementary figures to confirm that the observed division of labor is not an artifact of the interventions. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical interventions on external model

full rationale

The paper is an empirical study using three causal interventions (T2I Lens, Attention Knockout, I2I-to-I2I Patching) on the external FLUX.2 model to observe information flow. No derivations, equations, fitted parameters, or predictions are present. Central claims rest on external model behavior and datasets (SUN397, DreamBench++) rather than internal definitions or self-citations. No load-bearing self-citation chains, ansatzes, or renamings of known results are identified. This is a standard non-circular empirical finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper performs post-hoc causal analysis on a pre-trained model using standard interpretability techniques; it introduces no new free parameters, mathematical axioms, or postulated entities.

pith-pipeline@v0.9.1-grok · 5854 in / 1302 out tokens · 38564 ms · 2026-06-30T13:58:26.204657+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    Claude Opus 4.7 system card

    Anthropic. Claude Opus 4.7 system card. Technical report, Anthropic, 2026. URL https://anthropic. com/claude-opus-4-7-system-card

    2026

  2. [2]

    Localizing and editing knowledge in text-to-image generative models

    Samyadeep Basu, Nanxuan Zhao, Vlad Morariu, Soheil Feizi, and Varun Manjunatha. Localizing and editing knowledge in text-to-image generative models. InInternational Conference on Learning Represen- tations, 2024

    2024

  3. [3]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023

    2023

  4. [4]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

  5. [5]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235, 2023

    2023

  6. [6]

    How to use and interpret activation patching

    Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching.arXiv preprint arXiv:2404.15255, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Con- ceptattention: Diffusion transformers learn highly interpretable features

    Alec Helbling, Tuna Han Salih Meral, Benjamin Hoover, Pinar Yanardag, and Duen Horng Chau. Con- ceptattention: Diffusion transformers learn highly interpretable features. InForty-second International Conference on Machine Learning, 2025

    2025

  8. [8]

    Prompt-to- prompt image editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross-attention control. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  9. [9]

    What’s in the image? a deep-dive into the vision of vision language models

    Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? a deep-dive into the vision of vision language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14549–14558, 2025

    2025

  10. [10]

    Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models

    Guy Kaplan, Michael Toker, Yuval Reif, Yonatan Belinkov, and Roy Schwartz. Follow the flow: On information flow across textual tokens in text-to-image models.arXiv preprint arXiv:2504.01137, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    FLUX.2: Frontier Visual Intelligence.https://bfl.ai/blog/flux-2, 2025

    Black Forest Labs. FLUX.2: Frontier Visual Intelligence.https://bfl.ai/blog/flux-2, 2025. 10

    2025

  12. [12]

    Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space.arXiv preprint arXiv:2506.15742, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    I2am: Interpreting image-to-image latent diffusion models via bi- attribution maps

    Junseo Park and Hyeryung Jang. I2am: Interpreting image-to-image latent diffusion models via bi- attribution maps. InInternational Conference on Learning Representations, 2025

    2025

  14. [14]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  15. [15]

    Dreambench++: A human-aligned benchmark for personalized image generation

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation. InInternational Conference on Learning Representations, 2025

    2025

  16. [16]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015

    2015

  17. [17]

    Diffusion lens: Interpreting text encoders in text-to-image pipelines

    Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, and Yonatan Belinkov. Diffusion lens: Interpreting text encoders in text-to-image pipelines. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9713–9728, 2024

    2024

  18. [18]

    Padding tone: A mechanistic analysis of padding tokens in t2i models

    Michael Toker, Ido Galil, Hadas Orgad, Rinon Gal, Yoad Tewel, Gal Chechik, and Yonatan Belinkov. Padding tone: A mechanistic analysis of padding tokens in t2i models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7618–...

    2025

  19. [19]

    Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22(158):209–212, 1927

    Edwin B Wilson. Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22(158):209–212, 1927

    1927

  20. [20]

    Qwen-Image Technical Report

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Sun database: Large- scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010

    2010

  22. [22]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13294–13304, 2025

    2025

  23. [23]

    Group relative attention guidance for image editing.arXiv preprint arXiv:2510.24657, 2025

    Xuanpu Zhang, Xuesong Niu, Ruidong Chen, Dan Song, Jianhao Zeng, Penghui Du, Haoxiang Cao, Kai Wu, and An-an Liu. Group relative attention guidance for image editing.arXiv preprint arXiv:2510.24657, 2025

    work page arXiv 2025

  24. [24]

    Enabling instructional image editing with in-context generation in large scale diffusion transformer.Advances in Neural Information Processing Systems, 2026

    Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang. Enabling instructional image editing with in-context generation in large scale diffusion transformer.Advances in Neural Information Processing Systems, 2026

    2026

  25. [25]

    Add a lamp post,

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. 11 Appendix A Specific choice of layers in FLUX.2 Klein 9B to patch For the experiments in ...

    2023

  26. [26]

    add_object

    ADD: a single object NOT currently in the image but plausibly fits the scene. If no scene-agnostic addition fits, set "add_object": null

  27. [27]

    remove_object

    REMOVE: a single object IS visible and could be plausibly removed (not the entire subject; the scene would still read coherently without it). If nothing meets that bar, set "remove_object": null. SCENE-AGNOSTIC RULE: the proposed object names must NOT reveal the specific scene/location depicted. For a volcano photo, "lava plume" or "volcanic crater" is fo...

    work page arXiv 2010