pith. machine review for the scientific record. sign in

arxiv: 2604.10949 · v1 · submitted 2026-04-13 · 💻 cs.CV · cs.AI

Recognition: unknown

Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models

Anyi Rao, Songlin Yang, Xianghao Kong

Authors on Pith no claims yet

Pith reviewed 2026-05-10 16:22 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords unified multimodal modelspseudo-unificationentropy probinginformation-theoretic analysismodality asymmetrytext-to-image generationinformation flowmultimodal synergy
0
0 comments X

The pith

Unified multimodal models exhibit pseudo-unification from divergent entropy trajectories in vision and language.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to explain why unified multimodal models rarely achieve full synergy between language reasoning and image generation. It introduces an information-theoretic probing framework to track how inputs are encoded and outputs are generated inside these models. Analysis across ten models identifies two core issues: vision and language take different entropy paths during encoding, and generation splits into high-entropy creative text versus low-entropy precise images. A sympathetic reader would care because only models that align both patterns deliver stronger reasoning-driven image synthesis, even at smaller scale. The work shows that shared parameters are not enough; consistent information flow across modalities is required for genuine unification.

Core claim

Pseudo-unification stems from a dual divergence: Modality-Asymmetric Encoding, where vision and language follow different entropy trajectories, and Pattern-Split Response, where text generation exhibits high-entropy creativity while image synthesis enforces low-entropy fidelity. This is revealed by the proposed information-theoretic probing framework applied to ten representative unified multimodal models. Only models that unify both sides, such as through contextual prediction, achieve more genuine unification and enable stronger reasoning-based text-to-image generation even with fewer parameters.

What carries the argument

The information-theoretic probing framework that jointly tracks entropy trajectories during input encoding and output generation while respecting prompt-response dependencies.

If this is right

  • Only models that align entropy patterns on both encoding and response sides achieve genuine multimodal synergy.
  • Consistency in information flow enables stronger reasoning-based text-to-image generation even when parameter counts are reduced.
  • Shared parameters alone cannot produce real unification without matching entropy behaviors across modalities.
  • Real multimodal synergy requires internal consistency in how information is handled, not merely architectural sharing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future model training objectives could explicitly penalize entropy divergence between modalities to promote unification.
  • The same probing approach could be extended to test unification in other combined domains such as video or audio-language models.
  • Evaluation of multimodal systems may need to incorporate internal entropy consistency checks alongside task accuracy.

Load-bearing premise

The entropy measurements and trajectory interpretations in the probing framework accurately expose the internal causes of unification failure without introducing measurement artifacts or biases.

What would settle it

A model engineered to enforce matching entropy trajectories across vision and language that nevertheless fails to transfer reasoning to image generation, or a model with mismatched trajectories that still succeeds at unified performance, would disprove the central explanation.

Figures

Figures reproduced from arXiv: 2604.10949 by Anyi Rao, Songlin Yang, Xianghao Kong.

Figure 1
Figure 1. Figure 1: An Illustration of Pseudo-Unification. We conduct an “unfair” comparison between BAGEL (14B) [13] and the much smaller Harmon (1.5B) [66] on a reasoning task about the Ameri￾can flag. Two key observations emerge: (i) Response Divergence: text correctly retrieves “American flag,” but image generation fails to produce it; (ii) Superior Cross-Modal Reasoning in a Small Model: despite lower fidelity and shorte… view at source ↗
Figure 2
Figure 2. Figure 2: Architectural Taxonomy of UMMs. Current UMMs fall into two categories: (i) Native UMMs, which unify text and image generation within a single architecture (e.g., Harmon [66], Janus-Pro [10], and Show-o2 [70]), which employ an all-in-one Transformer to jointly produce text and image tokens, while BAGEL [13] uses a Mixture-of-Transformers (MoT) [34] to separately generate text tokens and image tokens, fused … view at source ↗
Figure 3
Figure 3. Figure 3: Information-Theoretic Probing of UMMs. Left: Extract prompt, response, and hidden-state embedding sequences from a Transformer-based UMM and compute entropy (measuring representational quality for encoding patterns) and conditional entropy (mea￾suring output uncertainty given the input for response patterns). Right-Top: Matrix-based entropy increases with the number of independent information clusters. Rig… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of Text Prompt Length on Embedding Entropy (1st) and Layer Entropy (2nd-4th). 1st Sub-Fig: Entropy of text prompts increases with length, but absolute levels vary by architecture. 2nd-4th Sub-Figs: UMMs exhibit scale- and architecture-dependent early-layer compression strategies (e.g., entropy collapse), and middle-length prompts uniquely show larger entropy oscillations [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 5
Figure 5. Figure 5: Effect of Text Prompt Type on Layer Entropy. The same model exhibits nearly identical layer-wise entropy dynamics across different text types. 5.1.3. Effect of Prompt Type on Layer Entropy As shown in Tab. 1, embedding entropy levels across text types are similar. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of Image Prompt Type on Layer Entropy. The same model exhibits nearly identical layer-wise entropy trajecto￾ries across image types. 5.2. Image Prompt We probe image prompt representations across perception￾based and reasoning-based tasks, spanning low to high se￾mantic density and structural complexity. Despite this di￾versity, all models exhibit nearly identical layer-wise en￾tropy trajectories ac… view at source ↗
Figure 7
Figure 7. Figure 7: Response Patterns of Different UMMs. Except for Harmon, all UMMs exhibit a divergent response pattern, where layer￾wise text conditional entropy consistently exceeds that of images. In contrast, Harmon shows a unique cross-modal convergence, with conditional entropy aligning to a similar level in the final layers. For Omnigen2, the image response directly uses the prompt-encoding layer, making prompt and r… view at source ↗
read the original abstract

Unified multimodal models (UMMs) were designed to combine the reasoning ability of large language models (LLMs) with the generation capability of vision models. In practice, however, this synergy remains elusive: UMMs fail to transfer LLM-like reasoning to image synthesis and exhibit divergent response behaviors. We term this phenomenon pseudo-unification. Diagnosing its internal causes is important, but existing probing methods either lack model-internal insight or ignore prompt-response dependencies. To address these limitations, we propose an information-theoretic probing framework that jointly analyzes how UMMs encode inputs and generate outputs. Applied to ten representative UMMs, our framework reveals that pseudo-unification stems from a dual divergence: (i) Modality-Asymmetric Encoding, where vision and language follow different entropy trajectories, and (ii) Pattern-Split Response, where text generation exhibits high-entropy creativity while image synthesis enforces low-entropy fidelity. Only models that unify both sides (e.g., via contextual prediction) achieve more genuine unification, enabling stronger reasoning-based text-to-image generation even with fewer parameters. Our work provides the first model-internal probing of unification, demonstrating that real multimodal synergy requires consistency in information flow, not just shared parameters.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces the concept of 'pseudo-unification' in unified multimodal models (UMMs), claiming that these models fail to achieve true synergy between LLM-style reasoning and vision generation. It proposes a new information-theoretic probing framework that jointly examines input encoding and output generation. When applied to ten representative UMMs, the framework diagnoses pseudo-unification as arising from a dual divergence: (i) Modality-Asymmetric Encoding, in which vision and language inputs exhibit distinct entropy trajectories, and (ii) Pattern-Split Response, in which text generation displays high-entropy creative behavior while image synthesis enforces low-entropy fidelity. The authors conclude that only models achieving consistency across both aspects (e.g., via contextual prediction) attain more genuine unification, enabling stronger reasoning-based text-to-image generation even with fewer parameters.

Significance. If the probing framework can be shown to be free of modality-specific measurement artifacts, the work would constitute the first systematic model-internal diagnosis of unification failures in multimodal architectures. It supplies an empirical basis for the claim that genuine synergy requires aligned information flow rather than shared parameters alone, and the application across ten models offers comparative data that could guide future design choices. The emphasis on entropy trajectories as a diagnostic tool is a potentially useful addition to the interpretability literature, provided the estimation procedures are made reproducible.

major comments (3)
  1. [Methods] Methods section: the entropy probing framework is described at a high level but provides no concrete specification of the estimator used for continuous high-dimensional vision representations versus discrete token sequences. Because histogram binning, kernel density estimation, and Monte-Carlo sampling each carry modality-dependent bias and variance, the reported Modality-Asymmetric Encoding could be an artifact of the chosen approximation rather than evidence of internal unification failure. This detail is load-bearing for the central causal claim.
  2. [Results and §4] Results and §4: the assertion that 'only models that unify both sides achieve more genuine unification' lacks controls for confounding variables such as model scale, training objective, or data composition. Without ablation or statistical tests isolating the contribution of entropy-pattern consistency, the link between the observed dual divergence and improved text-to-image reasoning remains correlational.
  3. [Abstract and §3] Abstract and §3: the claim that the framework was applied to ten models and revealed the divergences is stated without accompanying implementation details, hyper-parameter choices, or validation of entropy-trajectory stability. This absence prevents independent verification that the dual divergence is not produced by the measurement procedure itself.
minor comments (2)
  1. [Introduction] The term 'pseudo-unification' is introduced in the abstract and introduction without a concise formal definition or contrast to 'genuine unification'; a short definitional paragraph would improve precision.
  2. [Figures] Figure captions and axis labels for entropy-trajectory plots should explicitly state the estimator, bin width or kernel bandwidth, and whether trajectories are averaged across prompts or computed per sample.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed review. The comments have prompted us to strengthen the methodological transparency, add controls and statistical support, and improve reproducibility. We respond to each major comment below and indicate the revisions made.

read point-by-point responses

  1. Referee: [Methods] Methods section: the entropy probing framework is described at a high level but provides no concrete specification of the estimator used for continuous high-dimensional vision representations versus discrete token sequences. Because histogram binning, kernel density estimation, and Monte-Carlo sampling each carry modality-dependent bias and variance, the reported Modality-Asymmetric Encoding could be an artifact of the chosen approximation rather than evidence of internal unification failure. This detail is load-bearing for the central causal claim.

    Authors: We agree that the original high-level description left the estimator choice underspecified and that modality-specific biases must be ruled out. In the revised manuscript we have inserted a new Methods subsection 'Entropy Estimation Procedures' that explicitly defines the estimators: for discrete token sequences we use the empirical Shannon entropy H = −∑ p_i log p_i with frequencies obtained from the model's softmax output (or input embedding counts); for continuous high-dimensional vision latents we apply the Kozachenko–Leonenko k-NN differential entropy estimator with k=10 and bias correction, which is known to be consistent in high dimensions and avoids binning or kernel artifacts. We include pseudocode, the exact hyper-parameter settings, and a short validation experiment on synthetic Gaussian and uniform data demonstrating that the estimator recovers ground-truth entropy within 3 % relative error. These additions directly address the concern that the observed Modality-Asymmetric Encoding could be an estimation artifact. revision: yes

  2. Referee: [Results and §4] Results and §4: the assertion that 'only models that unify both sides achieve more genuine unification' lacks controls for confounding variables such as model scale, training objective, or data composition. Without ablation or statistical tests isolating the contribution of entropy-pattern consistency, the link between the observed dual divergence and improved text-to-image reasoning remains correlational.

    Authors: We acknowledge that the original claim was stated too strongly and that confounding factors were not explicitly controlled. In the revision we have (i) added a supplementary table that stratifies the ten models by parameter count and primary training objective, (ii) computed partial correlations between entropy-consistency score and text-to-image reasoning metrics while controlling for scale and objective (resulting in a significant partial r = 0.61, p < 0.05), and (iii) replaced the word 'only' with 'models that achieve consistency across both aspects tend to' throughout §4 and the abstract. Full causal ablations (e.g., controlled retraining) remain outside the scope of the present study, but the cross-model evidence with statistical controls now provides stronger support for the reported relationship. revision: partial

  3. Referee: [Abstract and §3] Abstract and §3: the claim that the framework was applied to ten models and revealed the divergences is stated without accompanying implementation details, hyper-parameter choices, or validation of entropy-trajectory stability. This absence prevents independent verification that the dual divergence is not produced by the measurement procedure itself.

    Authors: We agree that reproducibility details were insufficient. We have expanded §3 with a new paragraph listing the exact ten models and their public checkpoints, the shared hyper-parameters (temperature = 1.0, maximum sequence length 512 for text, 256×256 resolution for images), and the stability protocol: 100 bootstrap resamples of each trajectory yielding standard deviations below 4 % of the mean entropy value. All configuration files and a minimal reproduction script are now provided in the supplementary material and will be released publicly upon acceptance. These changes allow independent verification that the dual divergence is not an artifact of the measurement procedure. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical observations via new probing method

full rationale

The paper introduces an information-theoretic probing framework and applies it empirically to ten UMMs to observe modality-asymmetric encoding and pattern-split responses. No equations, derivations, or fitted parameters are presented that reduce the reported dual divergence to self-definitional constructs, renamed known results, or self-citation chains. The central claims rest on direct application of the framework to model internals rather than any load-bearing step that equates outputs to inputs by construction. This is a standard empirical analysis with independent content.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim depends on the assumption that entropy measurements validly capture information encoding and generation dynamics, plus the interpretation that observed divergences cause rather than correlate with pseudo-unification.

axioms (1)
  • domain assumption Entropy trajectories in model internals accurately reflect differences in how modalities are encoded and how responses are generated.
    Invoked to link the probing results to the cause of pseudo-unification.
invented entities (1)
  • pseudo-unification no independent evidence
    purpose: Term to label the observed failure of true multimodal synergy.
    New descriptive label for the phenomenon diagnosed by the framework.

pith-pipeline@v0.9.0 · 5515 in / 1327 out tokens · 39219 ms · 2026-05-10T16:22:53.892935+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 34 canonical work pages · 12 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards.α-ReQ: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay

  3. [3]

    Understanding inter- mediate layers using linear classifier probes.ICLR, 2017

    Guillaume Alain and Yoshua Bengio. Understanding inter- mediate layers using linear classifier probes.ICLR, 2017. 3

    2017

  4. [4]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Why do LLMs attend to the first token?

    Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veli ˇckovi´c, and Razvan Pascanu. Why do LLMs attend to the first token?

  6. [6]

    Guillotine regulariza- tion: Why removing layers is needed to improve generaliza- tion in self-supervised learning

    Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regulariza- tion: Why removing layers is needed to improve generaliza- tion in self-supervised learning. 2023. 3

    2023

  7. [7]

    On identi- fiability in transformers.ICLR, 2020

    Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identi- fiability in transformers.ICLR, 2020. 3

    2020

  8. [8]

    Isotropy in the contextual embedding space: Clusters and manifolds

    Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. InInternational conference on learning repre- sentations, 2021. 2

    2021

  9. [9]

    BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Sil- vio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025. 2

    work page Pith review arXiv 2025

  10. [10]

    Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus- pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811,

    work page internal anchor Pith review arXiv

  11. [11]

    Emer- gence of a high-dimensional abstraction phase in language transformers.ICLR, 2025

    Emily Cheng, Diego Doimo, Corentin Kervadec, Iuri Ma- cocco, Jade Yu, Alessandro Laio, and Marco Baroni. Emer- gence of a high-dimensional abstraction phase in language transformers.ICLR, 2025. 3

    2025

  12. [12]

    Language modeling is compression.ICLR,

    Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau- Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression.ICLR,

  13. [13]

    Emerging Properties in Unified Multimodal Pretraining

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025. 1, 2, 3, 5, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning,

  15. [15]

    Unified autoregressive visual generation and understanding with continuous tokens.arXiv preprint arXiv:2503.13436, 2025

    Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens.arXiv preprint arXiv:2503.13436, 2025. 2

    work page arXiv 2025

  16. [16]

    Not all layers of LLMs are necessary during infer- ence

    Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of LLMs are necessary during infer- ence. 2024. 3

    2024

  17. [17]

    RankMe: Assessing the downstream perfor- mance of pretrained self-supervised representations by their rank

    Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. RankMe: Assessing the downstream perfor- mance of pretrained self-supervised representations by their rank. 2023. 3

    2023

  18. [18]

    When attention sink emerges in language models: An empirical view.ICLR,

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.ICLR,

  19. [19]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. 2023. 3

    2023

  20. [20]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 5

    2022

  21. [21]

    Large language models implicitly learn to straighten neural sentence trajec- tories to construct a predictive representation of natural lan- guage

    Eghbal Hosseini and Evelina Fedorenko. Large language models implicitly learn to straighten neural sentence trajec- tories to construct a predictive representation of natural lan- guage. 2023. 3

    2023

  22. [22]

    Corvid: Improving multimodal large language models towards chain-of-thought reasoning

    Jingjing Jiang, Chao Ma, Xurui Song, Hanwang Zhang, and Jun Luo. Corvid: Improving multimodal large language models towards chain-of-thought reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 3034–3046, 2025. 3

    2025

  23. [23]

    Co-reinforcement learning for unified mul- timodal understanding and generation.arXiv preprint arXiv:2505.17534, 2025

    Jingjing Jiang, Chongjie Si, Jun Luo, Hanwang Zhang, and Chao Ma. Co-reinforcement learning for unified mul- timodal understanding and generation.arXiv preprint arXiv:2505.17534, 2025. 2

    work page arXiv 2025

  24. [24]

    Exploring concept depth: How large language models acquire knowledge at different layers?

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge at different layers?

  25. [25]

    Rare text semantics were always there in your diffusion transformer.arXiv preprint arXiv:2510.03886, 2025

    Seil Kang, Woojung Han, Dayun Ju, and Seong Jae Hwang. Rare text semantics were always there in your diffusion transformer.arXiv preprint arXiv:2510.03886, 2025. 8

    work page arXiv 2025

  26. [26]

    Flux.https://github.com/ black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/ black-forest-labs/flux, 2024. 1

    2024

  27. [28]

    Easier painting than thinking: Can text-to-image models set the stage, but not direct the play?arXiv preprint arXiv:2509.03516, 2025

    Ouxiang Li, Yuan Wang, Xinting Hu, Huijuan Huang, Rui Chen, Jiarong Ou, Xin Tao, Pengfei Wan, Xiaojuan Qi, and Fuli Feng. Easier painting than thinking: Can text-to-image models set the stage, but not direct the play?arXiv preprint arXiv:2509.03516, 2025. 5

    work page arXiv 2025

  28. [29]

    InThe Fourteenth Inter- national Conference on Learning Representations

    Yang Li, Songlin Yang, Wei Wang, Xiaoxuan Han, and Jing Dong.alpha-dpo: Robust preference alignment for diffu- sion models viaalpha-divergence. InThe Fourteenth Inter- national Conference on Learning Representations. 1

  29. [30]

    Large language model evaluation via matrix nuclear-norm.arXiv preprint arXiv:2410.10672, 2024

    Yahan Li, Tingyu Xia, Yi Chang, and Yuan Wu. Large language model evaluation via matrix nuclear-norm.arXiv preprint arXiv:2410.10672, 2024. 4

    work page arXiv 2024

  30. [31]

    Unieval: Uni- fied holistic evaluation for unified multimodal understanding and generation.arXiv preprint arXiv:2505.10483, 2025

    Yi Li, Haonan Wang, Qixiang Zhang, Boyu Xiao, Chen- chang Hu, Hualiang Wang, and Xiaomeng Li. Unieval: Uni- fied holistic evaluation for unified multimodal understanding and generation.arXiv preprint arXiv:2505.10483, 2025. 2

    work page arXiv 2025

  31. [32]

    Instant preference alignment for text-to-image diffusion models.arXiv preprint arXiv:2508.17718, 2025

    Yang Li, Songlin Yang, Xiaoxuan Han, Wei Wang, Jing Dong, Yueming Lyu, and Ziyu Xue. Instant preference alignment for text-to-image diffusion models.arXiv preprint arXiv:2508.17718, 2025. 1

    work page arXiv 2025

  32. [33]

    Beyond inserting: Learning subject embedding for semantic-fidelity personalized diffusion generation.IEEE Transactions on Circuits and Systems for Video Technology, 2025

    Yang Li, Songlin Yang, Wei Wang, and Jing Dong. Beyond inserting: Learning subject embedding for semantic-fidelity personalized diffusion generation.IEEE Transactions on Circuits and Systems for Video Technology, 2025. 1

    2025

  33. [34]

    arXiv preprint arXiv:2411.04996 , year =

    Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996, 2024. 3

    work page arXiv 2024

  34. [35]

    UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic en- coders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025. 2

    work page internal anchor Pith review arXiv 2025

  35. [36]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [37]

    Visual representations inside the language model.arXiv preprint arXiv:2510.04819,

    Benlin Liu, Amita Kamath, Madeleine Grunde-McLaughlin, Winson Han, and Ranjay Krishna. Visual representations in- side the language model.arXiv preprint arXiv:2510.04819,

    work page arXiv

  37. [38]

    Linguistic knowledge and trans- ferability of contextual representations

    Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and trans- ferability of contextual representations. 2019. 3

    2019

  38. [39]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 5, 6

    2024

  39. [40]

    Mmmg: A massive, multidisciplinary, multi-tier gener- ation benchmark for text-to-image reasoning.arXiv preprint arXiv:2506.10963, 2025

    Yuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, and Zhouhui Lian. Mmmg: A massive, multidisciplinary, multi-tier gener- ation benchmark for text-to-image reasoning.arXiv preprint arXiv:2506.10963, 2025. 2

    work page arXiv 2025

  40. [41]

    Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal under- standing and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autore- gression and rectified flow for unified multimodal under- standing and generation. InProceedings of the Computer Vi- sion and Pattern Recognition Conference, pages 7739–7751,

  41. [42]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265,

    Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, et al. Wise: A world knowledge-informed seman- tic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025. 2

    work page arXiv 2025

  42. [43]

    Repre- sentation learning with contrastive predictive coding.ICLR,

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.ICLR,

  43. [44]

    The geometry of categorical and hierarchical concepts in large language models.ICML 2024 Workshop on Mecha- nistic Interpretability, 2024

    Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models.ICML 2024 Workshop on Mecha- nistic Interpretability, 2024. 3

    2024

  44. [45]

    SVCCA: Singular vector canonical correla- tion analysis for deep learning dynamics and interpretability

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular vector canonical correla- tion analysis for deep learning dynamics and interpretability

  45. [46]

    The shape of learning: Anisotropy and intrin- sic dimensions in transformer-based models

    Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Gon- charova, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. The shape of learning: Anisotropy and intrin- sic dimensions in transformer-based models. 2024. 3

    2024

  46. [47]

    On measures of entropy and information.Pro- ceedings of the fourth Berkeley symposium on mathematical statistics and probability, 1961

    Alfr ´ed R´enyi. On measures of entropy and information.Pro- ceedings of the fourth Berkeley symposium on mathematical statistics and probability, 1961. 4

    1961

  47. [48]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  48. [49]

    The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in transformer training

    Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, and Benjamin Grewe. The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in transformer training. 2025. 3

    2025

  49. [50]

    Realunify: Do unified models truly benefit from unification? a comprehensive benchmark.arXiv preprint arXiv:2509.24897, 2025

    Yang Shi, Yuhao Dong, Yue Ding, Yuran Wang, Xuanyu Zhu, Sheng Zhou, Wenting Liu, Haochen Tian, Rundong Wang, Huanqian Wang, et al. Realunify: Do unified models truly benefit from unification? a comprehensive benchmark. arXiv preprint arXiv:2509.24897, 2025. 2, 3

    work page arXiv 2025

  50. [51]

    PhD thesis, Hebrew University, 2022

    Ravid Shwartz-Ziv.Information flow in deep neural net- works. PhD thesis, Hebrew University, 2022. 3

    2022

  51. [52]

    Opening the black box of deep neural networks via information

    Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. 2019. 3

    2019

  52. [53]

    DiME: Maxi- mizing mutual information by a difference of matrix-based entropies

    Oscar Skean, Jhoan Keider Hoyos Osorio, Austin J Brock- meier, and Luis Gonzalo Sanchez Giraldo. DiME: Maxi- mizing mutual information by a difference of matrix-based entropies. 2023. 2, 4

    2023

  53. [54]

    Endogenous re- prompting: Self-evolving cognitive alignment for unified multimodal models.arXiv preprint arXiv:2601.20305, 2026

    Zhenchen Tang, Songlin Yang, Zichuan Wang, Bo Peng, Yang Li, Beibei Dong, and Jing Dong. Endogenous re- prompting: Self-evolving cognitive alignment for unified multimodal models.arXiv preprint arXiv:2601.20305, 2026. 1

    work page arXiv 2026

  54. [55]

    Chameleon: Mixed-Modal Early-Fusion Foundation Models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 2

    work page internal anchor Pith review arXiv 2024

  55. [56]

    BERT redis- covers the classical nlp pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT redis- covers the classical nlp pipeline. 2019. 3

    2019

  56. [57]

    LiDAR: Sensing linear probing performance in joint embedding ssl architectures.ICLR, 2024

    Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. LiDAR: Sensing linear probing performance in joint embedding ssl architectures.ICLR, 2024. 3

    2024

  57. [58]

    The geometry of hidden representations of large transformer models

    Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. 2023. 3

    2023

  58. [59]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. 30, 2017. 5

    2017

  59. [60]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. 2

    2017

  60. [61]

    The bottom- up evolution of representations in the transformer: A study with machine translation and language modeling objectives

    Elena V oita, Rico Sennrich, and Ivan Titov. The bottom- up evolution of representations in the transformer: A study with machine translation and language modeling objectives

  61. [62]

    Ovis-u1 technical report

    Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiao- hao Chen, Jianshan Zhao, et al. Ovis-u1 technical report. arXiv preprint arXiv:2506.23044, 2025. 2

    work page arXiv 2025

  62. [63]

    Emu3: Next-Token Prediction is All You Need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024. 1

    work page internal anchor Pith review arXiv 2024

  63. [64]

    OmniGen2: Towards Instruction-Aligned Multimodal Generation

    Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, et al. Omnigen2: Exploration to advanced multimodal generation.arXiv preprint arXiv:2506.18871, 2025. 3, 5, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  64. [65]

    Liquid: Language models are scalable multi-modal generators.arXiv preprint arXiv:2412.04332, 2024

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Heng- shuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liq- uid: Language models are scalable and unified multi-modal generators.arXiv preprint arXiv:2412.04332, 2024. 2

    work page arXiv 2024

  65. [66]

    Harmonizing visual representations for unified multimodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified mul- timodal understanding and generation.arXiv preprint arXiv:2503.21979, 2025. 1, 3, 5, 6

    work page arXiv 2025

  66. [67]

    Efficient streaming language models with attention sinks.ICLR, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.ICLR, 2024. 3

    2024

  67. [68]

    Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024. 5, 6

    work page internal anchor Pith review arXiv 2024

  68. [69]

    Reconstruction alignment improves unified multimodal models.arXiv preprint arXiv:2509.07295, 2025a

    Ji Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multi- modal models.arXiv preprint arXiv:2509.07295, 2025. 5, 6

    work page arXiv 2025

  69. [70]

    Show-o2: Improved Native Unified Multimodal Models

    Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show- o2: Improved native unified multimodal models.arXiv preprint arXiv:2506.15564, 2025. 3, 5, 6

    work page internal anchor Pith review arXiv 2025

  70. [71]

    Mme-unify: A comprehensive benchmark for unified multimodal understanding and generation models.arXiv preprint arXiv:2504.03641, 2025

    Wulin Xie, Yi-Fan Zhang, Chaoyou Fu, Yang Shi, Bingyan Nie, Hongkai Chen, Zhang Zhang, Liang Wang, and Tie- niu Tan. Mme-unify: A comprehensive benchmark for uni- fied multimodal understanding and generation models.arXiv preprint arXiv:2504.03641, 2025. 2

    work page arXiv 2025

  71. [72]

    Can understanding and generation truly benefit together–or just coexist?arXiv e-prints, pages arXiv–2509, 2025

    Zhiyuan Yan, Kaiqing Lin, Zongjian Li, Junyan Ye, Hui Han, Zhendong Wang, Hao Liu, Bin Lin, Hao Li, Xue Xu, et al. Can understanding and generation truly benefit together–or just coexist?arXiv e-prints, pages arXiv–2509, 2025. 2

    2025

  72. [73]

    Diffusion models: A comprehensive survey of methods and applications.ACM computing surveys, 56(4): 1–39, 2023

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Run- sheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications.ACM computing surveys, 56(4): 1–39, 2023. 5

    2023

  73. [74]

    Human-centric content generation with diffu- sion models: A survey.Authorea Preprints, 2026

    Songlin Yang, Yueming Lyu, Ziyuan Chen, Yang Li, Beibei Dong, Xiaoxuan Han, Pei Yang, Ziye Wang, Anyi Rao, Zi- wei Liu, et al. Human-centric content generation with diffu- sion models: A survey.Authorea Preprints, 2026. 1

    2026

  74. [75]

    Shotverse: Advancing cinematic cam- era control for text-driven multi-shot video creation.arXiv preprint arXiv:2603.11421, 2026

    Songlin Yang, Zhe Wang, Xuyi Yang, Songchun Zhang, Xi- anghao Kong, Taiyi Wu, Xiaotong Zhao, Ran Zhang, Alan Zhao, and Anyi Rao. Shotverse: Advancing cinematic cam- era control for text-driven multi-shot video creation.arXiv preprint arXiv:2603.11421, 2026. 1

    work page arXiv 2026

  75. [76]

    Memory retrieval and consolidation in large language models through function tokens.arXiv preprint arXiv:2510.08203, 2025

    Shaohua Zhang, Yuan Lin, and Hang Li. Memory retrieval and consolidation in large language models through function tokens.arXiv preprint arXiv:2510.08203, 2025. 8

    work page arXiv 2025

  76. [77]

    Doracy- cle: Domain-oriented adaptation of unified generative model in multimodal cycles

    Rui Zhao, Weijia Mao, and Mike Zheng Shou. Doracy- cle: Domain-oriented adaptation of unified generative model in multimodal cycles. InProceedings of the Computer Vi- sion and Pattern Recognition Conference, pages 2835–2846,

  77. [78]

    Layer by layer: Uncovering where multi-task learning happens in instruction-tuned large language models

    Zheng Zhao, Yftah Ziser, and Shay B Cohen. Layer by layer: Uncovering where multi-task learning happens in instruction-tuned large language models. 2024. 2, 4

    2024

  78. [79]

    Pairuni: Pairwise training for unified multimodal lan- guage models.arXiv preprint arXiv:2510.25682, 2025

    Jiani Zheng, Zhiyang Teng, Xiangtai Li, Anran Wang, Yu Tian, Kunpeng Qiu, Ye Tian, Haochen Wang, and Zhuochen Wang. Pairuni: Pairwise training for unified multimodal lan- guage models.arXiv preprint arXiv:2510.25682, 2025. 2

    work page arXiv 2025

  79. [80]

    Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark

    Kai Zou, Ziqi Huang, Yuhao Dong, Shulin Tian, Dian Zheng, Hongbo Liu, Jingwen He, Bin Liu, Yu Qiao, and Ziwei Liu. Uni-mmmu: A massive multi-discipline multimodal unified benchmark.arXiv preprint arXiv:2510.13759, 2025. 2, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025