Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Concurrent LoRA training during MAE pretraining lets a frozen LLM block improve vision transformers.

desk verdict Solid empirical recipe with good controls; headline IN-1K claim needs baseline reconciliation, but the co-adaptation evidence survives the stress-test. read the letter →

arxiv 2507.00754 v2 pith:H67FM4VT submitted 2025-07-01 cs.CV

Selim Kuzucu , Muhammad Ferjad Naeem , Anna Kukleva , Federico Tombari , Bernt Schiele This is my paper
classification cs.CV
keywords maskedauto-encodinglow-rankadaptationlargelanguagemodelvisiontransformerself-supervisedlearningmodalityalignmentimageclassificationdomain-shiftrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a text-pretrained large language model block can be made useful for purely visual tasks if it is co-adapted with the vision encoder during self-supervised pretraining, rather than simply bolted on frozen. Its recipe is to run Masked Auto-Encoding (MAE) — reconstructing masked image patches from visible ones — while simultaneously training Low-Rank Adaptation (LoRA) layers inside a frozen LLM block on the same reconstruction loss. The paper argues that this joint optimization bridges the modality gap from both ends: the ViT learns features the LLM block can interpret, and the LLM block learns to refine visual features with its pretrained semantic knowledge. In experiments, the resulting LUViT/B reaches 83.6% top-1 accuracy on ImageNet-1K, 0.4 points above the MAE pretrained ViT/B baseline, with larger gains on robustness benchmarks such as ImageNet-A (+2.2). If correct, the method offers a parameter-efficient way to inject LLM knowledge into discriminative vision models without language inputs, full-LLM finetuning, or a second alignment stage.

What carries the argument

The central mechanism is a shared backpropagation path: one MAE reconstruction loss trains the ViT encoder and, through it, the LoRA low-rank adapters (rank 16 on the query and value projections) inside a frozen LLM transformer block, together with the linear projections that bridge the ViT and LLM hidden dimensions. LoRA keeps the LLM's original weights untouched, so the only new language-side parameters are small low-rank update matrices. The paper's key architectural adjustments are bidirectional attention inside the LLM block and removal of rotary positional embeddings, so the text-trained block treats image patches as a non-sequential set. The claim is that because the reconstruction loss flows through the LLM block, the ViT is pushed toward features the LLM block can enhance, and the LoRA adapters are pushed toward interpreting those visual features, aligning the two modalities during pretraining rather than at finetuning time.

What would settle it

Run the full LUViT pretraining but replace the LLM block's pretrained weights with a randomly initialized transformer of identical size and the same LoRA path at multiple seeds: the paper's own single random-init run trails by only 0.4 points, so a decisive test would check whether the pretrained-language advantage grows with scale or model size. A complementary check is to probe whether LLM hidden states after LUViT pretraining organize visual patches semantically in a way that the MAE-only ViT's features do not.

Watch

Extended reading notes

Core claim

The paper's central claim is that concurrent LoRA training during MAE pretraining is the ingredient that unlocks the LLM block's contribution. The authors state this as a synergistic co-adaptation: the reconstruction loss updates the ViT, the linear projections, and the LoRA adapters together, so that neither modality is frozen while the other adapts. Their ablations are designed to show that the gain is not capacity or architecture: a frozen LLM block without LoRA matches the MAE baseline, a parameter-matched MLP is 0.3 points behind, and a randomly initialized LLM block with the same LoRA setup is 0.4 points behind. The paper concludes that language-pretrained representations, steered by LoRA under the MAE objective, are what produce the improvements, and it ties the effect to more focused attention on foreground regions, which it associates with the large ImageNet-A and background-swap gains.

Load-bearing premise

The load-bearing premise is that the MAE pixel-reconstruction loss, backpropagated through a frozen text-pretrained LLM block via LoRA, is a sufficient alignment signal to make the LLM's language-pretrained knowledge useful for vision; the paper never directly measures this alignment and instead infers it from downstream accuracy.

Editorial extensions

If this is right

  • LUViT/B reports 83.6% top-1 accuracy on ImageNet-1K, a new state-of-the-art among comparable frozen-LLM-augmented ViT models and 0.4 points above the MAE-pretrained ViT/B baseline.
  • Domain-shift robustness improves most where spurious cues fail: +2.2 points on ImageNet-A, +0.8 on ImageNet-Sketch, +0.7 on ImageNet-V2, and +0.5 on ImageNet-C.
  • Fine-grained transfer also improves: +0.5 bounding-box AP and +0.2 mask AP on COCO object detection and instance segmentation under the paper's detection training protocol.
  • LoRA adaptation is necessary: without it the frozen LLM block is performance-neutral relative to MAE pretraining, and with a randomly initialized block the gains shrink by 0.4 points.
  • The method shifts attention: LUViT has higher attention entropy on background patches than foreground patches for about 83% of images, and it cuts the background dependence gap on ImageNet-9 mixed-random by 2.0 points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper infers modality alignment from downstream accuracy; a direct test would probe the LLM block's hidden states after LUViT pretraining for visual semantic structure, such as class-conditional linear probing in LLM token space, to confirm that language knowledge rather than gradient regularization is being transferred.
  • Because the improvement over an equal-parameter MLP is only 0.3 points on ImageNet-1K, the 'language' contribution might be small at this scale; scaling to larger ViTs, multiple LLM blocks, or other text-pretrained blocks would show whether the gap grows or stays flat.
  • The foreground-attention effect suggests a testable extension: use LUViT as a teacher to distill a vanilla ViT that keeps the robustness gains without the LLM block's inference cost, isolating whether the co-adapted features transfer.
  • The method trains on ImageNet-1K only; trying the same co-adaptation on larger unlabeled corpora would show whether the MAE-plus-LoRA recipe scales with data, which is where a semantic-knowledge transfer story would be most visible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes LUViT, an architecture and pretraining strategy that inserts a frozen LLaMA block, adapted by LoRA, between a ViT encoder and a lightweight MAE decoder, and trains the ViT and LoRA jointly with the MAE reconstruction objective. After pretraining on ImageNet-1K for 800 epochs, the decoder is discarded and the model is fine-tuned for classification and object detection. The paper reports 83.6% top-1 accuracy on ImageNet-1K, which it presents as a new state-of-the-art in this setting and as a +0.4% improvement over its reproduced MAE ViT-B baseline, together with larger gains on ImageNet-A and moderate gains on IN-Sketch, IN-V2, IN-C, and COCO detection/segmentation. Ablations compare against parameter-matched MLP baselines and randomly initialized LLM blocks, and a 10-seed experiment on ImageNet-1K reports non-overlapping 99% confidence intervals against the baselines.

Significance. If the reported gains hold under a properly calibrated comparison, LUViT is a useful contribution: it shows a parameter-efficient way to exploit a frozen LLM block for discriminative vision, and the paper's internal controls are stronger than what is typical for this type of work. The parameter-matched MLP baselines, the random-initialization ablation, and the repeated-seed experiment on ImageNet-1K are genuine strengths. However, the headline ImageNet-1K result is currently anchored to a self-trained MAE baseline that is never reconciled with the published MAE ViT-B accuracy, and the co-adaptation claim lacks a control that isolates LoRA training during the MAE pretraining phase. The significance of the contribution therefore hinges on resolving these two points.

major comments (4)
  1. [4.1, Table 1] The central comparison against the MAE baseline is not calibrated to the published MAE ViT-B result. The paper reports 83.2% for its self-trained MAE ViT-B baseline and claims a "new state-of-the-art" at 83.6%, but the published MAE ViT-B top-1 accuracy is 83.6% (He et al., 2022, ViT-B row in their ImageNet fine-tuning table), and the manuscript neither cites this number nor compares against the released MAE checkpoint. Consequently, the headline +0.4% increment over "MAE Pretrained ViT/B" is, on the primary benchmark, indistinguishable from the gap between the authors' reproduced baseline and the published baseline. Please add the official MAE result to Table 1, state precisely the pretraining protocol (e.g., 800 vs 1600 epochs) and any other differences, and either compare against the released checkpoint or explicitly restrict the state-of-the-art claim to the protocol used in this paper.
  2. [3.2.3, Table 3] The claim that concurrent LoRA training during MAE pretraining is the key to modality bridging is not isolated by the ablations. Row (e) is MAE-pretrained with the LLM block present but no LoRA, while row (h) has LoRA during both pretraining and fine-tuning; the difference therefore conflates LoRA's pretraining contribution with its fine-tuning contribution. A control in which the ViT is MAE-pretrained without the LLM block and the LoRA-adapted LLM block is inserted only at the fine-tuning stage would directly test the "synergistic co-adaptation" mechanism. Without such a control, the +0.4%/+0.3% increments in Table 3 and Table 4 could be due to LoRA adaptation during fine-tuning alone.
  3. [4.1, Table 1; 5, Table 5] The paper uses "significantly" for the robustness gains, but the repeated-seed analysis in Section B.1 covers only ImageNet-1K. The robustness differences in Table 1 (IN-A +2.2, IN-SK +0.8, IN-V2 +0.7, IN-C +0.5) and in Table 5 (IN-9 gaps) are single-run results, and several are small relative to typical seed noise. Please report seed-averaged accuracies with error bars for these benchmarks, or qualify the significance claims accordingly.
  4. [4.3, Table 4] The random-initialization ablation controls for capacity and architecture, but it does not support the specific claim that text-derived semantic knowledge from the LLM is the source of the gain. A control using a pretrained non-linguistic transformer block of comparable capacity (for example, a ViT block pretrained by MAE or DINO) would be needed to attribute the improvement to language pretraining rather than to generic pretrained representations. If such a control is not feasible, the paper should soften the language-semantics interpretation in the abstract and in Section 3.2.3.
minor comments (5)
  1. [5, Eq. (2)] Equation (2) is dimensionally inconsistent: for X in R^{T x d} and WQ, WK in R^{d x dk}, the product WQ WK^T is d x d, not T x T. The intended expression is presumably softmax((X WQ)(X WK)^T / sqrt(d)); please correct it.
  2. [4.3, Table 3] The ablation table skips row (g) but the text refers to "row g" when describing the LUViT parameter match; the "+0.3" entries in row (h) are also unexplained. Please make the row labels and deltas consistent.
  3. [3.2.2, Appendix D] The LoRA configuration is incompletely specified: rank 16 is given, but LoRA alpha and dropout are not reported. Please add these to the training details.
  4. [5] The sentence "This is a direct indication of the focus of the attention patterns for the informative foreground regions for LUViT, informative regions, resulting in..." contains a duplicated fragment and should be rewritten.
  5. [Abstract, 4.1] The phrase "new state-of-the-art" should be scoped to the exact setting (frozen LLM block, MAE-based pretraining, ViT-B, ImageNet-1K under the protocol used here); as written, it invites comparison with the broader self-supervised literature.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: LUViT's claimed gains are empirical comparisons against reproduced MAE and parameter-matched baselines; self-citations are background only.

full rationale

The paper's central claim is an empirical demonstration that jointly training LoRA layers in a frozen LLM block during MAE pretraining improves downstream vision accuracy. No equation in Section 3 defines the outcome in terms of the input, and no fitted parameter is renamed as a prediction. The main comparisons in Tables 1, 3, 4, and 5 are against the authors' own reproduced MAE ViT/B baseline and parameter-matched MLP/LoRA baselines, plus external published numbers from Pang et al.; these are genuine benchmarks, not inputs to a derivation. The ablations (frozen LLM without LoRA, randomly initialized LLM, MLP capacity matches) are controlled experiments rather than circular reductions. The only in-house references (SILC, SigLIP 2) appear in related-work context and do not carry the argument. A reviewer concern that LUViT's 83.6% equals the published MAE number rather than exceeding the reproduced 83.2% baseline is a baseline-validity or correctness issue, not a circularity issue, because 83.6% is a measured outcome and the comparison is not forced by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger contains only hand-chosen hyperparameters and domain assumptions inherited from MAE and LoRA; no physical constants, fitted constants, or invented entities. The paper does not sweep LoRA rank, LoRA target matrices, LLM block index, or the architectural changes (bidirectional attention, RoPE removal).

free parameters (4)
  • LoRA rank r = 16
    Chosen without ablation; LoRA rank controls adapter capacity and is not varied in the paper, yet the central results depend on this configuration.
  • LoRA target projections = query and value
    LoRA is applied only to W_q and W_v; no comparison to other targets such as W_k or the MLP weights, so the choice is not justified experimentally.
  • LLaMA block index = 32 (final block of LLaMA-1 7B)
    Chosen following Pang et al. and Lai et al.; the paper does not study which LLM block depth or which LLM family works best.
  • Thresholds for pseudo-mask IoU analysis = empirically set per model and component
    In Appendix B.3, fixed thresholds for frequency and magnitude masks are determined empirically for each model, which limits the strength of the mask-IoU comparison.
assumptions (4)
  • domain assumption An LLM transformer block adapted only through LoRA can preserve and transfer knowledge from text pretraining to visual tokens.
    This is the paper's central premise, tested only indirectly through downstream accuracy and the random-initialization ablation in Table 4.
  • domain assumption MAE reconstruction is an effective self-supervised pretraining objective for producing features compatible with an LLM block.
    Inherited from He et al. 2022 and assumed to transfer when the LLM block is inserted; stated in Section 3.2.1.
  • domain assumption Bidirectional attention and removal of RoPE make the LLM block suitable for visual token sequences without retraining.
    Section 3.3 states these changes are beneficial based on prior work; no ablation of these architectural choices is provided.
  • domain assumption Average pooling of the final ViT features is a stable classification readout.
    The paper follows the official MAE codebase in using average pooling instead of the CLS token; stated in Appendix D.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs." pith.science (2026). https://pith.science/paper/H67FM4VT

@misc{pith2026250700754,
  author       = {Pith},
  title        = {Pith review of: Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H67FM4VT}},
  note         = {Machine review of arXiv:2507.00754}
}
read the original abstract

The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the rich semantic knowledge and reasoning capabilities of LLMs. However, a fundamental challenge lies in the inherent modality mismatch between text-centric pretraining of LLMs and vision-centric training of ViTs. Direct fusion often fails to fully exploit the LLM's potential and suffers from unstable finetuning. As a result, LLM blocks are kept frozen while only the vision components are learned. As a remedy to these challenges, we introduce Language-Unlocked Vision Transformers (LUViT), a novel approach that bridges this modality mismatch through a synergistic pre-training strategy. LUViT co-adapts a ViT backbone and an LLM fusion block by (1) employing Masked Auto-Encoding (MAE) to pre-train the ViT for richer visual representations, and (2) concurrently training Low-Rank Adaptation (LoRA) layers within the LLM block using the MAE objective. This joint optimization guides the ViT to produce LLM-aligned features and the LLM to effectively interpret visual information. We demonstrate through extensive experiments that LUViT significantly improves performance on various downstream vision tasks, showcasing a more effective and efficient pathway to harness LLM knowledge for visual understanding.

Figures

Figures reproduced from arXiv: 2507.00754 by the authors.

Figure 1
Figure 1. Architecture diagram of our Language-Unlocked Vision Transformer (LUViT). Input image patches are processed by the ViT Encoder. The resulting visual features are then passed through an LLM Fusion Block (comprising linear projections and an LLM transformer block adapted with LoRA). For MAE pre-training, a lightweight decoder reconstructs masked patches. For fine-tuning, the decoder is removed, and a task-specific hea… view at source ↗
Figure 2
Figure 2. Comparison of the image-level average foreground attention entropies vs the image-level [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Visualized attention entropies and patch norms of both LUViT/B and the MAE pre-trained [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of the image-level average foreground attention entropies vs the image-level [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the image-level average foreground attention entropies vs the image-level [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Top-1 accuracy bars along with the means and the standard error ranges of MAE pretrained [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 25 canonical work pages

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    work page 1901

  2. [3]

    In these works, they provided litmus tests for measuring how focused the attention patterns of particular models are and how they relate to model robustness

    Attention entropy patterns have been utilized in the context of neural network robustness in earlier works [Guo et al., 2023, Zhang et al., 2024]. In these works, they provided litmus tests for measuring how focused the attention patterns of particular models are and how they relate to model robustness. As stated in Section 5, we quantify the attention en...

    work page 2023

  3. [4]

    the standard deviation of the accuracy values divided by the number of different seeds

    Finally, we report the average accuracy across the seeds with the accompanying standard error values, i.e. the standard deviation of the accuracy values divided by the number of different seeds. From Table 6 we observe that the LoRA version achieves a slightly improved performance compared to the LM1+ViT/B. Concretely, the LoRA-adapted version of Pang et ...

    work page 2023

  4. [5]

    Frozen Transformers in Language Models Are Effective Visual Encoder Layers

    Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. Frozen transformers in language models are effective visual encoder layers. arXiv preprint arXiv:2310.12973,

    work page Pith review arXiv

  5. [6]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirect...

    arXiv 2019

  6. [11]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202,

    arXiv

  7. [12]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193,

    arXiv

  8. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

    2009

Show all 55 references
  1. [16]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  2. [17]

    Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-based pre-trained language models

    Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, and Dong Yu. Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-based pre-trained language models. arXiv preprint arXiv:2412.16545,

    arXiv

  3. [18]

    Stabilizing transformer training by preventing attention entropy collapse

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770–40803. PML...

    arXiv

  4. [19]

    Understanding why neural networks generalize well through gsnr of parameters

    Jinlong Liu, Guoqing Jiang, Yunzhi Bai, Ting Chen, and Huayan Wang. Understanding why neural networks generalize well through gsnr of parameters. arXiv preprint arXiv:2001.07384,

    2001 arXiv

  5. [20]

    Distilling the knowledge in a neural network

    13 Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  6. [22]

    A single transformer for scalable vision- language modeling

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision- language modeling. arXiv preprint arXiv:2407.06438, 2024c. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are im...

    arXiv

  7. [23]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023b. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer ...

    arXiv

  8. [26]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  9. [28]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017b. Kevin Clark, Minh-Thang Luong, Quoc V L...

    2003 arXiv

  10. [29]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 646–661. Springer,

    2016

  11. [30]

    mixup: Beyond empirical risk minimization

    14 Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412,

    arXiv

  12. [31]

    Pytorch: An imperative style, high-performance deep learning library

    A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703,

    1912 arXiv

  13. [32]

    Mmdetection: Open mmlab detection toolbox and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155,

    1906 arXiv

  14. [33]

    Quantifying the carbon emissions of machine learning

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700,

    1910 arXiv

  15. [34]

    Attention Entropies

    Supplementary Material Contents 1 Introduction 1 2 Background and Related Work 2 3 LUViT: Language-Unlocked Vision Transformers 3 3.1 LUViT: Language-Unlocked Vision Transformer . . . . . . . . . . . . . . . . . . 3 3.2 Synergistic Pre-training for Modality Alignment . . . . ....

    2020

  16. [36]

    For Figure 3, we perform a per-image normalization for both the patch norms and attention entropies to achieve more interpretable visualizations. This corresponds to performing the normalizations based on the lowest and highest attention entropy score or token norm value for e...

    2023

  17. [37]

    Simultaneously, the patch norms are more salient and achieve better coverage of foreground regions for LUViT compared to ViT/B. This behavior is specifically important, since we are utilizing average pooling instead of relying on the [CLS] token, following the default implemen...

    2022

  18. [38]

    Finally, we highlight the high quality of the attention maps of LUViT in the Imagenet-Segmentation dataset [Gao et al., 2022] in Section B.3

    In particular, we report error bars for the Imagenet-1K results presented in Table 1 in Section B.1, and additional results of performing LoRA adaptation of the LLM block in a supervised-only setting in Section B.2. Finally, we highlight the high quality of the attention maps ...

    2022

  19. [39]

    The results of both our reproduction of Pang et al

    in a supervised-only setting. The results of both our reproduction of Pang et al. [2023]’s model and its LoRA-adapted version is presented in Table

    2023

  20. [40]

    For Table 6, we train both Pang et al. [2023]’s LM1+ViT/B and its LoRA- adapted version in a supervised-only setting on Imagenet-1K with three different random seeds, (0, 1, 2), while adhering to all training settings in Pang et al. [2023]. Furthermore, we directly utilize the...

    2023

  21. [42]

    Each reported value is an average of three training runs with three seeds, (0, 1, 2), and the subscript ± denotes the standard error for each setting

    with a supervised-only training regime. Each reported value is an average of three training runs with three seeds, (0, 1, 2), and the subscript ± denotes the standard error for each setting. Note that we report two significant digits in the decimal for highlighting the effect ...

    2022

  22. [43]

    A cell in the downsampled mask is assigned a value of 1 if it overlaps with the original high-resolution mask

    downsample the ground-truth segmentation masks in the ImageNet-Segmentation-50 dataset [Gao et al., 2022] to match the resolution of the model’s feature map. A cell in the downsampled mask is assigned a value of 1 if it overlaps with the original high-resolution mask. Concrete...

    2022

  23. [44]

    first follows from the DeiT [Touvron et al., 2021] family of models and perform classification based on the [CLS] token. Then, the authors made the claim that to achieve a better 21 performance compared to the vanilla ViT/B, either the attention weights should be improving or ...

    2021

  24. [45]

    While LUViT differs from Pang et al

    made the qualitative observation that the attention weights, wv, were noisy, thus concluding that the M 2 L · MLLM projection must be amplifying the most informative tokens. While LUViT differs from Pang et al. [2023]’s frozen-LLM-appended ViTs in several key architectural and...

    2023

  25. [46]

    The authors quantified this alignment through demonstrating improved gradient-signal-to-noise ratio (GSNR) under the presence of the LLM block

    and demonstrated that the gradient flow from different samples towards the weights of the model are more aligned in the presence of the frozen LLM block. The authors quantified this alignment through demonstrating improved gradient-signal-to-noise ratio (GSNR) under the presen...

    2020

  26. [47]

    Following up from this observation and taking inspirations from Tiwari and Shenoy [2023], Bai et al

    also showed that this effect is more pronounced towards layers closer to the LLM block, and that the similar representations between the ViT blocks and the LLM block could be indicative of improvements. Following up from this observation and taking inspirations from Tiwari and...

    2023

  27. [48]

    This auxiliary training objective distills the representations of the frozen-LLM-appended ViT to a vanilla ViT through a similarity loss in-between [Hinton et al., 2015]

    then proposes an auxiliary training objective with the aim of removing the additional inference costs incurred by the LLM block. This auxiliary training objective distills the representations of the frozen-LLM-appended ViT to a vanilla ViT through a similarity loss in-between ...

    2015

  28. [51]

    During the pretraining stage, for both LUViT and our baselines, we additionally employ a lightweight Transformer [Vaswani et al., 2017] decoder, which consists of8 blocks and has a hidden size of

    2017

  29. [53]

    Notably, we utilize average pooling setting instead of relying on the [CLS] token for performing classification

    with 1.0, and Randaugment with (9, 0.5) [Cubuk et al., 2020]. Notably, we utilize average pooling setting instead of relying on the [CLS] token for performing classification. We do so, following the official MAE Github repository’s 3 report of potential instabilities in the lo...

    2020

  30. [54]

    and a Mask R-CNN [He et al., 2017] as the final detection/segmentation head. Notably, achieving competitive fine-grained visual recognition results is very hard with supervised- only ViT backbones, with neither of Imagenet-1K nor Imagenet-22K supervised-pretrained ViT/B models...

    2017

  31. [55]

    renditions

    For each of these corruptions, there are 5 severity levels, with higher number indicating tougher corruptions. In our experiments, we report the average results on all of the aforementioned corruptions with all of their severities for a more comprehensive evaluation. Imagenet-...

    2019

  32. [512]

    Finally, the additional capacity baselines in Section 4 all have additional linear projection layers at the head, analogously with where they are placed in LUViT

    The design of both the ViT/B encoder and the lightweight decoder closely mirror the original MAE design with no changes with the exception of the LLaMA 1 block and the linear projections around it. Finally, the additional capacity baselines in Section 4 all have additional lin...

    2022

  33. [768]

    In addition, for LUViT, we always utilize the 32nd (i.e the final) Transformer [Vaswani et al., 2017] block of the smallest LLaMA 1 [Touvron et al., 2023a] model with 7 billion parameters, which has a hidden size of

    2017

  34. [2009]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261,

    1903 arXiv

  35. [2010]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning , pages 1691–1703. PMLR, 2020b. 11 Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-...

    arXiv

  36. [2014]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  37. [2015]

    Evev2: Improved baselines for encoder-free vision-language models

    Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models. arXiv preprint arXiv:2502.06788,

    arXiv

  38. [2016]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 20...

    arXiv

  39. [2017]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,

    arXiv

  40. [2019]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international c...

    2006 arXiv

  41. [2020]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

    URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ll...

    2020 arXiv

  42. [2021]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832,

    arXiv

  43. [2022]

    Siglip 2: 10 Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: 10 Multilingual vision-language encoders with improved semantic understanding, localization,...

    arXiv

  44. [2023]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024a. Zh...

    arXiv

  45. [2024]

    Vision as lora

    Han Wang, Yongjie Ye, Bingru Li, Yuxiang Nie, Jinghui Lu, Jingqun Tang, Yanjie Wang, and Can Huang. Vision as lora. arXiv preprint arXiv:2503.20680,

    arXiv

  46. [2025]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  47. [4096]

    We choose this block of LLaMA 1 following its success in similar works [Lai et al., 2024, Pang et al., 2023, Bai et al., 2025]. There are two additional linear projections without any non-linearities or additional activations around the LLaMA 1 block to allow matching the hidd...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.