Pith. sign in

REVIEW 3 major objections 3 minor 47 references

Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the reusable knowledge in a pre-trained CLIP model can be compressed into a small weighted-sum block set that initializes descendants matching or beating full pre-training plus fine-tuning.

desk verdict The multimodal-block idea is a real incremental contribution, but the headline comparison against PT-FT is built on an unfair data baseline. read the letter →

arxiv 2506.16673 v1 pith:W7TZCUD5 submitted 2025-06-20 cs.CV

Ruiming Chen , Junming Yang , Shiyu Xia , Xu Yang , Jing Wang , Xin Geng This is my paper · ORCID
classification cs.CV
keywords multimodallearngeneCLIPdistillationknowledgeweighted-sumdecompositionmodelinitializationcross-modalretrievalimagecaptioningparameter-efficienttransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the knowledge a pre-trained CLIP model uses to connect images and text can be extracted into a compact 'multimodal learngene': a few blocks of Transformer weights plus learned coefficients. The extraction is done by training a 12-layer auxiliary CLIP to imitate the larger ancestry model through contrastive distillation, then using the blocks and coefficients to numerically initialize descendant models of 6, 8, or 12 layers, in unimodal or multimodal form. Across image classification, cross-modal retrieval, and image captioning, these descendants perform comparably to or better than models obtained by full pre-training plus fine-tuning, while storing about 25% of the parameters and reducing pre-training cost by about 2.8 times. If true, this would make it cheap to produce CLIP-like models at many scales from a single extraction, instead of re-pre-training each size.

What carries the argument

The weighted-sum parameter decomposition: for layer $i$ in the vision encoder, $P_i^v = c_k^v \theta_j^v + c_k^{mv}\theta_j^m$, with an analogous formula for the language encoder using $\theta_j^l$ and $c_k^{ml}$. Here $\theta_j$ are two groups of blocks (each containing vision, language, and multimodal sub-blocks) and $c_k$ are learned coefficient vectors of half the layer length, with each scalar reused twice. This mechanism does two jobs: it forces the auxiliary model to separate shared cross-modal knowledge (the multimodal block) from modality-specific knowledge during distillation, and it provides a recipe for recombining the same blocks at different repetition patterns to numerically initialize descendant models of varying depth.

What would settle it

Reconstruct each layer's parameters from the extracted blocks and coefficients using the paper's Eqs. (1) and (2), and compare them with the actual layer weights of the ancestry CLIP. If the relative L2 reconstruction error is large (say, above a few percent) and yet the descendant models still match PT-FT performance, then the decomposition itself is not the reason for the gains; if the error is large and the descendants fail, the central claim collapses.

Watch

Extended reading notes

Core claim

The central discovery is that the multimodal and unimodal generalizable knowledge in CLIP can be separated and stored explicitly. The paper proposes MM-LG, which decomposes each Transformer layer's parameters as a weighted sum of a shared multimodal block and a unimodal (vision or language) block, with coefficients of half the layer length; two alternating block groups are shared across the auxiliary model. After training this auxiliary model with a combination of CLIP contrastive loss and distillation loss against the ancestry CLIP, the extracted blocks and coefficients constitute the learngene. These components are then recombined, with repetitions eliminated, to numerically initialize descendant models of different depths and modalities. The experiments show these descendants match or beat the pre-training-and-fine-tuning upper bound on several benchmarks while using only about a quarter of the parameter storage and about 2.8 times less pre-training cost.

Load-bearing premise

The assumption that every layer of the ancestry model can be faithfully represented as a weighted sum of one shared multimodal block and one unimodal block, using only two alternating block groups and coefficient vectors half the layer length, is load-bearing; if that approximation is not faithful, the extracted blocks may not actually contain the teacher's generalizable knowledge.

Editorial extensions

If this is right

  • One extraction pass yields initializations for 6-, 8-, and 12-layer descendants, so new model scales do not require re-pre-training.
  • Descendant models outperform existing learngene baselines (e.g., +3.1% on Oxford-IIIT PET and +4.13% on Flickr30k over TLEG) and sometimes beat the PT-FT upper bound.
  • The same extracted components serve both unimodal tasks (image classification) and multimodal tasks (cross-modal retrieval, image captioning), indicating the multimodal block carries shared knowledge across modalities.
  • Storage for all scales is about 37.4M parameters versus 151.4M for PT-FT, a roughly 75% reduction, and extraction costs about 407.8 GPU-hours versus 1142.4 for PT-FT.
  • Ablation shows removing either the multimodal or the unimodal block causes large drops (e.g., from 65.18 to about 23 on Flickr30k I2T), so the two components cooperate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the weighted-sum decomposition genuinely captures shared multimodal structure, the same extraction could be reused as a seed for other dual-encoder architectures or later fine-tuned on new modality pairs, though the paper does not test this.
  • Beyond the paper: the coefficient-sharing pattern (each scalar used twice, blocks alternating every two layers) resembles a low-rank or weight-multiplexing compression; measuring the actual reconstruction error of Eqs. (1)-(2) against the ancestry weights would tell whether the blocks encode real shared structure or merely fit the auxiliary model's layout.
  • Beyond the paper: because the extracted blocks are dense, combining MM-LG with pruning or quantization at initialization could push storage below 25% while preserving the reported downstream gains; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes MM-LG (Multimodal Learngene), a framework for extracting generalizable knowledge from a pre-trained CLIP model and using it to initialize smaller descendant models of varying depth and modality. The extraction stage builds an auxiliary ViT-S CLIP model whose per-layer parameters are constrained to be weighted sums of shared unimodal (vision/language) blocks and a shared multimodal block, with learnable coefficient vectors; the auxiliary model is trained by a combination of contrastive loss and distillation against a CLIP ViT-B teacher. The initialization stage uses the extracted blocks and coefficients to numerically initialize 12-, 8-, and 6-layer vision and multimodal descendants, followed by a short activation phase and downstream fine-tuning. Experiments cover cross-modal retrieval (COCO, Flickr30k), image classification (CIFAR-100, Food-101, Oxford-IIIT PET), and image captioning (COCO). The authors report that MM-LG outperforms existing Learngene baselines and is comparable or superior to a pre-training-and-fine-tuning (PT-FT) baseline, while storing about 25% of the parameters and reducing pre-training time by about 2.8x.

Significance. If the central claim were established, MM-LG would be a practically useful method for transferring CLIP-like knowledge to multiple smaller models without repeated large-scale pre-training. The paper's pipeline is clearly described: the auxiliary model construction, distillation losses, numerical initialization, and activation procedure are consistent, and the ablation in Table 4 shows that both the multimodal and unimodal components contribute to downstream performance. The strongest asset is the breadth of evaluation across three task families and three model scales, which goes beyond many prior Learngene papers. However, the headline comparison against PT-FT is not currently controlled for the data used by the teacher, and the efficiency accounting omits the teacher's pre-training cost and storage. These issues are load-bearing for the paper's main claims, so the result, while promising, needs additional experimental support before it can be taken as established.

major comments (3)
  1. [Section 4.1 and Appendix A.1] The PT-FT baseline is not a valid upper bound for the claimed comparison. The teacher model is CLIP ViT-B/16 trained on OpenAI's WIT-400M (footnote in Appendix A.1), while the PT-FT baseline is pre-trained from scratch on CC12M+CC3M, roughly 15M image-text pairs. MM-LG therefore inherits knowledge distilled from 400M pairs, whereas PT-FT sees only about 15M pairs. The reported gains over PT-FT (e.g., +1.9% on PET and +3.65% on Flickr30k I2T in the 12-layer setting) may reflect the teacher's data advantage rather than the proposed extraction mechanism. Please add a controlled comparison in which the teacher is pre-trained on the same CC12M+CC3M data used for PT-FT, or in which PT-FT is pre-trained on a comparable data budget to the teacher, and report whether the advantage over PT-FT survives.
  2. [Section 4.5 and Table 3] The efficiency claim is incomplete because it omits the ancestry model's pre-training cost and storage. Table 3 reports 407.8 GPU-hours for MM-LG, but this count excludes the GPU-hours required to pre-train the CLIP ViT-B teacher on 400M image-text pairs; the method depends on that teacher. Similarly, the 37.4M parameter storage count excludes the teacher weights, which must be retained or re-downloaded to run extraction. The abstract's statement that MM-LG 'requires only around 25% of the parameter storage while reducing around 2.8 times pre-training costs compared to the pre-training and fine-tuning paradigm' is therefore not a system-level accounting. Please report the total storage and compute including the teacher, or explicitly scope the claim to the descendant-model storage and the extraction phase only.
  3. [Section 3.1, Eqs. (1) and (2)] The central decomposition—that every Transformer layer's parameters can be faithfully approximated by a weighted sum of a shared multimodal block and a unimodal block, using only two block groups and coefficient vectors of half the layer length—is introduced as a design choice without theoretical justification or a direct test of its expressiveness. The ablation in Table 4 shows that the multimodal block contributes within this specific architecture, and the comparison to TLEG shows an empirical advantage, but the paper does not examine whether the two-group, half-length sharing pattern is too restrictive or whether a different sharing schedule would capture the teacher's knowledge better. Please add an analysis of the approximation error of the learned weighted-sum reconstruction against the teacher layers, or an ablation varying the number of block groups and the coefficient sharing pattern.
minor comments (3)
  1. [Table 1] In the 12-layer Flickr30k row for MM-LG, the I2T and T2I values are both exactly 65.18, which is suspicious and should be rechecked; if it is a typo, the corrected value may affect the reported margin over PT-FT.
  2. [Throughout] There are several typos, including 'leargene' in the abstract, 'Muiltimodal Learngene' in Figure 1, and 'co-corresponding auther' in the author footnote; these should be corrected.
  3. [Appendix A.1] The zero-shot ImageNet accuracy of the auxiliary model (41.71%) and the retrieval recalls on CC3M are useful sanity checks, but they are reported without comparison to the teacher or to a ViT-S trained from scratch on the same data; adding these numbers would help calibrate how much knowledge is retained in the auxiliary model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MM-LG fits a distilled decomposition to CLIP, then tests on held-out downstream tasks; the central claims do not reduce to the fitted values.

full rationale

MM-LG's derivation chain is a two-stage initialization scheme rather than a circular prediction. Stage 1 (Sections 3.1-3.2) defines an auxiliary model whose layer weights are a weighted sum of shared multimodal/unimodal blocks and coefficients (Eqs. 1-2), and fits these blocks/coefficients by distilling CLIP logits (Eqs. 5-7) plus a contrastive objective (Eq. 4), giving L_train = L_CLIP + lambda * L_dist (Eq. 8). Stage 2 (Section 3.3) numerically initializes descendant models by reusing the same blocks/coefficients. Because the fitted parameters are the initialization, not the claimed output, there is no self-definitional step: the paper's claims (gains over TLEG, parity or gains over PT-FT, storage/cost efficiency) are evaluated on independent benchmarks (COCO, Flickr30k, CIFAR-100, Food-101, PET, COCO captioning) that do not appear in the extraction loss. The weighted-sum ansatz is an explicit modeling choice, not an imported uniqueness theorem, and Table 4's ablation provides an empirical, non-circular check on the role of the multimodal block. The Learngene framing is heavily self-cited (e.g., [Wang et al., 2022], [Xia et al., 2024b]), but those citations only motivate the paradigm and are not the sole evidence for the central result; the experiments are self-contained against external baselines. The main legitimate concerns are data/accounting fairness: the CLIP teacher saw 400M image-text pairs while the PT-FT baseline uses only roughly 15M pairs, and the teacher's pre-training cost/storage is excluded from the efficiency tally. These are correctness risks, not circularity under the rubric. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The free parameters listed are hand-chosen design choices that affect the extracted learngene and downstream performance. The axioms are the load-bearing assumptions about the decomposition and distillation transfer. The invented entity is the multimodal block, which lacks independent evidence outside the paper's own experiments.

free parameters (4)
  • Loss weight lambda = 1
    Chosen in Appendix A.1 to balance the CLIP loss and distillation loss; not swept or justified.
  • Activation data fraction = 10%
    Appendix A.2 states that 10% of the original pre-training data for one epoch is enough; this hand-chosen value affects downstream performance (Table 5).
  • Number of block groups = 2
    Section 3.1 specifies two groups of blocks used alternately; this structural choice is not derived from theory.
  • Block sharing pattern = Each group shared twice before switching
    The alternating sharing scheme in Section 3.1 and Figure 3(a) is an ad hoc design choice that determines how coefficients map to layers.
assumptions (4)
  • ad hoc to paper The parameters of each Transformer layer in the ancestry model can be closely approximated by a weighted sum of a shared multimodal block and a unimodal block with learnable coefficients.
    Central to Equations (1) and (2) in Section 3.1; no proof or empirical validation that this decomposition captures the teacher's function.
  • domain assumption Knowledge distillation on CC12M+CC3M transfers the generalizable knowledge of the much larger CLIP teacher into the small auxiliary model.
    The extraction relies on this transfer (Section 3.2); it is a standard but unproven assumption in distillation.
  • domain assumption Fine-tuning on downstream tasks measures the quality of the extracted generalizable knowledge.
    The paper validates the method through downstream performance (Section 4), which assumes these metrics reflect generalization.
  • domain assumption The pre-trained CLIP ViT-B teacher is a freely available resource, so its 400M-pair pre-training does not count against the method's cost.
    The efficiency claim (Table 3) counts only the extraction cost, not the teacher's pre-training; this makes the comparison to PT-FT asymmetric.
invented entities (1)
  • Multimodal block (theta_m)
    purpose: To capture cross-modal generalizable knowledge shared between vision and language branches during extraction and initialization.
    It is a new architectural component introduced in this paper; its benefit is shown only through the in-paper ablation (Table 4), and no external or theoretical evidence supports it as a distinct carrier of cross-modal knowledge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge." pith.science (2026). https://pith.science/paper/W7TZCUD5

@misc{pith2026250616673,
  author       = {Pith},
  title        = {Pith review of: Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7TZCUD5}},
  note         = {Machine review of arXiv:2506.16673}
}
read the original abstract

CLIP (Contrastive Language-Image Pre-training) has attracted widespread attention for its multimodal generalizable knowledge, which is significant for downstream tasks. However, the computational overhead of a large number of parameters and large-scale pre-training poses challenges of pre-training a different scale of CLIP. Learngene extracts the generalizable components termed as learngene from an ancestry model and initializes diverse descendant models with it. Previous Learngene paradigms fail to handle the generalizable knowledge in multimodal scenarios. In this paper, we put forward the idea of utilizing a multimodal block to extract the multimodal generalizable knowledge, which inspires us to propose MM-LG (Multimodal Learngene), a novel framework designed to extract and leverage generalizable components from CLIP. Specifically, we first establish multimodal and unimodal blocks to extract the multimodal and unimodal generalizable knowledge in a weighted-sum manner. Subsequently, we employ these components to numerically initialize descendant models of varying scales and modalities. Extensive experiments demonstrate MM-LG's effectiveness, which achieves performance gains over existing learngene approaches (e.g.,+3.1% on Oxford-IIIT PET and +4.13% on Flickr30k) and comparable or superior results to the pre-training and fine-tuning paradigm (e.g.,+1.9% on Oxford-IIIT PET and +3.65% on Flickr30k). Notably, MM-LG requires only around 25% of the parameter storage while reducing around 2.8 times pre-training costs for diverse model scales compared to the pre-training and fine-tuning paradigm, making it particularly suitable for efficient deployment across diverse downstream tasks.

Figures

Figures reproduced from arXiv: 2506.16673 by the authors.

Figure 1
Figure 1. (a) Learngene paradigm comprises two stages. The first [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of our proposed method comprises two stages. In the first stage, we construct an auxiliary model wherein the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) For a 12-layer auxiliary model, during its construction, two groups of blocks [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison across layers and methods on CIFAR-100, Food-101, and Oxford-IIIT PET datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages

  1. [1]

    [Baet al., 2016 ] Jimmy Ba, Jamie Ryan Kiros, and Ge- offrey E. Hinton. Layer normalization.ArXiv, abs/1607.06450,

    arXiv 2016

  2. [11]

    Distilling the knowledge in a neural network

    [Hintonet al., 2015 ] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv 2015

  3. [13]

    Scaling up visual and vision-language representation learning with noisy text su- pervision

    [Jiaet al., 2021 ] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. InInternational conference on machine learn- ing, pages 4904–4916. PMLR,

    work page 2021

  4. [14]

    Learning multiple layers of features from tiny im- ages.Handbook of Systemic Autoimmune Diseases, 1(4),

    [Krizhevskyet al., 2009 ] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages.Handbook of Systemic Autoimmune Diseases, 1(4),

    work page 2009

  5. [15]

    Clipath: Fine-tune clip with visual fea- ture fusion for pathology image analysis towards min- imizing data collection efforts

    [Laiet al., 2023 ] Zhengfeng Lai, Zhuoheng Li, Luca Cerny Oliveira, Joohi Chauhan, Brittany N Dugger, and Chen- Nee Chuah. Clipath: Fine-tune clip with visual fea- ture fusion for pathology image analysis towards min- imizing data collection efforts. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2374–2380,

    work page 2023

  6. [16]

    Albert: A lite bert for self-supervised learning of lan- guage representations.ArXiv, abs/1909.11942,

    [Lanet al., 2019 ] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of lan- guage representations.ArXiv, abs/1909.11942,

    arXiv 2019

  7. [17]

    Align before fuse: Vision and language representation learning with momentum distilla- tion.Advances in neural information processing systems, 34:9694–9705,

    [Liet al., 2021 ] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distilla- tion.Advances in neural information processing systems, 34:9694–9705,

    work page 2021

  8. [18]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    [Liet al., 2023 ] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR,

    2023

Show all 47 references
  1. [19]

    Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics

    [Lin and Och, 2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. InProceedings of the 42nd Annual Meeting on Associa- tion for Computational Linguistics - ACL ’04, Jan

    2004

  2. [21]

    Foldgpt: Simple and effective large language model com- pression scheme.ArXiv, abs/2407.00928,

    [Liuet al., 2024 ] Songwei Liu, Chao Zeng, Lianqiang Li, Chenqian Yan, Lean Fu, Xing Mei, and Fangmin Chen. Foldgpt: Simple and effective large language model com- pression scheme.ArXiv, abs/2407.00928,

    2024 arXiv

  3. [22]

    Clip-branches: Interactive fine-tuning for text- image retrieval

    [L¨ulfet al., 2024 ] Christian L¨ulf, Denis Mayr Lima Martins, Marcos Antonio Vaz Salles, Yongluan Zhou, and Fabian Gieseke. Clip-branches: Interactive fine-tuning for text- image retrieval. InProceedings of the 47th International ACM SIGIR Conference on Research and Developme...

    2024

  4. [23]

    [Mokady and Hertz, 2021] Ron Mokady and Amir Hertz

    Association for Computing Machinery. [Mokady and Hertz, 2021] Ron Mokady and Amir Hertz. Clipcap: Clip prefix for image captioning.ArXiv, abs/2111.09734,

    2021 arXiv

  5. [24]

    Compact language models via pruning and knowledge distillation

    [Muralidharanet al., 2024 ] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Moham- mad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. InTh...

    2024

  6. [25]

    CHiLS: Zero- shot image classification with hierarchical label sets

    [Novacket al., 2023 ] Zachary Novack, Julian Mcauley, Zachary Chase Lipton, and Saurabh Garg. CHiLS: Zero- shot image classification with hierarchical label sets. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edi- ...

    2023

  7. [26]

    [Papineniet al., 2001 ] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu. InProceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics - ACL ’02, Jan

    2001

  8. [29]

    Language models are unsupervised multitask learners

    [Radfordet al., 2019 ] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9,

    2019

  9. [30]

    Learning transferable visual models from nat- ural language supervision

    [Radfordet al., 2021 ] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. InInternational conferen...

    2021

  10. [31]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    [Sharmaet al., 2018 ] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol...

    2018

  11. [33]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    [Vaswaniet al., 2017 ] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Neural Information Processing Systems,Neural In- formation Processing Systems, Jun

    2017

  12. [35]

    Characterizing and avoid- ing negative transfer

    [Wanget al., 2019 ] Zirui Wang, Zihang Dai, Barnabas Poc- zos, and Jaime Carbonell. Characterizing and avoid- ing negative transfer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June

    2019

  13. [36]

    Learngene: From open-world to your learning task

    [Wanget al., 2022 ] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi-Yu Xia, Lei Qi, and Ning Xu. Learngene: From open-world to your learning task. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557–8565,

    2022

  14. [37]

    Learngene: Inheriting con- densed knowledge from the ancestry model to descendant models.arXiv preprint arXiv:2305.02279,

    [Wanget al., 2023 ] Qiufeng Wang, Xu Yang, Shuxia Lin, Jing Wang, and Xin Geng. Learngene: Inheriting con- densed knowledge from the ancestry model to descendant models.arXiv preprint arXiv:2305.02279,

    2023 arXiv

  15. [38]

    Vision transformers as probabilistic expan- sion from learngene

    [Wanget al., 2024 ] Qiufeng Wang, Xu Yang, Haokun Chen, and Xin Geng. Vision transformers as probabilistic expan- sion from learngene. InForty-first International Confer- ence on Machine Learning,

    2024

  16. [39]

    Exploring learngene via stage-wise weight sharing for initializing variable-sized models.arXiv preprint arXiv:2404.16897,

    [Xiaet al., 2024a ] Shi-Yu Xia, Wenxuan Zhu, Xu Yang, and Xin Geng. Exploring learngene via stage-wise weight sharing for initializing variable-sized models.arXiv preprint arXiv:2404.16897,

    arXiv

  17. [40]

    Kind: Knowledge integra- tion and diversion in diffusion models.arXiv preprint arXiv:2408.07337,

    [Xieet al., 2024 ] Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, and Yong Rui. Kind: Knowledge integra- tion and diversion in diffusion models.arXiv preprint arXiv:2408.07337,

    2024 arXiv

  18. [41]

    Clip-cid: Efficient clip distillation via cluster-instance discrimination.ArXiv, abs/2408.09441,

    [Yanget al., 2024b ] Kaicheng Yang, Tiancheng Gu, Xiang An, Haiqiang Jiang, Xiangzi Dai, Ziyong Feng, Wei- dong Cai, and Jiankang Deng. Clip-cid: Efficient clip distillation via cluster-instance discrimination.ArXiv, abs/2408.09441,

    arXiv

  19. [42]

    [Younget al., 2014 ] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the Association for Computational Linguistics, 2:67–78,

    2014

  20. [43]

    Sigmoid loss for language image pre-training

    [Zhaiet al., 2023 ] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 11975– 11986,

    2023

  21. [44]

    Minivit: Compressing vision transformers with weight multiplexing

    [Zhanget al., 2022 ] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12145...

    2022

  22. [45]

    Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582,

    [Zhanget al., 2024 ] Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582,

    2024 arXiv

  23. [46]

    Learning clip guided visual- text fusion transformer for video-based pedestrian attribute recognition

    [Zhuet al., 2023 ] Jun Zhu, Jiandong Jin, Zihan Yang, Xiao- hao Wu, and Xiao Wang. Learning clip guided visual- text fusion transformer for video-based pedestrian attribute recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2023

  24. [47]

    8 with the loss weightλ set to1

    The distillation loss for the auxiliary model is provided in Eq. 8 with the loss weightλ set to1. Through our proposed extraction methodology, we successfully isolate three key components: (1) the multimodal learngene that captures generalizable knowledge across modalities, (2...

    2021

  25. [2001]

    Cats and dogs

    [Parkhiet al., 2012 ] Omkar M Parkhi, Andrea Vedaldi, An- drew Zisserman, and CV Jawahar. Cats and dogs. In2012 IEEE conference on computer vision and pattern recogni- tion, pages 3498–3505. IEEE,

    2012

  26. [2004]

    Microsoft coco: Com- mon objects in context

    [Linet al., 2014 ] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- be...

    2014

  27. [2009]

    Bert: Pre-training of deep bidirectional transformers for language understand- ing

    [Devlinet al., 2019 ] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. InNorth American Chapter of the Association for Computational Linguistics,

    2019

  28. [2012]

    Clipping: Distilling clip-based models with a stu- dent base for video-language retrieval

    [Peiet al., 2023 ] Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with a stu- dent base for video-language retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  29. [2014]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    [Changpinyoet al., 2021 ] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2021

  30. [2015]

    Openclip, July

    [Ilharcoet al., 2021 ] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Ro- han Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July

    2021

  31. [2016]

    Vlmo: Unified vision-language pre-training with mixture-of-modality- experts.Advances in Neural Information Processing Sys- tems, 35:32897–32912,

    [Baoet al., 2022 ] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Sub- hojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality- experts.Advances in Neural Information Processing Sys- tems, 35...

    2022

  32. [2017]

    Lawrence Zitnick, and Devi Parikh

    [Vedantamet al., 2015 ] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun

    2015

  33. [2018]

    Building variable- sized models via learngene pool

    [Shiet al., 2024 ] Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, and Xin Geng. Building variable- sized models via learngene pool. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14946–14954,

    2024

  34. [2019]

    An image is worth 16x16 words: Transformers for image recognition at scale

    [Dosovitskiy, 2020] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    2020 arXiv

  35. [2020]

    Transferring core knowledge via learngenes.arXiv preprint arXiv:2401.08139,

    [Fenget al., 2024a ] Fu Feng, Jing Wang, and Xin Geng. Transferring core knowledge via learngenes.arXiv preprint arXiv:2401.08139,

    arXiv

  36. [2021]

    Reproducible scaling laws for contrastive language-image learning

    [Chertiet al., 2023 ] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF Conferen...

    2023

  37. [2022]

    Food-101–mining discriminative com- ponents with random forests

    [Bossardet al., 2014 ] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative com- ponents with random forests. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446–46...

    2014

  38. [2023]

    Imagenet: A large-scale hierarchical image database

    [Denget al., 2009 ] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

    2009

  39. [2024]

    Wave: Weight template for adaptive initialization of variable-sized models.arXiv preprint arXiv:2406.17503,

    [Fenget al., 2024b ] Fu Feng, Yucheng Xie, Jing Wang, and Xin Geng. Wave: Weight template for adaptive initialization of variable-sized models.arXiv preprint arXiv:2406.17503,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.