Pith. sign in

REVIEW 3 major objections 5 minor 24 references

CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLIP-UP claims that sparse upcycling—converting a pre-trained dense CLIP into a mixture-of-experts model—makes text-to-image retrieval substantially better and cheaper than training a larger dense CLIP.

desk verdict A practical single-stage sparse-upcycling recipe for CLIP with honest ablations, but the headline retrieval margins rest on one hand-picked capacity factor and single runs, so treat the exact numbers as preliminary. read the letter →

arxiv 2502.00965 v2 pith:ZPXWVRZV submitted 2025-02-03 cs.CV cs.LG

Xinze Wang , Chen Chen , Yinfei Yang , Hong-You Chen , Bowen Zhang , Aditya Pal , Xiangxin Zhu , Xianzhi Du This is my paper · ORCID
classification cs.CVcs.LG
keywords mixture-of-expertsCLIPsparseupcyclingtext-to-imageretrievalcontrastivelearningexpertcapacitymultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CLIP-UP, a training recipe that converts a pre-trained dense CLIP model into a sparse mixture-of-experts (MoE) model by replacing half of each encoder's MLP layers with expert layers initialized from the dense weights. The central claim is that this 'sparse upcycling' warm start is cheaper and more stable than training an MoE CLIP from scratch, and that it improves text-image retrieval substantially: a B/16 sparse model beats its dense counterpart by 7.2% and 6.6% on COCO and Flickr30k text-to-image Recall@1, and even beats a larger dense L/14 model on retrieval while using only about 30% of the inference FLOPs. The paper also reports that the same recipe transfers from B/32 to L/14. It acknowledges a trade-off: the gains are mostly on retrieval, while ImageNet classification improves only marginally, and the authors have not found a single configuration that is best for both.

What carries the argument

The central mechanism is sparse upcycling with per-modality separated experts: each MoE layer replaces a dense MLP with E=8 expert MLPs initialized by copying the pre-trained dense weights, plus a randomly initialized router that selects the top-2 experts per token. Each expert has a fixed token buffer set by the capacity factor C, giving capacity $B_e=(B_t/E)\times C$ per expert on a first-come-first-served basis; tokens that exceed capacity are dropped. This token-dropping behavior, together with the load-balance and router-z losses, is what the paper credits for the retrieval gains, and the same capacity factor is the dial that shifts performance between classification and retrieval.

What would settle it

Run the identical CLIP-UP recipe at expert capacity factors C=2.0 and C=4.0 on the same data with several random seeds, and compare COCO text-to-image Recall@1 to the dense baseline; if the C=2.0 model does not consistently beat dense CLIP, or if C=4.0 also beats it while matching ImageNet accuracy, the claimed retrieval advantage and trade-off mechanism are not stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a pre-trained dense CLIP checkpoint can be turned into a high-performing sparse MoE model by copying the dense MLP weights into E experts, randomly initializing routers, and fine-tuning with a reduced learning rate. With separated per-modality backbones, 8 experts per modality, top-2 routing, and an expert capacity factor C=2.0, the resulting CLIP-UP B/16 model reaches 52.1% COCO text-to-image Recall@1 and 80.9% Flickr30k text-to-image Recall@1, surpassing the dense B/16 model (44.9% and 74.3%) and even the dense L/14 model (50.2% and 79.3%) at a fraction of the inference compute. The paper further claims this upcycling recipe requires fewer total training FLOPs than training a sparse MoE from scratch, and that simple load-balance and router z-loss auxiliary losses suffice, without the more complex entropy losses used by prior MoE CLIP training.

Load-bearing premise

The retrieval advantage depends on the hand-picked expert capacity factor C=2.0 and the chosen loss weights; the paper provides no rule for setting them, and its own appendix shows C=4.0 flips the result.

Editorial extensions

If this is right

  • If CLIP-UP is correct, scaling a CLIP model's capacity no longer requires training a larger dense model or an MoE from scratch: a dense checkpoint can be upgraded in roughly the same training budget.
  • Retrieval systems could ship a B/16 sparse model that beats an L/14 dense model at a fraction of the inference cost, making high-quality text-to-image search cheaper to deploy.
  • The recipe's insensitivity to complex auxiliary losses (load balance and router-z loss suffice) lowers the barrier to adopting MoE in multimodal models.
  • The reported classification–retrieval trade-off means the expert capacity factor becomes a task-specific dial, not a free lunch: raising it helps ImageNet but hurts COCO retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If token dropping is genuinely what drives the retrieval gains, then an adaptive capacity factor that drops more tokens on cluttered images and fewer on single-object images could break the ImageNet–COCO trade-off; this is testable.
  • Editorial inference: Because experts are copied from a dense model that already aligns images and text, CLIP-UP could serve as a cheap way to upgrade existing deployed CLIP models without retraining from scratch, a deployment consequence the paper does not discuss.
  • Editorial inference: The observed expert specialization patterns suggest that upcycled experts may become interpretable modules; probing whether these specializations transfer to other datasets or modalities could clarify what the router actually learns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CLIP-UP, a training recipe that converts a pretrained dense CLIP model into a mixture-of-experts (MoE) CLIP via sparse upcycling: alternating MLP layers are replaced by MoE layers whose experts are initialized from the dense checkpoint, routers are random, and fine-tuning uses load-balance and router z-losses. The central empirical claims are that the upcycled B/16 model outperforms its dense counterpart by 7.2 and 6.6 points on COCO and Flickr30K text-to-image Recall@1, respectively, that it surpasses a dense L/14 model on retrieval while using roughly 31% of the inference GFLOPs, and that this is achieved with less total training compute than training from scratch. The paper also reports ablations of shared versus separated backbones, the LIMOE auxiliary loss, expert capacity factor, per-modality MoE placement, and routing analyses.

Significance. If the recipe reproduces across seeds and settings, it is a practically valuable result: it offers a simple, low-cost route from an existing dense checkpoint to a sparse model with better retrieval at lower inference cost. The paper is unusually honest about the ImageNet/retrieval trade-off, and the ablation coverage is broad, including capacity factor, backbone sharing, and auxiliary losses. However, the headline numerical claims rest on a single hand-selected hyperparameter and on single-run measurements, so the current evidence supports the direction of the result more strongly than the specific magnitudes claimed.

major comments (3)
  1. [§3.3, Table 2; Appendix B.2, Table 7] The headline B/16 gains over the dense 790k baseline (COCO T2I R@1 52.1 vs 44.9, Flickr30K T2I R@1 80.9 vs 74.3) are produced by the default expert capacity factor C=2.0. The paper's own ablation, Appendix Table 7, shows that setting C_image=4.0 changes COCO T2I R@1 from 52.1 to 46.3 and Flickr30K T2I R@1 from 80.9 to 75.5, shrinking the gains to +1.4 and +1.2 and eliminating the margin over dense L/14 (46.3 vs 50.2 on COCO). Since no multi-seed evidence and no principled selection rule for C are provided, the paper does not currently establish that the reported advantage is a stable property of the recipe rather than a tuned operating point. The limitation section acknowledges the trade-off, but it does not address the stability of the central quantitative claims.
  2. [§3.3, Table 2; Appendix B.2, Table 7] All reported evaluations appear to be single runs with no error bars or repeated seeds. This matters specifically because the decision-relevant comparison 'CLIP-UP B/16 surpasses dense L/14' is only 1.9 COCO points (52.1 vs 50.2), while the capacity sweep in Table 7 moves the same metric by 5.8 points. Without variance estimates, the reader cannot tell whether the margin over L/14 is significant or an artifact of the selected C and initialization seed. Please report multiple seeds or bootstrap confidence intervals for at least the main configurations, including the dense baselines.
  3. [§1, §3, Appendix A.1] The recipe's exact settings are reported inconsistently, which undermines reproducibility for a paper whose contribution is a recipe. Section 3 states the router z-loss weight as β=0.001, while Appendix Table 3 reports 0.0001. The abstract claims a 6.6% gain on Flickr30K while the introduction's contribution 2 claims 5.5%. These discrepancies should be reconciled and the final numbers aligned across abstract, introduction, tables, and appendix.
minor comments (5)
  1. [§3.3 vs Abstract] The abstract says '30% of the inference FLOPs' while §3.3 says '31% of the GFLOPS'; Table 2 gives 54.3/175.5 ≈ 31%, so please standardize the wording and numbers.
  2. [Figure 6, Appendix B.2] The x-axis of Figure 6 is labeled 'Top-K' even though the surrounding text and Table 7 describe the expert capacity factor C; please correct the axis label or the caption to avoid confusion.
  3. [Throughout] There are several typos and formatting issues: 'ZFLOPs' in §3.1 and Figure 1, 'seprated' in Figure 3's axis, 'F LICKR30K' in Table 1, and 'A CC@5' in Table 2. A copyedit pass is needed.
  4. [Figure 8, Appendix B.3] The caption and text say that post-routing normalization improves image-to-text retrieval but worsens text-to-image retrieval, but the figure does not clearly label which curve is 'before' and which is 'after' beyond the axis text; please add an explicit legend.
  5. [§5, Appendix B.2] The limitation section discusses the ImageNet/retrieval trade-off qualitatively; citing the concrete numbers from Appendix Table 7 (e.g., C_image=2.0 gives 76.9 ImageNet / 52.1 COCO T2I vs C_image=4.0 giving 78.4 / 46.3) in the main text would make the trade-off much clearer to readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CLIP-UP reports measured retrieval/classification results against matched dense baselines; the gains are empirical outcomes, not predictions encoded by construction in the recipe.

full rationale

This is an empirical recipe paper, not a derivation, so the definitional circularity patterns do not apply. The reported CLIP-UP numbers (COCO/Flickr T2I R@1, ImageNet Acc@1) are measurements on standard benchmarks after training; none of the paper's equations (1)-(4) or the sparse-upcycling initialization contains the target eval values, and the dense comparison models are trained with matched step counts (790k dense vs. 440k dense + 350k upcycle). The expert capacity factor C=2.0 and loss weights are user-chosen hyperparameters, not fitted to reproduce the headline results. Appendix Table 7 does show sensitivity (C=4.0 changes COCO T2I R@1 from 52.1 to 46.3 while improving ImageNet from 76.9 to 78.4), but hyperparameter sensitivity is a robustness/selection concern, not circularity: the reported evaluations are not determined by the choice of C through any identity. Self-citations such as Wu et al. for the WIT-300M training data and Radford et al. for CLIP are data/method provenance, not load-bearing uniqueness arguments. The limitation section explicitly acknowledges the retrieval/classification trade-off and states that no configuration gives large gains on both, which is consistent with an honest empirical report rather than a result forced by construction. The internal inconsistencies (abstract says 6.6% on Flickr30k while the introduction says 5.5%; Section 3 gives router-z loss weight beta=0.001 while Appendix Table 3 gives 0.0001) are copy-editing or reporting inconsistencies, not circular steps, and do not alter the circularity verdict.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a set of hand-chosen hyperparameters (capacity factor, loss weights, learning rate, MoE structure) and on standard CLIP/domain assumptions about data and zero-shot transfer. No new physical or mathematical entities are introduced, but the capacity factor C in particular is a free parameter that the paper shows to be highly influential on the reported results.

free parameters (5)
  • Expert capacity factor C = 2.0 for both modalities
    Chosen by hand; Appendix B.2 and Table 7 show it dictates the retrieval/classification trade-off, and C=4.0 reverses the headline COCO gain.
  • Load-balance loss weight alpha = 0.01
    Set following ST-MoE; the paper reports this value in Section 3 but does not ablate it.
  • Router z-loss weight beta = 0.001 in Section 3, 0.0001 in Appendix A
    The paper gives inconsistent values for this hyperparameter, which affects the auxiliary loss used in all reported models.
  • Upcycling peak learning rate and weight decay = LR 5e-5, WD 0.05
    Reduced from dense training values (LR 5e-4, WD 0.2) for stability; chosen by the authors without ablation.
  • MoE configuration (experts, top-K, placement) = 8 experts, top-2, alternating dense/sparse MLP layers
    These architectural choices follow prior MoE work and are not derived from first principles or ablated in this paper.
assumptions (4)
  • domain assumption The CLIP contrastive objective transfers to zero-shot classification and retrieval benchmarks.
    The entire evaluation relies on zero-shot transfer to ImageNet, COCO, and Flickr30K, an established but unproved property of CLIP training.
  • domain assumption WIT-300M and DFN-5B provide a sufficient and appropriately filtered training corpus for the reported dense and MoE CLIP models.
    The paper uses these datasets without reporting contamination checks on the downstream retrieval and classification benchmarks.
  • ad hoc to paper The alternating [dense, sparse] MoE pattern and top-2 routing are stable and effective for upcycled CLIP models.
    These choices are taken from ST-MoE and other language-model MoE work; the paper does not ablate the placement pattern for CLIP specifically.
  • domain assumption The load-balance and router z-losses are sufficient to stabilize sparse upcycling without more complex LIMOE entropy losses.
    The paper shows LIMOE losses help only in one configuration and hurt in others, but it does not provide a thorough search over loss combinations for the final recipe.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling." pith.science (2026). https://pith.science/paper/ZPXWVRZV

@misc{pith2026250200965,
  author       = {Pith},
  title        = {Pith review of: CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZPXWVRZV}},
  note         = {Machine review of arXiv:2502.00965}
}
read the original abstract

Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. While integrating MoE into multimodal models like CLIP improves performance, training these models is notoriously challenging and expensive. We propose CLIP-Upcycling (CLIP-UP), an efficient alternative training strategy that converts a pre-trained dense CLIP model into a sparse MoE architecture. Through extensive experimentation with various settings and auxiliary losses, we demonstrate that CLIP-UP significantly reduces training complexity and cost. Remarkably, our sparse CLIP B/16 model, trained with CLIP-UP, outperforms its dense counterpart by 7.2% and 6.6% on COCO and Flickr30k text-to-image Recall@1 benchmarks respectively. It even surpasses the larger CLIP L/14 model on this task while using only 30% of the inference FLOPs. We further demonstrate the generalizability of our training recipe across different scales, establishing sparse upcycling as a practical and scalable approach for building efficient, high-performance CLIP models.

Figures

Figures reproduced from arXiv: 2502.00965 by the authors.

Figure 1
Figure 1. Our proposed MoE CLIP pre-training recipe. We highlight key factors for efficient training, including backbone sharing, training from scratch vs. sparse upcycling, and auxiliary losses. A detailed analy￾sis is provided in Section 3.1 and Section 3.2. LIMOE outperforms dense CLIP but demands 1.35× more training FLOPs (Mustafa et al., 2022). To address this, we explore sparse upcycling (Ko￾matsuzaki et al., 2023), whi… view at source ↗
Figure 2
Figure 2. CLIP-UP overview with sparse upcycling initialization. Selected MLP layers are replaced with MoE layers, initialized from the dense checkpoint, while routers are randomly initialized. via sparse upcycling, avoiding complex auxil￾iary losses and outperforming existing methods across shared and separated backbones. 2. CLIP-UP significantly improves performance on text-image retrieval, surpassing dense CLIP by 7.2% and… view at source ↗
Figure 3
Figure 3. Impact of LIMOE auxiliary loss under dif [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance vs. training EFLOPS for CLIP [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: CLIP-UP with MoE upcycling for only the text encoder, image encoder, or both. We observe upcycling [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: , increasing Cimage to 4.0 significantly boosts the ImageNet zero-shot metrics. However, this adjustment results in a noticeable drop in performance on COCO and Flickr30k retrieval tasks. 1 5 10 Top-K 40 50 60 70 80 COCO T2I R@1 (%) 1 5 10 Top-K 50 60 70 80 90 COCO I2T…
Figure 7
Figure 7. Figure 7: Visualization of image tokens dropped by the router (i.e., not being assigned to any expert due to capacity [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Model performance for gating normalization applied before or after routing [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Visualization of the token assignment ratios to each expert in each layer by the router on ImageNet — [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Visualization of the token assignment ratios to each expert in each layer by the router on COCO — [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Visualization of expert preference pattern examples. Red bounding boxes highlight the image tokens [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 7 canonical work pages

  1. [4]

    Preprint, arXiv:2309.17425

    Data filtering networks. Preprint, arXiv:2309.17425. William Fedus, Barret Zoph, and Noam Shazeer

    arXiv

  2. [5]

    Preprint, arXiv:2101.03961

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961. Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao

    arXiv

  3. [6]

    Preprint, arXiv:2210.09263

    Vision-language pre- training: Basics, recent advances, and future trends. Preprint, arXiv:2210.09263. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul De- sai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. 2021a. The many faces of robustness: A critical analy- sis of out-of-...

    arXiv

  4. [7]

    Preprint, arXiv:2212.05055

    Sparse upcycling: Training mixture-of-experts from dense checkpoints. Preprint, arXiv:2212.05055. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick

    arXiv

  5. [9]

    Preprint, arXiv:2304.08485

    Visual instruction tuning. Preprint, arXiv:2304.08485. Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby

    arXiv

  6. [10]

    Preprint, arXiv:2206.02770

    Mul- timodal contrastive learning with limoe: the language-image mixture of experts. Preprint, arXiv:2206.02770. Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik

    arXiv

  7. [12]

    Preprint, arXiv:1910.10683

    Exploring the limits of transfer learning with a unified text-to-text trans- former. Preprint, arXiv:1910.10683. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever

    arXiv 1910

  8. [13]

    Preprint, arXiv:2102.12092

    Zero-shot text-to-image gener- ation. Preprint, arXiv:2102.12092. Yongming Rao, Wenliang Zhao, Guangyi Chen, Yan- song Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu

    arXiv

Show all 24 references
  1. [14]

    Preprint, arXiv:2112.01518

    Denseclip: Language-guided dense prediction with context-aware prompting. Preprint, arXiv:2112.01518. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar

    arXiv

  2. [19]

    Preprint, arXiv:2402.01739

    Openmoe: An early effort on open mixture-of-experts language models. Preprint, arXiv:2402.01739. Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, and Xipeng Qiu

    arXiv

  3. [20]

    Preprint, arXiv:2402.12399

    Turn waste into worth: Rectifying top- k router of moe. Preprint, arXiv:2402.12399. Jihai Zhang, Xiaoye Qu, Tong Zhu, and Yu Cheng

    arXiv

  4. [21]

    Preprint, arXiv:2409.19291

    Clip-moe: Towards building mixture of experts for clip with diversified multiplet upcycling. Preprint, arXiv:2409.19291. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu

    arXiv

  5. [22]

    Preprint, arXiv:2202.08906

    St-moe: Designing stable and transferable sparse expert models. Preprint, arXiv:2202.08906. 6 Preprint A Training Details Below, we provide detailed training hyper-parameters and setups for dense CLIP (weights are used for sparse upcycling), sparse CLIP trained from scratch, a...

    arXiv

  6. [23]

    Table 3 summarizes the hyper-parameters for all experiments, including MoE-specific configurations and parameters for dense CLIP, sparse CLIP, and CLIP-UP

    training datasets. Table 3 summarizes the hyper-parameters for all experiments, including MoE-specific configurations and parameters for dense CLIP, sparse CLIP, and CLIP-UP. Table 3: Training hyper-parameters and settings for dense CLIP used for sparse upcycling and CLIP-UP G...

    2023

  7. [24]

    We also explore the effect of adding MoE layers to only one modality while keeping the other modality fully dense

    ROUTER TYPE TOP-2 ROUTING EXPERT CAPACITY FACTOR(C) 2.0 ( BOTH TEXT AND IMAGE) MOE POSITION [DENSE, SPARSE] (HALF OFMLPLAYERS REPLACED BYMOE LAYERS) LOAD BALANCE LOSS WEIGHT 0.01 ROUTER-Z LOSS WEIGHT 0.0001 DENSEMODEL STEPS 439087 (i.e.,∼ 14B EXAMPLES SEEN) PEAK LEARNING RATE(...

    2019

  8. [2009]

    In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255

    Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat...

    2009

  9. [2014]

    In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755

    Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee

    2014

  10. [2017]

    Preprint, arXiv:1701.06538

    Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. Wentao Wu, Aleksei Timofeev, Chen Chen, Bowen Zhang, Kun Duan, Shuangning Liu, Yantao Zheng, Jonathon Shlens, Xianzhi Du, Zhe Gan, and Yin- fei Yang

    arXiv

  11. [2019]

    Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt

    Do imagenet classifiers generalize to imagenet? Preprint, arXiv:1902.10811. Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt

    1902 arXiv

  12. [2020]

    Preprint, arXiv:2002.04745

    On layer normalization in the transformer architecture. Preprint, arXiv:2002.04745. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zang- wei Zheng, Wangchunshu Zhou, and Yang You

    2002 arXiv

  13. [2021]

    Preprint, arXiv:2103.00020

    Learn- ing transferable visual models from natural language supervision. Preprint, arXiv:2103.00020. 5 Preprint Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu

    arXiv

  14. [2022]

    Preprint, arXiv:2112.06905

    Glam: Efficient scaling of language models with mixture-of- experts. Preprint, arXiv:2112.06905. Alex Fang, Albin Madappally Jose, Amit Jain, Lud- wig Schmidt, Alexander Toshev, and Vaishaal Shankar

    arXiv

  15. [2023]

    In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR)

    Reproducible scaling laws for con- trastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). IEEE. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei

    2023

  16. [2024]

    Preprint, arXiv:2306.07952

    Mofi: Learning image representa- tions from noisy entity annotated images. Preprint, arXiv:2306.07952. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.