Pith. sign in

REVIEW 4 major objections 5 minor 73 references

UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UNIFORM claims one student model, trained without labels, can learn recognition from the collective consensus of over one hundred off-the-shelf pretrained models by voting away teacher disagreement.

desk verdict UNIFORM is a sensible multi-teacher distillation method, but the evaluation protocol is ambiguous about whether test images enter training, which must be resolved before the numbers can be trusted. read the letter →

arxiv 2508.19498 v1 pith:RYYHFDX4 submitted 2025-08-27 cs.CV cs.LG

Yimu Wang , Weiming Zhuang , Chen Chen , Jiabo Huang , Jingtao Li , Lingjuan Lyu This is my paper · ORCID
classification cs.CVcs.LG
keywords knowledgedistillationmulti-teacherlearningunsupervisedobjectrecognitionfeaturevotinglogitmodelzoopseudo-labelingheterogeneousarchitectures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UNIFORM claims that a single student network can learn object recognition without manual labels by distilling the collective opinion of a large, heterogeneous crowd of off-the-shelf pretrained models. It divides those teachers into predictive teachers, which can score some of the target classes, and descriptive teachers, which only supply visual features. Instead of averaging teacher outputs directly, it keeps only opinions that survive two voting procedures: an element-wise sign vote over mapped features and a pseudo-class vote over logits. Across 11 benchmark datasets with 104 teachers, the paper reports the highest average accuracy among the knowledge-transfer baselines it compares against, and the margin grows as teachers are added while baselines saturate. If true, this means freely available model collections can act as a substitute for labeled data in building a compact recognizer.

What carries the argument

Two voting mechanisms carry the method. Feature voting is coordinate-wise: after each teacher's features are mapped into a shared latent space by a learned encoder, the sign of each dimension is tallied across teachers, and only features whose signs match the majority are averaged into the target the student must imitate; reconstruction decoders keep these mapped features from collapsing. Logit voting is class-wise: each predictive teacher contributes one hard vote, the class with the most votes becomes the pseudo-label, and the student's distillation loss is decomposed into pseudo and non-pseudo logit terms so the voted class is emphasized over the distracting tails of teacher distributions.

What would settle it

Re-run the 11-dataset experiment once with the official test images included only for evaluation and never used to compute teacher features or student training targets; if average accuracy drops materially, the headline numbers depend on transductive access to the test set.

Watch

Extended reading notes

Core claim

On its own terms, UNIFORM establishes that teacher disagreement, not teacher diversity, is the obstacle to large-scale knowledge distillation. Naive averaging of teacher features can cancel almost all information when sign patterns conflict, and naive averaging of teacher logits mixes contradictory predictions; UNIFORM's feature voting filters each coordinate to the majority sign before aggregation, and its logit voting first selects a pseudo-class by hard votes, then transfers teacher logits with the pseudo class weighted more heavily. With these mechanisms, 104 public teachers of varied architectures and training data give the student enough supervision to beat strong baselines on all three combinations studied (2, 5, and 11 datasets) and to keep improving as descriptive teachers are added beyond the point where the CFL+ baseline stops improving. The paper reports that the Swin Transformer student reaches 91.89% average dataset accuracy on the 5-dataset setting and that UNIFORM sometimes exceeds the supervised predictive teachers.

Load-bearing premise

The load-bearing premise is that the reported accuracies measure true recognition on held-out data, but the protocol says all samples in each combined dataset are used for training without clarifying whether the official test images appear there as unlabeled inputs.

Editorial extensions

If this is right

  • A student trained without labels from 104 teachers can match or exceed the supervised teachers on several of the benchmark datasets, so public model zoos can plausibly serve as label-free supervision at scale.
  • UNIFORM keeps scaling with teacher count, at least up to 60 descriptive teachers, whereas the CFL+ baseline saturates near 30; the voting mechanisms are what absorb the extra teachers without being overwhelmed by disagreement.
  • The framework makes no assumption about teacher architecture or label space, so predictive teachers with only partial class coverage can be combined with purely descriptive teachers under one loss.
  • Only the student model survives deployment; the teachers and their encoders and decoders are discarded after training, converting a large model collection into one compact classifier.
  • Stronger student architectures, such as Swin Transformer and ConvNeXt-base, translate teacher consensus into higher accuracy, so the student architecture remains a free design choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation protocol says all samples in the combined datasets are used during training but does not state whether the official test splits enter the unlabeled pool; if they do, the reported accuracies are transductive and would not be comparable to the supervised test numbers. Re-running with held-out images removed from the teacher-feature pool would settle this.
  • Coordinate-wise sign voting is essentially a majority operation, so it may discard information that only a minority of teachers carry; if the target domain is far from most teachers' training data, the student may inherit the majority's blind spots even when a few specialized teachers saw the relevant distribution.
  • The same teacher-consensus interface could be applied to dense prediction tasks such as segmentation or detection, but the paper's experiments are image classification only; whether spatial feature voting transfers to pixel-level labels is untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents UNIFORM, a knowledge-distillation framework for training a single student model from a large, heterogeneous set of pre-trained teachers without using manual labels. Teachers are split into predictive teachers that output class predictions on the target label space and descriptive teachers that only provide feature representations. The method maps teacher features into a shared latent space, uses a sign-based voting rule to filter conflicting dimensions, and aggregates the surviving entries; on the logit side, it derives a pseudo-label by hard voting and then applies a decoupled KD loss that weights the pseudo-class and the remaining classes separately. Experiments are conducted on 11 benchmark datasets, combining up to 104 public teachers, and report consistent improvements over adapted baselines (KD, CFL, OFA, CFL+).

Significance. If the reported numbers are taken at face value, the paper makes a useful empirical contribution: it demonstrates that a single student can distill knowledge from a model zoo containing over 100 models with different architectures and training data, using two relatively simple voting mechanisms, and that this approach scales better with the number of teachers than a strong CFL+ baseline. The method makes only mild assumptions about teacher architectures and label spaces, and the ablation study isolating the two voting modules supports their individual contributions. The paper also carefully documents the teacher pool in the appendix, which is useful for reproducibility. The main qualification is that the evaluation protocol is ambiguous about whether test images enter the training set; the headline claims of 'outperforming teachers' and 'unsupervised object recognition' depend on resolving this ambiguity.

major comments (4)
  1. [Section 4, Evaluation Protocol] The sentence 'All the samples in the datasets selected for combination will be used for training' is unqualified. If the official test splits of the 11 benchmark datasets are included in the unlabeled training set, then the reported accuracies are transductive fits to the evaluation set. In that case, comparisons with the 'Predictive Teacher' rows (which are inductive supervised accuracies) are invalid, and the statement that UNIFORM 'sometimes even outperform[s] the teachers' (Section 1) is an artifact of the protocol. This issue affects every table in Section 4 and the scaling study in Figure 5, because all methods would be measured on images they have already seen during training. The manuscript needs to state explicitly which images are used for training and which for evaluation, and if the test splits were used, the experiments must be redone with a proper inductive split or explicitly presented as transductive with appropriate transductive baselines.
  2. [Section 3.1.1, Eqs. (3)-(4) and Section 4, Implementation details] The feature encoders are described as consisting of four 2D convolutional layers with ReLU activation, which forces all mapped teacher features to be non-negative. Consequently, the sign function in Eq. (3) can only take values in {0, +1}, so the 'sign conflicts' between positive and negative elements that motivate the voting mechanism cannot occur in the mapped space as described. Either the encoder output can be negative (e.g., a linear layer before the sign function), or the mechanism actually implements a sparsity filter that discards zero activations; the current text and Figure 2 do not match the implementation. This needs to be clarified or corrected, since it affects the interpretation of the central feature-voting contribution.
  3. [Section 4.1, Tables 1-3] The reported accuracies are single-run numbers with no standard deviations or significance tests. Some per-dataset differences between UNIFORM and CFL+ are small or negative (e.g., in Table 3, Flowers102: 96.83 vs 96.96; Cifar100: 88.54 vs 89.00), and the 'superior performance' claim rests on averaged gains. The authors should provide variance across at least three seeds and, where possible, a paired significance test over the 11 datasets, especially if the protocol is inductive.
  4. [Section 4.3 and Figure 5] The scaling experiment varies only the number of descriptive teachers (up to 60) while keeping the 20 predictive teachers fixed. The conclusion that 'existing methods saturate at a much smaller scale' is drawn from a single baseline (CFL+) on one dataset configuration. The claim about benefiting from 'over one hundred teachers' in the abstract is not directly tested because no experiment compares 40 vs. 100 teachers in the same setup; the 104-teacher result appears only in Table 3. The scaling claim would be stronger if the experiment included the full 104-teacher pool and at least one more baseline.
minor comments (5)
  1. [Eq. (6)] The term H(\hat{p}^t) is written as a loss term but is constant with respect to the student parameters; consider removing it or explicitly stating that it does not affect gradients.
  2. [Eq. (4)] If for some dimension all teachers have sign equal to zero or the denominator sum is zero, the aggregation in Eq. (4) would divide by zero; the paper should state how such dimensions are handled.
  3. [Appendix 6.2, CFL+ description] The text contains a duplicated fragment: 'adopts the same encoder and decoder structure (fe(·) and fd(·)) as UNIFORM. and fd(·)) as UNIFORM.' This should be corrected.
  4. [Table 9] In the last row, 'SwinTransfromer-Base' is misspelled and should be 'SwinTransformer-Base'.
  5. [General] No code is released; providing the teacher-model list and code would substantially aid reproducibility, especially given the large number of external models used.

Circularity Check

1 steps flagged · score 6.0 of 10

The central empirical claim reduces to a transductive fit: the protocol trains on 'all samples' of each dataset and then reports accuracy on the same datasets, with no held-out test split stated.

  1. fitted input called prediction [Section 4, Evaluation Protocol; Tables 1-3; Figure 5]
    "All the samples in the datasets selected for combination will be used for training. ... We report the accuracy of each dataset unless otherwise specified. ... Predictive teachers are individually trained and tested on each dataset."

    Under a literal reading, the student is trained on every image of each combined dataset, including the images whose classes are later reported as accuracies; the paper never states that official test splits are withheld. The 'Predictive Teacher' rows are standard inductive accuracies from teachers 'trained and tested on each dataset,' so the comparison in Tables 1-3 is transductive-student versus inductive-teacher. The reported 'accuracy of each dataset' is therefore a fitted value on the training set, and the scaling curves in Figure 5 measure how well more teachers help fit the evaluation images rather than how well knowledge generalizes.

full rationale

UNIFORM's training objective (Eq. 7) is a standard distillation loss: the student is supervised by teacher features and teacher-voted logits, and the final evaluation uses ground-truth class labels. There is no equation-level circularity: the pseudo-labels in Eq. (5) are an input derived from the teachers, and the losses in Eqs. (2), (4), and (6) are alignment objectives, not definitions of the reported accuracy. No load-bearing self-citation or imported uniqueness theorem appears; references to CFL, OFA, DKD, and AM-Radio are external prior work. The single circular step is in the evaluation protocol. Section 4 says 'All the samples in the datasets selected for combination will be used for training' and then reports 'the accuracy of each dataset' without ever stating that official test splits are withheld. Under the literal protocol, every evaluation image is in the student's unlabeled training set, so Tables 1-3 and Figure 5 report transductive fits to the evaluation set, not inductive predictions. The captions' 'Predictive Teacher' numbers come from teachers 'trained and tested on each dataset' on standard splits, so the comparison mixes transductive student numbers with inductive teacher numbers. This makes the central empirical claim—that UNIFORM 'benefits from over one hundred teachers' and 'sometimes even outperform[s] the teachers'—reduce, by construction, to fitting the evaluation data. If the authors intended 'all samples' to mean only the official training splits, that intent is not stated, and the footnote about 'training data' is too terse to cure the ambiguity. The derivation chain itself is otherwise self-contained.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several heuristic assumptions about teacher consensus and on a set of unreported hyperparameters. No entirely new entity or fundamental constant is introduced. The most consequential item is the ambiguity about whether test data is used during training, which is listed as an axiom.

free parameters (6)
  • alpha1 (logit pseudo-class weight)
    Hyperparameter in Eq. (6) weighting the pseudo-class term; no value or sensitivity analysis is reported.
  • alpha2 (logit non-pseudo class weight)
    Hyperparameter in Eq. (6) weighting non-pseudo classes; no value reported.
  • beta1 (feature loss weight)
    Hyperparameter in Eq. (7) weighting the feature loss; no value reported.
  • beta2 (reconstruction loss weight)
    Hyperparameter in Eq. (7) weighting the reconstruction loss; no value reported.
  • D (unified feature dimension)
    Dimension of the common latent space in Section 3.1.1; the paper does not state the chosen value.
  • Descriptive teacher pool selection = 50 top downloaded HuggingFace models plus 10 expert-selected models
    The composition of the teacher zoo is a design choice that affects results, not a fitted scalar.
assumptions (4)
  • domain assumption Teacher consensus is a reliable proxy for ground truth.
    The framework assumes majority directions across teacher features and logits filter noise and preserve signal; no formal justification is given (Sections 1, 3.1).
  • ad hoc to paper Sign conflicts in mapped teacher features indicate unreliable dimensions.
    The voting in Eq. (3) treats sign-disagreement as noise to be removed; this is a heuristic design choice specific to this paper.
  • domain assumption Teacher logits over disjoint label spaces can be merged by zero-padding and class-wise averaging.
    Section 4's evaluation protocol assumes that predictions on a subset of classes can be compared and averaged in a unified label space.
  • domain assumption The unlabeled training data is a valid representation of the target task and does not include evaluation data.
    The paper does not disambiguate whether the full dataset, including test splits, is used for training; the statement 'All the samples in the datasets selected for combination will be used for training' creates this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models." pith.science (2026). https://pith.science/paper/RYYHFDX4

@misc{pith2026250819498,
  author       = {Pith},
  title        = {Pith review of: UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYYHFDX4}},
  note         = {Machine review of arXiv:2508.19498}
}
read the original abstract

In the era of deep learning, the increasing number of pre-trained models available online presents a wealth of knowledge. These models, developed with diverse architectures and trained on varied datasets for different tasks, provide unique interpretations of the real world. Their collective consensus is likely universal and generalizable to unseen data. However, effectively harnessing this collective knowledge poses a fundamental challenge due to the heterogeneity of pre-trained models. Existing knowledge integration solutions typically rely on strong assumptions about training data distributions and network architectures, limiting them to learning only from specific types of models and resulting in data and/or inductive biases. In this work, we introduce a novel framework, namely UNIFORM, for knowledge transfer from a diverse set of off-the-shelf models into one student model without such constraints. Specifically, we propose a dedicated voting mechanism to capture the consensus of knowledge both at the logit level -- incorporating teacher models that are capable of predicting target classes of interest -- and at the feature level, utilizing visual representations learned on arbitrary label spaces. Extensive experiments demonstrate that UNIFORM effectively enhances unsupervised object recognition performance compared to strong knowledge transfer baselines. Notably, it exhibits remarkable scalability by benefiting from over one hundred teachers, while existing methods saturate at a much smaller scale.

Figures

Figures reproduced from arXiv: 2508.19498 by the authors.

Figure 1
Figure 1. (a) The number of publicly available models increase [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. ) and logits (Section 3.1.2 and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Logit distribution conflicts (top). As teachers usually produce inconsistent prediction distributions, it might confuse the student when directly forcing the student to mimic the behavior of all the teachers. Logits voting and transfer (bottom). Instead of averaging on the teachers’ logits, UNIFORM highlights transfer on the pseudo-class voted by teachers to avoid confusion. UNIFORM first gets the pseudo labels for … view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Performance when scaling up the number of public [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance when scaling up the number of public de [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 53 canonical work pages

  1. [1]

    Papers with Code - The latest in Machine Learning. 1, 3

    work page

  2. [2]

    Evolutionary Optimization of Model Merging Recipes,

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary Optimization of Model Merging Recipes,

    work page

  3. [3]

    Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

    Zeyuan Allen-Zhu and Yuanzhi Li. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. 2022. 4

    work page 2022

  4. [4]

    Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization. 2022. 2

    work page 2022

  5. [5]

    On the Inductive Bias of Neural Tangent Kernels

    Alberto Bietti and Julien Mairal. On the Inductive Bias of Neural Tangent Kernels. In Advances in Neural Information Processing Systems, pages 12873–12884, Vancouver, BC, Canada, 2019. 4

    work page 2019

  6. [6]

    Food-101 – Mining Discriminative Components with Random Forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – Mining Discriminative Components with Random Forests. In European Conference on Computer Vision, pages 446–461, 2014. 5

    work page 2014

  7. [7]

    SWAD: Domain Generalization by Seeking Flat Minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain Generalization by Seeking Flat Minima. In Advances in Neural Information Processing Systems , pages 22405– 22418. Curran Associates, Inc., 2021. 2

    work page 2021

  8. [8]

    Distilling Knowledge via Knowledge Review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling Knowledge via Knowledge Review. In 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5006–5015, Nashville, TN, USA, 2021. IEEE. 2

    work page 2021

Show all 73 references
  1. [9]

    Fusing finetuned models for better pretraining, 2022

    Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022. arXiv:2204.03044 [cs]. 2

    2022 arXiv

  2. [10]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and and A. Vedaldi. Describing Textures in the Wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014. 5

    2014

  3. [12]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition ...

    2021

  4. [13]

    Agree to Disagree: Adap- tive Ensemble Knowledge Distillation in Gradient Space

    Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang. Agree to Disagree: Adap- tive Ensemble Knowledge Distillation in Gradient Space. In Advances in Neural Information Processing Systems, pages 12345–12355. Curran Associates, Inc., 2020. 1, 3

    2020

  5. [14]

    Learning Factored Representations in a Deep Mixture of Ex- perts

    David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Ex- perts. 2013. 2

    2013

  6. [15]

    ImageBind: One Embedding Space To Bind Them All, 2023

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One Embedding Space To Bind Them All, 2023. arXiv:2305.05665 [cs]. 2

    2023 arXiv

  7. [16]

    Borgwardt, Malte J

    Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bern- hard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25):723–773,

  8. [17]

    STOCHASTIC WEIGHT A VERAGING IN PARAL- LEL: LARGE-BATCH TRAINING THAT GENERALIZES WELL

    Vipul Gupta, Santiago Akle Serrano, and Dennis De- Coste. STOCHASTIC WEIGHT A VERAGING IN PARAL- LEL: LARGE-BATCH TRAINING THAT GENERALIZES WELL. 2020. 2

    2020

  9. [18]

    Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

    Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

  10. [19]

    One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distil- lation

    Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distil- lation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 5, 6, 1

    2023

  11. [20]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. InIEEE Con- ference on Computer Vision and Pattern Recognition, pages 770–778, Las Vegas, NV , 2016. IEEE Computer Society. 1, 2, 4, 5, 7

    2016

  12. [21]

    A Comprehensive Overhaul of Feature Distillation

    Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, No- jun Kwak, and Jin Young Choi. A Comprehensive Overhaul of Feature Distillation. 1, 3

  13. [22]

    Distilling the Knowledge in a Neural Network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015. arXiv:1503.02531 [cs, stat]. 1, 2, 5, 6

    2015 arXiv

  14. [23]

    Edit- ing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Edit- ing models with task arithmetic. 2022. 2

    2022

  15. [24]

    Averaging Weights Leads to Wider Optima and Better Generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. 2

  16. [25]

    Dataless Knowledge Fusion by Merging Weights of Language Models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless Knowledge Fusion by Merging Weights of Language Models. 2022. 2

    2022

  17. [26]

    Khosla, N

    A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine-grained image categorization. InIEEE Confer- ence on Computer Vision and Pattern Recognition Workshop, pages 1–2, 2011. 5

    2011

  18. [27]

    3D object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. InIEEE International Conference on Computer Vision Workshops , pages 554–561. IEEE Computer Society, 2013. 5

    2013

  19. [28]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer,

  20. [29]

    Caltech 101, 2022

    Fei-Fei Li, Marco Andreeto, Marc’Aurelio Ranzato, and Pietro Perona. Caltech 101, 2022. 5

    2022

  21. [30]

    Smith, and Luke Zettlemoyer

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch- Train-Merge: Embarrassingly Parallel Training of Expert Language Models, 2022. arXiv:2208.03306 [cs]. 2

    2022 arXiv

  22. [31]

    Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. 2023. 2

    2023

  23. [32]

    Harmonious atten- tion network for person re-identification

    Wei Li, Xiatian Zhu, and Shaogang Gong. Harmonious atten- tion network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2285–2294, 2018. 2

    2018

  24. [33]

    Implicit Bias of Gradient Descent based Adversarial Training on Separable Data

    Yan Li, Ethan X.Fang, Huan Xu, and Tuo Zhao. Implicit Bias of Gradient Descent based Adversarial Training on Separable Data. 2019. 1

    2019

  25. [34]

    MoE-LLaV A: Mixture of Experts for Large Vision-Language Models, 2024

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Munan Ning, and Li Yuan. MoE-LLaV A: Mixture of Experts for Large Vision-Language Models, 2024. arXiv:2401.15947 [cs]. 1, 2

    2024 arXiv

  26. [35]

    Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 5, 7

    2021

  27. [36]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11966–11976. IEEE, 2022. 5, 7

    2022

  28. [37]

    Knowledge Amalgamation from Het- erogeneous Networks by Common Feature Learning

    Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge Amalgamation from Het- erogeneous Networks by Common Feature Learning. In Pro- ceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 3087–3093, Macao, China,

  29. [38]

    Fine-Grained Visual Classifi- cation of Aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-Grained Visual Classifi- cation of Aircraft. Technical report, 2013. _eprint: 1306.5151. 5

    2013 arXiv

  30. [39]

    Im- proved Knowledge Distillation via Teacher Assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Im- proved Knowledge Distillation via Teacher Assistant. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 5191–5198, 2020. Number: 04. 2

    2020

  31. [40]

    5, 6, 7, 1

    International Joint Conferences on Artificial Intelli- gence Organization. 5, 6, 7, 1

  32. [41]

    Parkhi, Andrea Vedaldi, Andrew Zisserman, and C

    Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V . Jawahar. Cats and Dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 5

    2012

  33. [42]

    Correlation Congruence for Knowledge Distillation, 2019

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation Congruence for Knowledge Distillation, 2019. arXiv:1904.01802 [cs]. 1, 3

    2019 arXiv

  34. [43]

    Automated Flower Classification over a Large Number of Classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Number of Classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 5

    2008

  35. [44]

    Diverse Weight Averaging for Out-of-Distribution General- ization

    Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse Weight Averaging for Out-of-Distribution General- ization. 2022. 2

    2022

  36. [45]

    Model Ratatouille: Recy- cling Diverse Models for Out-of-Distribution Generalization

    Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, and David Lopez-Paz. Model Ratatouille: Recy- cling Diverse Models for Out-of-Distribution Generalization. In Proceedings of the 40th International Conference on Ma- chine Learning, pages 28656–28679. PMLR...

    2023

  37. [46]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of th...

    2021

  38. [47]

    FitNets: Hints for Thin Deep Nets, 2015

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets, 2015. arXiv:1412.6550 [cs]. 1, 2, 3

    2015 arXiv

  39. [48]

    Unic: Universal classi- fication models via multi-teacher distillation

    Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Unic: Universal classi- fication models via multi-teacher distillation. arXiv preprint arXiv:2408.05088, 2024. 2, 3

    2024 arXiv

  40. [49]

    Am-radio: Agglomerative vision foundation model reduce all domains into one

    Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024. 2, 3, 7, 8

    2024

  41. [50]

    Scaling Vision-Language Models with Sparse Mixture of Experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling Vision-Language Models with Sparse Mixture of Experts. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023 , pages 11329–11344, Singapore, 2023. Association for Co...

    2023

  42. [51]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. 4

    2014 arXiv

  43. [52]

    RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval, 2022

    Burak Satar, Hongyuan Zhu, Hanwang Zhang, and Joo Hwee Lim. RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval, 2022. arXiv:2206.12845 [cs] version:

    2022 arXiv

  44. [53]

    Contrastive Representation Distillation, 2022

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Representation Distillation, 2022. arXiv:1910.10699 [cs, stat]. 1, 3

    2022 arXiv

  45. [54]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkor- eit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neu- ral Information Processing Systems, pages 5998–6008, Long Beach, CA, 2017. 4

    2017

  46. [55]

    An Empirical Study of Multimodal Model Merging

    Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An Empirical Study of Multimodal Model Merging. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 1563–1575, Singapore,

    2023

  47. [56]

    The Implicit Bias for Adaptive Optimization Algorithms on Ho- mogeneous Neural Networks

    Bohan Wang, Qi Meng, Wei Chen, and Tie-Yan Liu. The Implicit Bias for Adaptive Optimization Algorithms on Ho- mogeneous Neural Networks. In International Conference on Machine Learning, pages 10849–10858. PMLR, 2021. ISSN: 2640-3498. 1

    2021

  48. [57]

    Momentum Doesn’t Change The Implicit Bias

    Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, and Zhi-Ming Ma. Momentum Doesn’t Change The Implicit Bias. 2022. 1

    2022

  49. [58]

    SAM-CLIP: Merging Vision Foundation Mod- els towards Semantic and Spatial Understanding, 2023

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. SAM-CLIP: Merging Vision Foundation Mod- els towards Semantic and Spatial Understanding, 2023. arXiv:2310.15...

    2023 arXiv

  50. [59]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 5

    2011

  51. [60]

    SegGPT: Segmenting Every- thing In Context, 2023

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chun- hua Shen, and Tiejun Huang. SegGPT: Segmenting Every- thing In Context, 2023. arXiv:2304.03284 [cs]. 4

    2023 arXiv

  52. [61]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Ma...

    2020

  53. [62]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without...

    2022

  54. [63]

    Deep Visual Domain Adapta- tion: A Survey

    Mei Wang and Weihong Deng. Deep Visual Domain Adapta- tion: A Survey. Neurocomputing, 312:135–153, 2018. arXiv: 1802.03601. 2

    2018 arXiv

  55. [64]

    Exclusive Supermask Sub- network Training for Continual Learning

    Prateek Yadav and Mohit Bansal. Exclusive Supermask Sub- network Training for Continual Learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 569–587, Toronto, Canada, 2023. Association for Computa- tional Linguistics. 2

    2023

  56. [65]

    TIES-Merging: Resolving Interference When Merging Models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-Merging: Resolving Interference When Merging Models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1

    2023

  57. [66]

    Wide Residual Networks

    Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In British Machine Vision Conference, pages 87.1– 87.12. BMV A Press, 2016. 4

    2016

  58. [67]

    Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts, 2024

    Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts, 2024. arXiv:2312.00968 [cs]. 1, 2

    2024 arXiv

  59. [68]

    Hospedales, and Huchuan Lu

    Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep Mutual Learning. In2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4320–4328, Salt Lake City, UT, 2018. IEEE. 2

    2018

  60. [69]

    Decoupled Knowledge Distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jia- jun Liang. Decoupled Knowledge Distillation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11943–11952, New Orleans, LA, USA, 2022. IEEE. 1, 4 UNIFORM : Unifying Knowledge from Large-s...

    2022

  61. [71]

    Task-Oriented Feature Distillation

    Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma, and Chenglong Bao. Task-Oriented Feature Distillation. In Advances in Neural Information Processing Systems, pages 14759–14771. Curran Associates, Inc., 2020. 1, 2

    2020

  62. [74]

    Experiments 6.1. How Well Does UNIFORM Scale - Number of Descriptive Teachers We present the visualization of the performance ofUNIFORM when scaling up the number of public descriptive teachers in Figure 6. The results are consistent with the findings in the main paper, demons...

  63. [2019]

    arXiv: 1810.04805. 1, 2

    arXiv

  64. [2023]

    Association for Computational Linguistics. 2

  65. [2024]

    arXiv:2403.13187 [cs]. 1

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.