Pith. sign in

REVIEW 3 major objections 6 minor 60 references

RanDeS: Randomized Delta Superposition for Multi-Model Compression

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One compressed weight stack serves 20 fine-tuned models near full accuracy.

desk verdict RanDeS is a practical multi-model compression method with strong empirical results, but the paper's stated l2-norm interference mechanism is wrong and the appendix half-admits it. read the letter →

arxiv 2505.11204 v1 pith:6DDGKMYO submitted 2025-05-16 cs.LG cs.AI

Hangyu Zhou , Aaron Gokaslan , Volodymyr Kuleshov , Bharath Hariharan This is my paper · ORCID
classification cs.LGcs.AI
keywords modelmergingdeltasuperpositionmulti-modelservingrandomorthogonaltransformationslayershufflingcolumn-wisesignflipstaskinterferencecompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RanDeS tackles the memory cost of serving many models that share a pretrained base. Its claim is that the interference that ruins naive model merging can be suppressed by randomly transforming each model's parameter delta before summing them, then undoing that random transform when a specific task is requested. The random transforms—layer permutations and column-wise sign flips—make the interfering deltas nearly orthogonal, so their sum nearly cancels. On 8 CLIP-ViT-B/32 models, RanDeS-SRSF recovers 99.6% of fine-tuned average accuracy while using 1.58x the base model's memory, and on 20 CLIP-ViT-L/14 models it reaches 99.3% at a constant 2.87 GB. If right, this lets a serving system hold any number of fine-tunes in roughly one base model's footprint.

What carries the argument

The load-bearing object is the per-model orthogonal transformation $O_i$ applied to each delta before summation, with retrieval applying $O_i^{-1}$. In full generality it is a rotation drawn from the orthogonal group, but the paper instantiates it cheaply as a permutation of same-shaped layers inside each delta and a random diagonal matrix with $\pm 1$ entries applied column-wise per layer. What it does is convert the interference term $\lambda\sum_{j\neq i} O_i^{-1}O_j\Delta_j$ into a sum of near-orthogonal vectors whose $\ell^2$ norm, by Eq. (3), depends on pairwise cosine similarities; driving those cosines toward zero is what preserves task accuracy.

What would settle it

Take one fine-tuned model, add a second 'task' delta that is orthogonal to the first in cosine similarity but concentrated entirely on the parameters most critical to the first task's output layer, and merge them with RanDeS; if accuracy on the first task drops far more than the near-zero cosine similarity predicts, the norm proxy at the heart of the argument fails.

Watch

Extended reading notes

Core claim

Starting from task arithmetic, the paper rewrites model merging as compress-and-retrieve: the merged weights are the base weights plus a sum of deltas, and retrieving task $i$ should give $(1-\lambda)\Theta_0 + \lambda\Theta_i$ plus an interference term $\lambda\sum_{j\neq i} O_i^{-1}O_j\Delta_j$. The central discovery is that this interference term shrinks dramatically when each delta is first rotated by a random orthogonal transformation $O_i$, because in high dimension random rotations make the interfering deltas nearly orthogonal. The paper implements the rotations as layer shuffling within repetitive layer groups and as random column-wise sign flips per layer, both invertible and fully specified by random seeds. With both operations combined, the average pairwise cosine similarity among interfering deltas drops by orders of magnitude, and measured accuracy on vision, text-generation, and text-classification benchmarks lands near the individually fine-tuned models. The same procedure also compresses LoRA adapters.

Load-bearing premise

The load-bearing premise is that the size of the interference term, measured through pairwise cosine similarity among deltas, is what determines how much task accuracy is lost; if the direction or fine-grained structure of the interference matters more than its size, the decorrelation benefit could disappear.

Editorial extensions

If this is right

  • A serving system can add a new fine-tuned model at effectively zero extra storage, because each transformation is stored as a seed rather than as weights.
  • Holding accuracy near the fine-tuned upper bound no longer requires per-task weight copies: 8 ViT-B/32 models fit in 1.58x base memory and 20 ViT-L/14 models in a constant 2.87 GB.
  • Models can be hot-swapped in or out of the pool without retraining or recomputing a router or task masks.
  • The method transfers to parameter-efficient fine-tuning: LoRA adapters on Flan-T5-base reach 99.8% normalized accuracy at 1.20 GB.
  • Because the transformations are random and data-agnostic, the compression ratio grows with the number of models rather than degrading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same decorrelation mechanism could apply to any setting where multiple learned updates to one base must be stored and selectively applied, such as continual learning checkpoints, federated round updates, or per-client adapters, since nothing in the argument depends on the task modality.
  • The ablation result that random orthogonal sign flips beat random non-orthogonal diagonal matrices suggests that orthogonality, not merely randomness, is the active ingredient; a cheap orthogonal transform such as a Walsh-Hadamard rotation is a natural untested extension.
  • The strong negative correlation between cosine similarity and accuracy in the paper's Figure 7 implies a possible operating rule: manipulate only enough layers to push interfering-delta cosine below a task-specific threshold, trading a little accuracy for less compute and memory.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reformulates multi-model compression as storing a superposition of randomly transformed task deltas and retrieving individual deltas via inverse transforms. It proposes two orthogonal-transformation implementations (layer shuffling and column-wise random sign flips) and their combination. Experiments on CLIP-ViT-B/32 (8 tasks), CLIP-ViT-L/14 (up to 20 tasks), Flan-T5-base (full and LoRA), and GPT-2 show near-fine-tuned accuracy at roughly constant memory, with memory proportional to one base model plus one delta superposition. The theoretical motivation is that random orthogonal transforms decorrelate interfering deltas, reducing the l2 norm of the interference term in Eq. (3).

Significance. The method is simple, training-free, and seed-based, with a public codebase; the empirical coverage is broad (vision, language, PEFT, and a 20-model large-scale setting). If the mechanism were exactly as claimed, this would be a strong contribution to multi-model serving. However, the stated l2-norm mechanism does not quantitatively account for the reported gains, and the paper itself concedes in Appendix C.2 that 'the method of achieving orthogonality matters beyond decorrelation levels.' The results are thus significant but the theoretical framing needs substantial revision.

major comments (3)
  1. [Section 3, Eq. (3); Figure 3; Table 1] The central claim that random orthogonal transformations reduce interference by lowering the l2 norm of the interference term is not supported by the reported numbers. For the T=8 CLIP-ViT-B/32 setup, Figure 3 reports pairwise cosine similarity among interfering deltas of approximately 0.02-0.03 before RanDeS and approximately 0.0001 after RanDeS-SRSF. Assuming comparable delta norms, Eq. (3) gives an interference norm of sqrt(7 + 42*0.025) ||Δ|| ≈ 2.84||Δ|| for task arithmetic and sqrt(7 + 42*0.0001)||Δ|| ≈ 2.65||Δ|| after RanDeS-SRSF, a reduction of about 6-7%. Yet Table 1 shows average accuracy rising from 69.8% (task arithmetic) to 89.9% (RanDeS-SRSF), a ~20-point gain that a 6-7% norm reduction is unlikely to explain. Moreover, orthogonal vectors do not 'self-cancel'; they add in quadrature, so the interference norm grows as sqrt(T-1) even at zero cosine. The gain likely comes from the direction of the randomized interference relative to task-relevant parameter directions, not from its magnitude. Please revise the theoretical justification or provide controlled experiments that isolate the l2-norm component (e.g., rescaling the interference term to match norms across conditions).
  2. [Section 3, sentence after Eq. (3)] The sentence 'the magnitude of interference is negatively correlated with the cosine similarity between deltas' misstates the direction of the relationship. The cross-term in Eq. (3) is +2 Σ cos(Δ_l, Δ_j) ||Δ_l|| ||Δ_j||, so larger cosine similarity increases the interference norm. The correct statement is that the interference magnitude is positively correlated with cosine similarity; the design goal is therefore to reduce positive cosines (or push them negative). This error should be corrected because it appears in the paper's core motivation.
  3. [Appendix C.2; Section 5.3] The paper's own analysis undermines the exclusive role of cosine-similarity reduction. Appendix C.2 reports that for GTSRB, RanDeS-S outperforms RanDeS-Shift despite having higher cosine similarity, 'indicating the method of achieving orthogonality matters beyond decorrelation levels.' Similarly, Figure 7 shows only correlational evidence (albeit with strong negative correlations) between cosine and accuracy. These observations are consistent with the concern in comment 1 that the mechanism is directional rather than norm-based. The paper should either advance a coherent directional theory or explicitly reframe the contribution as empirical, with the cosine-reduction claim demoted to a heuristic.
minor comments (6)
  1. [Eq. (3)] The summation bound uses an undefined symbol n; it should be T (or T-1 with the exclusion of i properly indicated).
  2. [Abstract and Section 6] The term 'self-cancellation' is misleading when applied to orthogonal vectors, which add in quadrature rather than cancel; consider using 'cross-term cancellation' or 'near-orthogonality' instead.
  3. [Section 5.5, Figure 5(a)] The RBD-vs-RD comparison is confounded because RD changes both orthogonality and the norm of each delta (entries are not ±1); please add a control with column-normalized random matrices to attribute the degradation to loss of orthogonality rather than magnitude distortion.
  4. [Tables 1 and 2] The main results are averages over three runs but no standard deviations are reported; please include error bars or state explicitly the actual variance magnitude instead of only noting that variances smaller than 0.1% are omitted.
  5. [Figure 3 caption] The phrase 'three out of eight' is ambiguous; please specify which tasks are shown or state that they are representative.
  6. [Table 1 header] The column label 'Bits(Gb)' is confusing; use 'Memory (Gb)' or 'Size (Gb)' instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RanDeS's core mechanism is a constructive random-orthogonal transform, not a fitted quantity; the only fitted scalar lambda is shared with baselines and not derived from the predicted accuracies.

full rationale

The paper's derivation chain is self-contained. Eq. (3) expands the l2 norm of the interference term in task arithmetic and expresses it via pairwise cosine similarities; this is an algebraic identity, not an assumption that defines the conclusion. RanDeS then constructs random layer-wise permutations and column sign flips (Eqs. 4-11) and evaluates retrieval accuracy on held-out tasks; the random operations are not fitted to the accuracies they predict. The merging coefficient lambda is grid-searched on validation sets, exactly as in the task-arithmetic baseline (Ilharco et al., 2022), so the RanDeS gains are not a renamed fit. Figure 7's negative correlation between cosine similarity and accuracy is correlational, and the appendix's admission that 'the method of achieving orthogonality matters beyond decorrelation levels' weakens the paper's mechanistic story, but a weak or incomplete explanation is not circularity. The only self-citation (Cho and Hariharan 2019 on knowledge distillation) appears in related work and is not load-bearing. No equation or parameter in the paper reduces to the result it is said to predict.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on high-dimensional decorrelation of random orthogonal transforms, on an empirical relationship between interference norm and accuracy, and on the usual validation-set fitting of lambda. No new physical entities or latent variables are introduced.

free parameters (1)
  • merging coefficient lambda = grid-searched over 0.1 to 1.0, extended to 0.1 to 2.0 for LoRA; exact values not reported in text
    Chosen per benchmark via validation set grid search for all methods including baselines; controls the balance between the base model and the deltas.
assumptions (5)
  • standard math Random orthogonal transformations applied to independent high-dimensional vectors produce near-orthogonal images
    Invoked in Section 4 as the basis for decorrelation; standard concentration-of-measure behavior, but no explicit dimension bound is given for the finite models used.
  • domain assumption Task interference effects on accuracy are governed by the l2 norm of the interference term
    Eq (3) and Figure 7 treat the norm of interfering deltas as the quantity to minimize; accuracy is assumed to be monotonically worse as this norm grows, which is only empirically correlated.
  • domain assumption Within-model layer deltas are more variable than across-model deltas at the same layer
    Figure 2 justifies layer shuffling; if false, shuffling would still be invertible for retrieval but would not decorrelate interference.
  • domain assumption Validation-set grid search of lambda transfers to test performance
    Standard in the merging literature following Ilharco et al. 2022, but it is a fitted choice rather than a derived value.
  • domain assumption Layer-wise implementations, permutations within same-type layers and column sign flips, are sufficient substitutes for full O(d) orthogonal transforms
    The paper proposes these as practical implementations in Sections 4.3 and 4.4; their equivalence to the idealized O(d) transform is not formally established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RanDeS: Randomized Delta Superposition for Multi-Model Compression." pith.science (2026). https://pith.science/paper/6DDGKMYO

@misc{pith2026250511204,
  author       = {Pith},
  title        = {Pith review of: RanDeS: Randomized Delta Superposition for Multi-Model Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DDGKMYO}},
  note         = {Machine review of arXiv:2505.11204}
}
read the original abstract

From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving.

Figures

Figures reproduced from arXiv: 2505.11204 by the authors.

Figure 1
Figure 1. Illustration of RanDeS with three deltas {∆i} 3 i=1. Concretely, given T fine-tuned models {Θi ∈ R d} T i=1 and the pre-trained model Θ0 ∈ R d , we first derive the deltas {∆i ∈ R d} T i=1. Then we transform each delta ∆i with an orthogonal transformation Oi sampled from the orthogonal group O(d) before merging back to the shared pre-trained model weights: ΘRanDeS ⋆ ← Θ0 + λ X T i=1 Oi∆i , (4) ∀i. Oi ∼ O(d). (5) whe… view at source ↗
Figure 2
Figure 2. Cosine similarity distributions between delta layers within and across models for CLIP [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average pairwise cosine similarity of three out of eight CLIP-ViT-B/32 task vectors during [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The impact of λ on average accuracy over eight image classification tasks. Here we examine the interplay between the merging coefficient λ and the average performance across different setup. For each variant, we perform a grid search on λ = {0.1, 0.2, · · · , 1.0} when…
Figure 5
Figure 5. Figure 5: (a) Impact of context matrix design to the average accuracy. RBD stands for random binary [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Average accuracy and cosine similarity among interfering deltas when retrieving SUN397 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Correlation between the pairwise cosine similarity among interfering deltas and the accuracy [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 18 canonical work pages

  1. [1]

    Structured pruning of deep convolutional neural networks

    Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13 0 (3): 0 1--18, 2017

    2017

  2. [2]

    Food-101--mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101--mining discriminative components with random forests. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446--461. Springer, 2014

    2014

  3. [3]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105 0 (10): 0 1865--1883, 2017

    2017

  4. [4]

    Superposition of many models into one

    Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. Advances in neural information processing systems, 32, 2019

    2019

  5. [5]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794--4802, 2019

    work page 2019

  6. [6]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition ( CVPR ) , 2014

    2014

  7. [7]

    Deep learning for classical japanese literature

    Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018

    arXiv 2018

  8. [8]

    An analysis of single-layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215--223. JMLR Workshop and Conference Proceedings, 2011

    work page 2011

Show all 60 references
  1. [9]

    Emnist: Extending mnist to handwritten letters

    Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921--2926. IEEE, 2017

    2017

  2. [10]

    The mnist database of handwritten digit images for machine learning research

    Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29 0 (6): 0 141--142, 2012

    2012

  3. [11]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091--16101, 2023

    2023

  4. [12]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259--3269. PMLR, 2020

    2020

  5. [13]

    A survey of quantization methods for efficient neural network inference

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291--326. Chapman and Hall/CRC, 2022

    2022

  6. [14]

    Challenges in representation learning: A report on three machine learning contests

    Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In Neural information processing...

    2013

  7. [15]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129 0 (6): 0 1789--1819, 2021

    2021

  8. [16]

    Pela: Learning parameter-efficient models with low-rank approximation

    Yangyang Guo, Guangzhi Wang, and Mohan Kankanhalli. Pela: Learning parameter-efficient models with low-rank approximation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15699--15709, 2024

    2024

  9. [17]

    Structured pruning for deep convolutional neural networks: A survey

    Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2023

    2023

  10. [18]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12 0 (7): 0 2217--2226, 2019

    2019

  11. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [20]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  13. [21]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849, 2022

    2022 arXiv

  14. [22]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554--561, 2013

    2013

  15. [23]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https://api.semanticscholar.org/CorpusID:18268744

    2009

  16. [24]

    Structured compression by weight encryption for unstructured pruning and quantization

    Se Jung Kwon, Dongsoo Lee, Byeongwook Kim, Parichay Kapoor, Baeseong Park, and Gu-Yeon Wei. Structured compression by weight encryption for unstructured pruning and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1909--...

    1909

  17. [25]

    Scalable learned model soup on a single gpu: An efficient subspace training strategy

    Tao Li, Weisen Jiang, Fanghui Liu, Xiaolin Huang, and James T Kwok. Scalable learned model soup on a single gpu: An efficient subspace training strategy. arXiv preprint arXiv:2407.03641, 2024

    2024 arXiv

  18. [26]

    Losparse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336--20350. PMLR, 2023

    2023

  19. [27]

    Can unstructured pruning reduce the depth in deep neural networks? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1402--1406, 2023

    Zhu Liao, Victor Qu \'e tu, Van-Tam Nguyen, and Enzo Tartaglione. Can unstructured pruning reduce the depth in deep neural networks? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1402--1406, 2023

    2023

  20. [28]

    Bitdelta: Your fine-tune may only be worth one bit

    James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. arXiv preprint arXiv:2402.10193, 2024

    2024 arXiv

  21. [29]

    Post-training quantization for vision transformer

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34: 0 28092--28103, 2021

    2021

  22. [30]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631--22648. P...

    2023

  23. [31]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 0 17703--17716, 2022

    2022

  24. [32]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011

    2011

  25. [33]

    What is being transferred in transfer learning? Advances in neural information processing systems, 33: 0 512--523, 2020

    Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33: 0 512--523, 2020

    2020

  26. [34]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722--729. IEEE, 2008

    2008

  27. [35]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [36]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967--3976, 2019

    2019

  29. [37]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498--3505. IEEE, 2012

    2012

  30. [38]

    Delta-come: Training-free delta-compression with mixed-precision for large language models

    Bowen Ping, Shuo Wang, Hanqing Wang, Xu Han, Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, and Maosong Sun. Delta-come: Training-free delta-compression with mixed-precision for large language models. arXiv preprint arXiv:2406.08903, 2024

    2024 arXiv

  31. [39]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  32. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  33. [41]

    Efficient storage of fine-tuned models via low-rank approximation of weight residuals

    Simo Ryu, Seunghyun Seo, and Jaejun Yoo. Efficient storage of fine-tuned models via low-rank approximation of weight residuals. arXiv preprint arXiv:2305.18425, 2023

    2023 arXiv

  34. [42]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language proce...

    2013

  35. [43]

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32: 0 323--332, 2012

    2012

  36. [44]

    Fusionbench: A comprehensive benchmark of deep model fusion

    Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Do, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint arXiv:2406.03280, 2024 a

    2024

  37. [45]

    Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models

    Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, and Dacheng Tao. Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models. arXiv preprint arXiv:2408.10174, 2024 b

    2024 arXiv

  38. [46]

    Merging multi-task models via weight-ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. arXiv preprint arXiv:2402.00433, 2024 c

    2024 arXiv

  39. [47]

    Rotation equivariant cnns for digital pathology

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention--MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proc...

    2018

  40. [48]

    Machine learning model sizes and the parameter gap

    Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, and Marius Hobbhahn. Machine learning model sizes and the parameter gap. arXiv preprint arXiv:2207.02852, 2022

    2022 arXiv

  41. [49]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  42. [50]

    Localizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Fran c ois Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813, 2024

    2024 arXiv

  43. [51]

    Structured pruning of large language models

    Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.04732, 2019

    1910 arXiv

  44. [52]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  45. [53]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  46. [54]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485--3492, 2010. doi:10.1109/CVPR.2010.5539970

    2010

  47. [55]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  48. [56]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023

    2023 arXiv

  49. [57]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024

    2024

  50. [58]

    On compressing deep models by low rank and sparse decomposition

    Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7370--7379, 2017

    2017

  51. [59]

    Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization

    Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In European conference on computer vision, pages 191--207. Springer, 2022

    2022

  52. [60]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953--11962, 2022

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.