Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

Dynamic Base model Shift for Delta Compression

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Shifting the base model toward each task keeps fine-tuned accuracy nearly intact at 99.8% delta compression.

desk verdict Nice idea, real evaluation flaw: fitting two scalars on 10% of each test set makes the headline gains hard to trust until rerun on a clean split. read the letter →

arxiv 2505.11344 v1 pith:K7HVOPZD submitted 2025-05-16 cs.CV cs.LG

Chenyu Huang , Peng Ye , Shenghe Zheng , Xiaohui Wang , Lei Bai , Tao Chen , Wanli Ouyang This is my paper · ORCID
classification cs.CVcs.LG
keywords deltacompressiontaskvectorsmodelmerging1-bitquantizationparameterpruningbaseshiftfine-tunedstoragemulti-taskadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to overturn a default in delta compression: instead of always compressing the difference between a fine-tuned model and the frozen pretrained weights, it shifts the base model toward the target task before compressing. The shift is controlled by two numbers per task, lambda_1 and lambda_2, chosen by low-cost optimization against the fine-tuned model's outputs. The claim is that this restores most of the fine-tuned model's accuracy at extreme compression rates, where existing methods collapse. If true, it means the storage savings of delta compression can be kept without sacrificing task quality, and the recipe can be bolted onto both pruning-based and quantization-based compressors.

What carries the argument

The load-bearing object is the reconstruction identity in Eq. 9: $W'_t = W_{\text{pre}} + \lambda_{1,t}\tau_{\text{base}} + \lambda_{2,t} C(W_t - W_{\text{pre}} - \lambda_{1,t}\tau_{\text{base}})$, with $\tau_{\text{base}} = \text{BitDelta}(W_{\text{avg}} - W_{\text{pre}})$ shared across tasks. Here $\tau_{\text{base}}$ is a 1-bit encoded vector capturing the average direction in which all tasks moved away from the pretrained model, $C$ is the delta compressor (DARE or BitDelta), and $\lambda_1, \lambda_2$ are per-task scalars. The base shift lowers the variance of the remaining delta, making the same pruning or quantization operator produce better reconstructions, and $\lambda_2$ then rescales the compressed per-task delta.

What would settle it

On one benchmark, keep the exact DBMS protocol except optimize $\lambda_1$ and $\lambda_2$ on the training split, or on a separate held-out validation set, instead of on 10% of the test set, then evaluate on the test set at 99.8% sparsity. If average accuracy falls back toward the vanilla DARE baseline (near 8.9 on ViT-B/32) instead of reaching roughly 80.8, the headline gain is test-set adaptation. A complementary check: replace $\tau_{\text{base}}$ with a fixed random direction of the same magnitude; if the gains persist, the shared task-average direction is not the active mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that the pretrained model is only one possible base and usually not the best one. Its Dynamic Base Model Shift builds a shared shifted base $W_{\text{pre}} + \lambda_1 \tau_{\text{base}}$, where $\tau_{\text{base}}$ is the 1-bit compressed average of all task vectors, and then reconstructs each task as $W'_t = W_{\text{pre}} + \lambda_{1,t} \tau_{\text{base}} + \lambda_{2,t} C(W_t - W_{\text{pre}} - \lambda_{1,t} \tau_{\text{base}})$. The two scalars are trained per task by minimizing the squared output difference against the fine-tuned model on a small data sample. The paper reports that this recipe rescues heavily compressed models: at 99.8% DARE sparsity, ViT-B/32 average accuracy rises from 8.9 to 80.8 and ViT-L/14 from 45.7 to 90.2, and similar gains appear with BitDelta-style 1-bit compression and on language, vision, and multi-modal models.

Load-bearing premise

The load-bearing premise is that tuning $\lambda_1$ and $\lambda_2$ on a random 10% slice of the very test sets used to report accuracy is a fair comparison with baselines that never see test data; if that tuning leaks signal from the test set, the reported gains are inflated.

Editorial extensions

If this is right

  • At extreme compression ratios, where vanilla DARE collapses (ViT-B/32 average 8.9 at 99.8% sparsity), DBMS restores usable accuracy (80.8), so very high delta sparsity becomes practical.
  • The same two-scalar recipe also improves BitDelta-style 1-bit quantization, not just pruning, so it operates at the level of the compression paradigm rather than one algorithm.
  • The shared base vector $\tau_{\text{base}}$ is stored once in 1-bit form plus two scalars per task, so the overhead is small when many fine-tuned models are stored from one pretrained checkpoint.
  • The pattern holds across language models (RoBERTa, GPT-2, Flan-T5), vision transformers (ViT-B/32, ViT-L/14), and a multi-modal model (BEiT3), suggesting the shift benefit is not specific to one architecture or modality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shift mechanism is real, base selection is a degree of freedom that other compression and merging methods could exploit; for example, a cluster of highly related tasks might benefit from its own shared base vector instead of one global average.
  • A natural extension is to make $\lambda_1$ and $\lambda_2$ per layer or per attention block rather than global per task; the paper's variance-reduction story predicts larger gains at 99.8% compression when the shift is applied where the delta is largest.
  • Because the tuning sample is drawn from the test sets, an evaluation that tunes on training data only would clarify how much of the gain generalizes; the paper does not report that experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes Dynamic Base Model Shift (DBMS), a plug-in method for delta compression. Instead of using the pretrained model as the static base, DBMS constructs a per-task shifted base W_base = W_pre + λ1 * BitDelta(W_avg - W_pre), then applies a per-task rescaling λ2 to the compressed delta before reconstruction via Eq. 9. The two scalars are initialized in closed form (Eq. 7) and then refined by MSE minimization on 10% of the test inputs (Eq. 8). Experiments cover GPT-2, RoBERTa, Flan-T5, ViT-B/32, ViT-L/14, and BEiT3, reporting large gains at high compression rates (e.g., DARE 99.8% on ViT-B/32 from 8.9 to 80.8). The claim is that the pretrained base is suboptimal and that the dynamic shift plus rescale preserves accuracy at extreme compression.

Significance. If the empirical claims survive a clean evaluation protocol, DBMS would be a useful and refreshingly simple contribution: the closed-form initialization of λ1 in Eq. 7 is parameter-free, and the method is explicitly orthogonal to both pruning-based (DARE) and quantization-based (BitDelta) compressors, with experiments spanning multiple modalities. The paper also identifies a real limitation of existing delta-compression work. However, the headline post-training results are currently confounded by test-set adaptation, so the significance is contingent on a re-evaluation with a proper held-out split.

major comments (2)
  1. [Section 3.2.2 (Eq. 8) and Tables 1-7] The post-training evaluation is circular in a benchmark sense: λ1 and λ2 are optimized on X_t, described in Section 3.2.2 as '10% of the input data randomly selected from the test datasets', and the reported accuracies in Tables 1-7 are measured on the full test benchmarks, which include those same 10% of inputs. Baselines such as DARE and BitDelta never see any test inputs, so the comparison is not apples-to-apples. The gains attributed to the dynamic-base construction may largely reflect two per-task scalars fitted to the evaluation distribution itself. Please re-run the protocol with a disjoint split: either fit on training/validation data and evaluate on the full test set, or fit on a 10% test subset and report accuracy only on the remaining 90%. The paper should also state clearly which evaluation protocol was used for every table.
  2. [Appendix A.2, Eq. 17] The theoretical support is mathematically unsound as written. From the least-squares initialization, minimizing ||W_pre + λ1 τ_base - W_t||^2 implies τ_base · (W_t - W_pre - λ1 τ_base) = 0, so the vectors λ1 τ_base and Δ_ours are orthogonal as vectors. However, Eq. 17 then writes Var(Δ_ori) = Var(Δ_ours) + (λ1)^2 Var(τ_base), which is not implied by vector orthogonality. Variance of a sum of random vectors requires zero-mean or uncorrelated components; elementwise orthogonality of two fixed vectors does not make their variances additive. This step needs a corrected derivation, an explicit zero-mean assumption on the delta parameters, or removal of this theoretical claim.
minor comments (4)
  1. [Section 3.2.2 / Algorithm 1] The training loop description is slightly inconsistent: Algorithm 1 loops once over x in X[t], while Figure 3 reports loss curves for 'the first 100 epochs'. Please clarify how many epochs/iterations are used and how X_t is sampled in each epoch.
  2. [Throughout] There are several typos: 'commpression' (Section 4.1), 'Trainig' (Table 7 header), 'perfromance' (Section 4.1), 'initaliztion' (Figure 3 caption). A careful proofread is needed.
  3. [Section 3.2.2 / Appendix E] The method requires 'partial unlabeled test data', but Appendix E lists this only as a computational cost and never discusses the legitimacy of using test inputs during model construction. Even if the data are unlabeled, using the test distribution for fitting parameters is a benchmark-contamination concern that should be addressed explicitly in the limitations.
  4. [Section 4.1 and Table 4] Table 4 reports λ1 and λ2 before and after training for RoBERTa, but the values are not analyzed. For example, several λ2 values after DARE 99.8% training are far below 1 (e.g., 0.1258 for MNLI), which suggests the rescaling is absorbing most of the compression error. A brief discussion of why these values are sensible would strengthen the paper.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline post-training gains are fitted on test-set samples and evaluated on the same benchmarks; the 'Post Training' results are in-sample fitted values, not clean predictions.

  1. fitted input called prediction [Section 3.2.2, Eq. 8-9; Tables 1-7; Appendix E]
    "Specifically, in our experiments, the X_t are 10% of the input data randomly selected from the test datasets. We use an Adam optimizer to update the lambda_1 and lambda_2 values ... After the training process, we obtain ... Before inference on task T_t, we obtain the compressed model through Eq. 9."

    The two scalars lambda_1 and lambda_2 are optimized by minimizing the MSE in Eq. 8 over X_t, which the paper explicitly states are 10% of the test datasets. Eq. 9 then defines the compressed model using those fitted scalars. All headline 'Post Training' accuracies in Tables 1-7 are measured on those same benchmark test sets. Thus the reported post-training performance is not an independent prediction of the method; it is the value of two per-task fitted parameters evaluated on the distribution used to fit them. Baselines DARE and BitDelta-Init receive no such adaptation. The 'Init' rows (Eq. 7) are a parameter-free contribution, but the paper's central claim relies on the post-training numbers.

full rationale

The mathematical derivation of DBMS (Eqs. 4-9) is not self-referential: the dynamic base is formed from the compressed average task vector, and the per-task scalars are legitimate free parameters. The closed-form lambda_1 initialization in Eq. 7 is computed from weights only and does not use test data. The circularity is in the evaluation protocol for the post-training results. Because lambda_1 and lambda_2 are fit by minimizing MSE on 10% of each test set and accuracy is then reported on the same test benchmarks, the large 'Post Training' gains (e.g., ViT-B/32 DARE 99.8% from 8.9 to 80.8) are in-sample fitted values rather than clean held-out predictions. This is a fitted-input-called-prediction pattern rather than a self-definitional identity. Appendix E acknowledges the need for partial unlabeled data and training cost, but it does not flag the benchmark-contamination issue; it frames the data as merely unlabeled. No load-bearing self-citation or imported uniqueness theorem was found: [16] and [17] are prior works by the same authors but are used only to describe settings and related work, not to justify the main derivation. Therefore the score is 6, reflecting partial circularity in the headline evaluation, while the initialization result remains independent.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contributions rest on two trainable per-task scalars, a shared 1-bit average task vector, and a transductive training protocol. No new physical entities or formal objects are introduced. The most consequential assumption is that fitting on a subset of the test set is an acceptable evaluation protocol.

free parameters (4)
  • per-task λ1 = e.g., 0.23 to 3.79 for RoBERTa with DARE 99.8% after training
    Each task gets a trainable scalar that determines the magnitude of the base model shift. It is fit on a 10% subset of the test set using MSE loss.
  • per-task λ2 = e.g., 0.13 to 0.90 for RoBERTa with DARE 99.8% after training
    Each task gets a trainable scalar that scales the compressed delta parameters. It is fit on the same test subset and directly controls the reported accuracy.
  • unlabeled test subset fraction = 10%
    The paper chooses 10% of each test set for scalar training. This fraction is a hand-selected design choice and is not ablated across values.
  • training epoch count = not specified in Algorithm 1
    The ablation mentions the first 100 epochs, but the main algorithm does not state how many epochs are used. This is a hidden training-budget choice that affects results.
assumptions (4)
  • domain assumption MSE distillation on a 10% subset of the test set is a valid optimization target for preserving task accuracy.
    Section 3.2.2 uses this objective to fit λ1 and λ2. If optimizing output matching on test inputs does not transfer to the full test set, the reported post-training gains are artifacts.
  • ad hoc to paper The average of the finetuned task vectors, once 1-bit compressed, is a useful shared base shift for every task.
    Equation 4 introduces τbase as a single shared base for all tasks. No independent evidence is given that one compressed average vector is a good base for every task.
  • domain assumption Variance of delta parameters is a valid proxy for compressibility.
    Appendix A.2 argues that reducing variance makes the residual easier to compress. This is a heuristic connection and is not itself established.
  • domain assumption Using unlabeled test inputs for training does not invalidate comparisons to methods that do not use test data.
    The paper discloses the test-data usage but never justifies it as a fair protocol against DARE and BitDelta baselines. This is the load-bearing assumption behind the headline results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Base model Shift for Delta Compression." pith.science (2026). https://pith.science/paper/K7HVOPZD

@misc{pith2026250511344,
  author       = {Pith},
  title        = {Pith review of: Dynamic Base model Shift for Delta Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7HVOPZD}},
  note         = {Machine review of arXiv:2505.11344}
}
read the original abstract

Transformer-based models with the pretrain-finetune paradigm bring about significant progress, along with the heavy storage and deployment costs of finetuned models on multiple tasks. Delta compression attempts to lower the costs by reducing the redundancy of delta parameters (i.e., the difference between the finetuned and pre-trained model weights) through pruning or quantization. However, existing methods by default employ the pretrained model as the base model and compress the delta parameters for every task, which may causes significant performance degradation, especially when the compression rate is extremely high. To tackle this issue, we investigate the impact of different base models on the performance of delta compression and find that the pre-trained base model can hardly be optimal. To this end, we propose Dynamic Base Model Shift (DBMS), which dynamically adapts the base model to the target task before performing delta compression. Specifically, we adjust two parameters, which respectively determine the magnitude of the base model shift and the overall scale of delta compression, to boost the compression performance on each task. Through low-cost learning of these two parameters, our DBMS can maintain most of the finetuned model's performance even under an extremely high compression ratio setting, significantly surpassing existing methods. Moreover, our DBMS is orthogonal and can be integrated with a variety of other methods, and it has been evaluated across different types of models including language, vision transformer, and multi-modal models.

Figures

Figures reproduced from arXiv: 2505.11344 by the authors.

Figure 1
Figure 1. Comparison between vanilla delta compression methods (left) and our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance heatmap of compressing RoBERTa models finetuned on the GLUE [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The loss curve of DBMS using different initaliztion strategies, respectively ours and ones-initialization [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of (a) parameter num￾bers and (b) performance when applied to eight RoBERTa models. Results. The experimental results for ViT-B/32 and ViT-L/14 models are shown in Tab. 5 and 6. We observe that DBMS performs well in vision models. When combined with DBMS, th…
Figure 5
Figure 5. Figure 5: Performance heatmap of compressing ViT-B/32 models finetuned on eight vision tasks [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Performance heatmap of compressing ViT-L/14 models finetuned on eight vision tasks [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Performance heatmap of compressing RoBERTa models finetuned on the GLUE [ [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Operationalising the Superficial Alignment Hypothesis via Task Complexity

    cs.LG 2026-02 conditional novelty 7.0 of 10

    A few kilobytes of program can adapt pre-trained LLMs to strong performance on math, translation, and instruction-following—evidence that task knowledge already lives in the model.

  2. Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Task Vector Bases compresses T task vectors into M softmax-mixed basis vectors that preserve task arithmetic operations, with empirical gains over PCA and random selection.

Reference graph

Works this paper leans on

57 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.arXiv preprint arXiv:1708.00055, 2017

    arXiv 2017

  3. [3]

    Cheng, J

    G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017

    work page 2017

  4. [4]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    work page 2024

  5. [5]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014

    2014

  6. [6]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009IEEEconferenceoncomputervisionandpatternrecognition, pages248–255.Ieee, 2009

    work page 2009

  7. [7]

    W. Deng, Y. Zhao, V. Vakilian, M. Chen, X. Li, and C. Thrampoulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models.arXiv preprint arXiv:2410.09344, 2024

    arXiv 2024

  8. [8]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

Show all 57 references
  1. [9]

    Dolan and C

    B. Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. InThird international workshop on paraphrasing (IWP2005), 2005

    2005

  2. [10]

    Heigold, S

    A.Dosovitskiy,L.Beyer,A.Kolesnikov,D.Weissenborn,X.Zhai,T.Unterthiner,M.Dehghani,M.Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [11]

    Frantar, S

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  4. [12]

    Giampiccolo, B

    D. Giampiccolo, B. Magnini, I. Dagan, and W. B. Dolan. The third pascal recognizing textual entailment challenge. InProceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9, 2007. 10

    2007

  5. [13]

    J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey.International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  6. [14]

    Goyal, T

    Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  7. [15]

    Helber, B

    P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019

    2019

  8. [16]

    Huang, P

    C. Huang, P. Ye, T. Chen, T. He, X. Yue, and W. Ouyang. EMR-merging: Tuning-free high-performance modelmerging. InAdvances inNeuralInformation ProcessingSystems, volume 37, pages122741–122769, 2024

    2024

  9. [17]

    Huang, P

    C. Huang, P. Ye, X. Wang, S. Zheng, B. Qi, L. Bai, W. Ouyang, and T. Chen. Seeing delta parameters as jpegimages: Data-freedeltacompressionwithdiscretecosinetransform.arXivpreprintarXiv:2503.06676, 2025

    2025 arXiv

  10. [18]

    Ilharco, M

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  11. [19]

    B. Isik, H. Kumbong, W. Ning, X. Yao, S. Koyejo, and C. Zhang. Gpt-zip: Deep compression of finetuned large language models. InWorkshop on Efficient Systems for Foundation Models@ ICML2023, 2023

    2023

  12. [20]

    S. Iyer, N. Dandekar, K. Csernai, et al. First quora dataset release: Question pairs. data. quora. com. 2017

    2017

  13. [21]

    Jiang, Z

    Y. Jiang, Z. Yang, B. Chen, S. Li, Y. Li, and T. Li. Deltadq: Ultra-high delta compression for fine-tuned llms via group-wise dropout and separate quantization.arXiv preprint arXiv:2410.08666, 2024

    2024 arXiv

  14. [22]

    X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  15. [23]

    Y. Kim, S. Lee, A. Jung, B. Ryu, and S. Hong. Task vector quantization for memory-efficient model merging.arXiv preprint arXiv:2503.06921, 2025

    2025 arXiv

  16. [24]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  17. [25]

    Krause, M

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013

    2013

  18. [26]

    Y. LeCun. The mnist database of handwritten digits.http://yann. lecun. com/exdb/mnist/, 1998

    1998

  19. [27]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. S...

    2014

  20. [28]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  21. [29]

    J. Liu, G. Xiao, K. Li, J. D. Lee, S. Han, T. Dao, and T. Cai. Bitdelta: Your fine-tune may only be worth one bit. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  22. [30]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  23. [31]

    Z. Lu, C. Fan, W. Wei, X. Qu, D. Chen, and Y. Cheng. Twin-merging: Dynamic integration of modular expertise in model merging.Advances in Neural Information Processing Systems, 37:78905–78935, 2024

    2024

  24. [32]

    B. Ping, S. Wang, H. Wang, X. Han, Y. Xu, Y. Yan, Y. Chen, B. Chang, Z. Liu, and M. Sun. Delta-come: Training-free delta-compression with mixed-precision for large language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  25. [33]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  26. [34]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. 11

    2019

  27. [35]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  28. [36]

    Rajpurkar, J

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text.arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  29. [37]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013

    2013

  30. [38]

    Stallkamp, M

    J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-classclassificationcompetition. InThe2011internationaljointconferenceonneuralnetworks,pages 1453–1460. IEEE, 2011

    2011

  31. [39]

    A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi. A corpus for reasoning about natural language grounded in photographs.arXiv preprint arXiv:1811.00491, 2018

    2018 arXiv

  32. [40]

    A. Tang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. FusionBench: A Comprehensive Benchmark of Deep Model Fusion, June 2024

    2024

  33. [41]

    Data-adaptiveweight-ensemblingformulti-task model fusion.International Journal of Computer Vision, pages 1–17, 2025

    A.Tang,L.Shen,Y.Luo,S.Liu,H.Hu,B.Du,andD.Tao. Data-adaptiveweight-ensemblingformulti-task model fusion.International Journal of Computer Vision, pages 1–17, 2025

    2025

  34. [42]

    Mergingmulti-taskmodelsviaweight-ensembling mixture of experts.arXiv preprint arXiv:2402.00433, 2024

    A.Tang,L.Shen,Y.Luo,N.Yin,L.Zhang,andD.Tao. Mergingmulti-taskmodelsviaweight-ensembling mixture of experts.arXiv preprint arXiv:2402.00433, 2024

    2024 arXiv

  35. [43]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  36. [44]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  37. [45]

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2023

  38. [46]

    Warstadt, A

    A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments.Transactions of the Association for Computational Linguistics, 7:625–641, 2019

    2019

  39. [47]

    Abroad-coveragechallengecorpusforsentenceunderstanding through inference.arXiv preprint arXiv:1704.05426, 2017

    A.Williams,N.Nangia,andS.R.Bowman. Abroad-coveragechallengecorpusforsentenceunderstanding through inference.arXiv preprint arXiv:1704.05426, 2017

    2017 arXiv

  40. [48]

    Wortsman, G

    M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. InInternational Conferen...

    2022

  41. [49]

    J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010

    2010

  42. [50]

    Yadav, D

    P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal. Ties-merging: Resolving interference when merging models. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  43. [51]

    E. Yang, L. Shen, G. Guo, X. Wang, X. Cao, J. Zhang, and D. Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  44. [52]

    Yao and A

    X. Yao and A. Klimovic. Deltazip: Multi-tenant language model serving via delta compression.arXiv preprint arXiv:2312.05215, 2023

    2023 arXiv

  45. [53]

    L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  46. [54]

    N. Yuval. Reading digits in natural images with unsupervised feature learning. InProceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011

    2011

  47. [55]

    Zheng and H

    S. Zheng and H. Wang. Free-merging: Fourier transform for model merging with lightweight experts. arXiv preprint arXiv:2411.16815, 2024. 12 Appendix forDynamic Base Model Shift A Theoretical Analysis A.1 Initialization Strategy DBMSapplies L2 distance minimization strategy to ...

    2024 arXiv

  48. [56]

    For taskt, the objective is shown in Eq. 6 and can be rewritten as: min λt 1 ∥Wpre +λt 1·τbase−Wt∥2 2 = min λt 1 h Wpre +λt 1·τbase−Wt T Wpre +λt 1·τbase−Wt i = min λt 1 h λt 1 2 τT baseτbase +λt 1τT base (Wpre−Wt) + (Wpre−Wt)T (Wpre−Wt) +λt 1 (Wpre−Wt)Tτbase i (10) We assume ...

  49. [57]

    with respect toλt 1 to 0. That is: df(λ t 1) dλt 1 =2λt 1τT baseτbase +τ T base (Wpre−Wt) + (Wpre−Wt)Tτbase =0 (12) Solve this function, and thus the initial value ofλt 1 can be calculated by: λt 1 init = (Wt−Wpre)·τ base τbase·τbase .(13) A.2 Effectiveness Analysis ofDynamic ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.