Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reweight task-vector subspaces before pruning fine-tuned models

desk verdict A practical, useful pruning/merging trick with broad experiments, but the core mechanistic claim needs matched-budget controls before I'd believe the subspace story. read the letter →

arxiv 2505.18713 v1 pith:OCYXO65C submitted 2025-05-24 cs.LG cs.AIcs.CL

Guodong Du , Zitao Fang , Jing Li , Junlin Li , Runhua Jiang , Shuyang Yu , Yifei Guo , Yangneng Chen
show 5 more authors
Sim Kuan Goh Ho-Kin Tang Daojing He Honghai Liu Min Zhang
This is my paper
classification cs.LGcs.AIcs.CL
keywords taskvectormodelpruningmergingcatastrophicforgettingknowledgetransfercompressionevolutionarysearchCMA-ES
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that fine-tuned models can be slimmed more effectively by first reweighting the parameter groups of their task vector, the difference from the pretrained weights, rather than pruning the raw fine-tuned weights. The proposed method, Neural Parameter Search (NPS), splits the task vector into magnitude-ranked subspaces, uses an evolutionary search to assign each subspace a scalar weight on a calibration set, and then prunes by magnitude. If this works, practitioners get a training-free way to reduce catastrophic forgetting when interpolating with the pretrained model, to merge multiple models without retraining, and to compress checkpoints while keeping near-original accuracy. The paper reports consistent gains across vision, NLP, and multimodal benchmarks, including a 4.3% improvement on T5-base fusion and about 40% better compression efficiency on vision tasks.

What carries the argument

The central object is the reweighted task vector $\tau = \sum_{m=1}^{M} w_m \cdot q_m$, where $q_m$ are magnitude-ranked subspaces of the difference between fine-tuned and pretrained weights. The scalar weights are searched by CMA-ES on a calibration set, and a final magnitude mask selects the surviving parameters. This machinery turns pruning into a low-dimensional search problem rather than a per-parameter choice, and the same pruned task vector plugs into interpolation, merging, and compression.

What would settle it

Take a fine-tuned model where the largest-magnitude task-vector entries are deliberately corrupted or irrelevant while small-magnitude entries carry the task signal; if NPS still prunes away the small entries and loses accuracy, then the magnitude-reweighting premise fails. A concrete version is to swap in a random perturbation on the top 10% of task-vector entries, run the NPS search on a calibration set, and then check whether the pruned model recovers the original accuracy.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the parameters of a task vector are not equally important, and that importance can be captured by learning a small number of scalar weights over magnitude-sorted subspaces. Starting from $\tau = \theta_{\mathrm{finetuned}} - \theta_{\mathrm{pretrained}}$, the method splits $\tau$ into $M$ subspaces $q_m$ by magnitude, searches weights $\{w_m\}$ with CMA-ES against validation accuracy on a calibration set, and only then applies a magnitude mask to obtain the pruned model $\hat{\theta}_{ft} = \theta_{\mathrm{pre}} + m \odot \tau$. The paper claims this reweighted-and-then-pruned task vector preserves performance at far higher sparsity than baselines such as TIES and DARE, and that the same pruned representation yields better interpolation for forgetting mitigation, better merging across tasks and modalities, and lower storage cost.

Load-bearing premise

The load-bearing premise is that a handful of scalar weights, one per magnitude-ranked chunk of the task vector, can be found on a calibration set that tells the method which parameters are actually important, and that pruning by magnitude after that reweighting keeps the important ones.

Editorial extensions

If this is right

  • At sparsity ratios where TIES and DARE degrade, NPS claims to preserve accuracy; for example, ViT-B/32 keeps near-original accuracy at a sparsity ratio of 0.04.
  • Merging NPS-pruned task vectors outperforms the baselines in all six reported settings, including fully fine-tuned NLP models, (IA)3 PEFT adapters, LLMs, vision models, and emotion-domain models.
  • Knowledge transfer on the LLaVA multimodal benchmark can be improved by interpolating the NPS-pruned model with the pretrained model, preserving performance at 10% sparsity while raising average metrics over prior forgetting-mitigation methods.
  • Knowledge compression can store only the pretrained weights, pruned task vectors, and binary masks, reducing storage below the TALL Mask + TIES baseline while maintaining roughly 99-100% normalized accuracy.
  • The whole search is gradient-free and completes in a few hours even for large language models, so it needs no backpropagation through the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If magnitude-ranked scalar reweighting generalizes, it suggests a cheap universal preprocessing step for any task-vector-based operation, since it only needs a handful of calibration examples.
  • Because CMA-ES is population-based, the search cost grows with the number of subspaces, so an adaptive partition could remove the need to tune that hyperparameter.
  • The reliance on a calibration set means NPS is not fully data-free, and in low-data regimes the search could overfit to the calibration split, so a natural test is whether performance holds on held-out tasks.
  • A testable extension is per-layer or per-block weighting instead of a single scalar per global magnitude stratum, which may further improve the pruning-versus-accuracy trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Neural Parameter Search (NPS), a pruning method for fine-tuned models built on task vectors. NPS splits a task vector into M subspaces by parameter magnitude, uses CMA-ES to reweight these subspaces against a calibration set, and then applies magnitude pruning to the reweighted vector. The resulting pruned models are used for knowledge transfer, model fusion, and compression. Experiments cover LLaVA-based multimodal transfer, T5/LLaMA/ViT/RoBERTa merging, PEFT adapter merging, and checkpoint compression; the paper reports consistent gains over baselines such as DARE, TIES, and Model Tailor, including a +2.1/+1.6 gain on T5-base/T5-large fusion and improved storage-accuracy trade-offs.

Significance. If the reported gains hold, NPS offers a simple, gradient-free way to improve pruned fine-tuned models for transfer, fusion, and compression, with only a few scalar weights per magnitude stratum. The paper's strengths include a broad set of benchmarks across NLP, vision, and multimodal settings, public code, and a Limitations section that honestly acknowledges the need for validation data and the computational cost of the search. However, the central claim that NPS's subspace-reweighting mechanism is responsible for the gains is not yet established, because the search budget given to NPS is far larger than that given to the baselines and because most tables lack error bars; the reported margins are often only 1–3 points, so the confounds are load-bearing.

major comments (3)
  1. [§3.2, Eq. (7), Table 13] The reported gains may partly reflect a larger validation-set search budget rather than the proposed subspace-reweighting mechanism. In §3.2 and §B.3, NPS runs 20–50 generations of CMA-ES on the per-subspace weights, and in Eq. (7) the fusion coefficients λ_i are also evolved; the baselines in Table 12 are only grid-searched over r and λ. Because the optimization objective is the calibration-set accuracy of the final pruned/merged model, NPS has substantially more capacity to fit the validation set. The paper does not report a control in which TIES, DARE, or Task Arithmetic are given the same number of validation-set evaluations (or in which NPS is restricted to the baseline grid budget). I would like to see such matched-budget experiments, or at least an analysis that varies the CMA-ES generations for NPS and shows that the gains persist with a comparable budget; without this, the claimed superiority over the baselines is not established.
  2. [Eq. (4)] The mask formula in Eq. (4) as printed, `md = 1 if τd ≥ sorted(τ)[r × d]`, appears to contain a typo: the threshold `[r × d]` depends on the index d, which is not a meaningful sparsity threshold. It should presumably be `[r × D]`, where D is the total number of parameters, or an equivalent quantile expression. Please correct the equation and clarify whether the threshold is computed over the original task vector or the reweighted task vector from Eq. (2).
  3. [Tables 1–3, §4.5] With the exception of the five-seed emotion experiment reported in §4.4, most tables report a single run and no variance information. Given that NPS's gains over the best baseline are modest (+0.9 to +3.0 percentage points in Table 2, and similar in Tables 1 and 3), the results could be within run-to-run noise. The authors should either provide multiple seeds with standard deviations for the main fusion, transfer, and compression comparisons, or explain why variance is negligible for these deterministic setups. Without this, the small margins are hard to interpret as robust improvements.
minor comments (4)
  1. [Abstract and §3.2] The abstract and §3.2 describe the method as searching 'within low-rank subspaces.' However, the actual construction partitions parameters by magnitude into strata and reweights them; this is not a low-rank decomposition. Please reword the description to avoid implying a low-rank structure that is not used (e.g., 'magnitude-based subspaces' or 'weight strata').
  2. [§3.2, Eq. (2)–(5)] The notation in Eqs. (2)–(5) is ambiguous because τ is reused for both the original task vector and the reweighted task vector. The text says 'we reallocated weights ... to obtain a new task vector: τ = ...' and then uses τ again in Eqs. (4)–(5). Please use a distinct symbol, such as τ_w, for the reweighted vector and state explicitly which vector enters the mask computation.
  3. [Table 1] The first row in both blocks of Table 1 is labeled 'Zero-shot-' with a trailing hyphen; this appears to be a typo for 'Zero-shot'. Please correct the label.
  4. [Appendix A/B] Minor naming inconsistencies: the method is referred to as both 'Talls Mask' and 'TALL-Mask' in different places (e.g., Appendix A, Section 4.2, Table 3). Please standardize the name.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: NPS is an explicit validation-set search; reported gains are fitted results, not predictions, and self-citations are not load-bearing.

full rationale

The method's derivation chain is self-contained. NPS partitions the task vector into magnitude subspaces (Eq. 2), optimizes the subspace weights with CMA-ES on a calibration set, and then magnitude-prunes the adjusted vector (Eqs. 3-5); the fusion coefficients in Eq. 7 are likewise evolved on validation accuracy. None of these quantities is defined in terms of the reported test result, and no fitted parameter is relabeled as an independent prediction. The paper explicitly discloses the validation-data requirement in the Limitations section, which is consistent with treating the test numbers as evaluations of a searched model rather than as predictions. Self-citations (Model Evolver, PCB Merging, CADE, Disentangling) appear only in related-work and baseline descriptions and carry no load in the derivation. The only substantive concern is a comparison-fairness confound: baselines are grid-searched over r and lambda while NPS receives a larger evolutionary search budget, so gains may partly reflect search budget. That is an experimental-control issue, not a circularity, and does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its free parameters (M, r, lambda, CMA-ES settings) are tuned on validation data, so the reported gains partly reflect this tuning. The axioms are the typical assumptions of task-vector methods: magnitude as importance, validation representativeness, and search convergence.

free parameters (4)
  • M (number of subspaces) = 5 in Figure 6; up to 8 in ablation
    Partitions the task vector into M magnitude-based subspaces; the paper searches over M and shows performance improves with M up to 8, but does not fix a single value for all experiments.
  • r (sparsity ratio) = 0.05 to 0.1 depending on task
    Controls the pruning mask; optimal r is selected on validation data, as reported in Table 12.
  • lambda (interpolation coefficient) = 0.9 to 2.2
    Scales the pruned task vector in transfer and fusion equations; tuned per task on validation data (Table 12).
  • CMA-ES generations and population size = 20 to 50 generations; population not specified
    Hyperparameters of the evolutionary search; the paper reports generation counts but not the full CMA-ES configuration, so these are effectively free parameters.
assumptions (3)
  • domain assumption Magnitude of task vector components correlates with importance for the task.
    The method partitions parameters by absolute magnitude and assumes reweighting these strata can improve pruning; this is not proven and is the core modeling assumption.
  • domain assumption The calibration (validation) set is representative of the test distribution.
    The search optimizes validation accuracy; if the validation set is not representative, the selected weights may not generalize to the test distribution.
  • domain assumption CMA-ES with limited generations finds a good enough optimum.
    The paper relies on convergence in 20-50 generations without a guarantee; the quality of the found weights is an empirical assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer." pith.science (2026). https://pith.science/paper/OCYXO65C

@misc{pith2026250518713,
  author       = {Pith},
  title        = {Pith review of: Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCYXO65C}},
  note         = {Machine review of arXiv:2505.18713}
}
read the original abstract

Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the original model. Recognizing that different task vector subspaces contribute variably to model performance, we introduce a novel method called Neural Parameter Search (NPS-Pruning) for slimming down fine-tuned models. This method enhances pruning efficiency by searching through neural parameters of task vectors within low-rank subspaces. Our method has three key applications: enhancing knowledge transfer through pairwise model interpolation, facilitating effective knowledge fusion via model merging, and enabling the deployment of compressed models that retain near-original performance while significantly reducing storage costs. Extensive experiments across vision, NLP, and multi-modal benchmarks demonstrate the effectiveness and robustness of our approach, resulting in substantial performance gains. The code is publicly available at: https://github.com/duguodong7/NPS-Pruning.

Figures

Figures reproduced from arXiv: 2505.18713 by the authors.

Figure 1
Figure 1. Knowledge transfer, fusion, and compression [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance of ViT-B/32 models on a specific [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The framework of Neural Parameter Search enhances the efficiency of pruning fine-tuned models. This is [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance variations of different methods [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Averaged normalized accuracy and storage cost versus the number of tasks on computer vision benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Performance convergence of the pruned fine [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Test set performance when merging ViT-B/32 and ViT-L/14 models on eight image classification tasks. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive Detoxification: Safeguarding General Capabilities of LLMs through Toxicity-Aware Knowledge Editing

    cs.CL 2025-05 conditional novelty 6.0 of 10

    ToxEdit uses a hidden-state toxicity classifier to route unsafe queries through an edited feed-forward network, preserving normal responses on safe queries.

  2. Multi-objective Large Language Model Alignment with Hierarchical Experts

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.

Reference graph

Works this paper leans on

89 extracted references · 53 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 579--586

    2005

  4. [4]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://arxiv.org/abs/2401.15024 Slicegpt: Compress large language models by deleting rows and columns . Preprint, arXiv:2401.15024

    arXiv 2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [6]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML), pages 1597--1607

    2020

  7. [7]

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. 2017. Remote sensing image scene classification: Benchmark and state of the art. In Proceedings of the IEEE, pages 1865--1883

    2017

  8. [8]

    Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044

    arXiv 2022

Show all 89 references
  1. [9]

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2014

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Rocktim Jyoti Das, Liqun Ma, and Zhiqiang Shen. 2023. https://arxiv.org/abs/2311.04902 Beyond size: How gradients shape pruning decisions in large language models . Preprint, arXiv:2311.04902

    2023 arXiv

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  5. [13]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognit...

    2021

  6. [14]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint...

    2020 arXiv

  7. [15]

    GuoDong Du, HaoJian Deng, JiaHao Su, and Yuan Huang. 2023. End-to-end rain streak removal with raw images. arXiv preprint arXiv:2312.13304

    2023 arXiv

  8. [16]

    Guodong Du, Runhua Jiang, Senqiao Yang, Haoyang Li, Wei Chen, Keren Li, Sim Kuan Goh, and Ho-Kin Tang. 2024 a . Impacts of darwinian evolution on pre-trained deep neural networks. In 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1907--1912. IEEE

    2024

  9. [17]

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. 2024 b . Parameter competition balancing for model merging. Advances in Neural Information Processing Systems (NeurIPS), 37

    2024

  10. [18]

    Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. 2024 c . Knowledge fusion by evolving weights of language models. arXiv preprint arXiv:2406.12208

    2024 arXiv

  11. [19]

    Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Jing Li, Ho-Kin Tang, and Sim Kuan Goh. 2025. Disentangling task interference within neurons: Model merging in alignment with neuronal mechanisms. arXiv preprint arXiv:2503.05320

    2025 arXiv

  12. [20]

    Ronald A Fisher. 1922. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character (PTRSL), pages 309--368

    1922

  13. [21]

    Elias Frantar and Dan Alistarh. 2023. SparseGPT : Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774

    2023 arXiv

  14. [22]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9

    2007

  15. [23]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages ...

    2017

  16. [24]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), ...

    2018

  17. [25]

    Nikolaus Hansen and Andreas Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation (ICEC), pages 312--317

    1996

  18. [26]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing (STAEORS), pages 2217--2226

    2019

  19. [27]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning (ICML), pages 2790--2799

    2019

  20. [28]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2022

  21. [29]

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2023. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269

    2023 arXiv

  22. [30]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 6700--6709

    2019

  23. [31]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023 a . Editing models with task arithmetic. In Proceedings of the International Conference on Learning Representations (ICLR)

    2023

  24. [32]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023 b . Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations (ICLR)

    2023

  25. [33]

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022. Patching open-vocabulary models by interpolating weights. In Proceedings of Neural Information Processing Systems (NeurIPS), pages...

    2022

  26. [34]

    Runhua Jiang, Guodong Du, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. 2024. Cade: Cosine annealing differential evolution for spiking neural network. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE

    2024

  27. [35]

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849

    2022 arXiv

  28. [36]

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In Proceedings of the International Conference on Learning Representations (ICLR)

    2023

  29. [37]

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 8082--8090

    2020

  30. [38]

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pages 554--561

    2013

  31. [39]

    Yann LeCun. 1998. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/

    1998

  32. [40]

    Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the International Conference on the Principles of Knowledge Representation and Reasoning (KR)

    2012

  33. [41]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023 a . Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

    2023 arXiv

  34. [42]

    Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957

    2017 arXiv

  35. [43]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023 b . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  36. [44]

    Liang, J

    T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, pages 370--403

    2021

  37. [45]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022 a . Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of Neural Information Processing Systems (NeurIPS), pages 1950--1965

    2022

  38. [46]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022 b . Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of Neural Information Processing Systems (NeurIPS), pages 1950--1965

    2022

  39. [47]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744

    2023 arXiv

  40. [48]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023 b . Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2023 arXiv

  41. [49]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of Neural Information Processing Systems...

    2022

  42. [50]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. In In Proceedings of Neural Information Processing Systems (NeurIPS), pages 21702--21720

    2023

  43. [51]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition (CVPR), pages 3195--3204

    2019

  44. [52]

    Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank : Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung (SUB), pages 107--124

    2019

  45. [53]

    Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. In Proceedings of Neural Information Processing Systems (NeurIPS), pages 17703--17716

    2022

  46. [54]

    Mohammad

    Saif M. Mohammad. 2012. \# emotional tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (SEM), pages 246--255

    2012

  47. [55]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 7

    2011

  48. [56]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI : A new benchmark for natural language understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2020

  49. [57]

    Laura Ana Maria Oberl \"a nder and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 2104--2119

    2018

  50. [58]

    Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600

    2023 arXiv

  51. [59]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC : The word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    2019

  52. [60]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of International conference on...

    2021

  53. [61]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), pages 1--67

    2020

  54. [62]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series

    2011

  55. [63]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2023 arXiv

  56. [64]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, pages 99--106

    2021

  57. [65]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In Proceedings of the International Conference...

    2022

  58. [66]

    Klaus R Scherer and Harald G Wallbott. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology (PSP), page 310

    1994

  59. [67]

    Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 752--757

    2018

  60. [68]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 8317--8326

    2019

  61. [69]

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. 2011. The german traffic sign recognition benchmark: a multi-class classification competition. In Proceedings of the International Joint Conference on Neural Networks (IJCNN)

    2011

  62. [70]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  63. [71]

    Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language...

    2019

  64. [72]

    Derek Tam, Mohit Bansal, and Colin Raffel. 2024. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Research (TMLR)

    2024

  65. [73]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  66. [74]

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Fran c ois Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813

    2024 arXiv

  67. [75]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasin...

    2022

  68. [76]

    M. Xia, Z. Zhong, and D. Chen. 2022. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1513--1528

    2022

  69. [77]

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. 2016. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), pages 3--22

    2016

  70. [78]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. In Proceedings of Neural Information Processing Systems (NeurIPS)

    2024

  71. [79]

    Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2013--2018

    2015

  72. [80]

    Ziqing Yang, Yiming Cui, Xin Yao, and Shijin Wang. 2023. Gradient-based intra-attention pruning on pre-trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)

    2023

  73. [81]

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. 2023. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. In arXiv preprint arXiv:2310.05175

    2023 arXiv

  74. [82]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), pages 67--78

    2014

  75. [83]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 a . Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099

    2023 arXiv

  76. [84]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023 b . Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  77. [85]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag : Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2019

  78. [86]

    Jinghan Zhang, Junteng Liu, Junxian He, et al. 2023 a . Composing parameter-efficient modules with arithmetic operation. In Proceedings of in Neural Information Processing Systems (NeurIPS), pages 12589--12610

    2023

  79. [87]

    Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023 b . Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arX...

    2023 arXiv

  80. [88]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)

    2019

  81. [89]

    Didi Zhu, Zhongyisun Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. 2024. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In Proceedings of Forty-first International Conference on Machine Learning (ICML)

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.